Barracuda PoseNet Tutorial Pt. 8 (Outdated)
Version 2: Part 1
Last Updated: Dec 1, 2020
Previous: Part 7
Introduction
So far, we’ve only worked with video sources that have 16:9
aspect ratios. You might need to work with input that has a taller or wider aspect ratio depending on your application and equipment. In this post, we’ll cover how to adjust to the current source resolution at runtime.
Add UnityEngine.Video
Namespace
Open the PoseNet
script and add a new using directive
at the top. We need the UnityEngine.Video
namespace to work with the Video Player
object.
Add Webcam Variables
Unity defaults to a resolution of 640 x 480
for webcams. There is no built-in method that returns the frame rate. However, the default does not appear to be over 30fps. We can request a resolution and frame rate when initializing the webcamTexture
. Unity should accept the requested settings as long as the camera supports them. Unity defaults to its closest preset if the requested settings are not supported.
Create webcamHeight
Variable
Add a new public int
variable so we can adjust the camera height from the Inspector
tab. Name the variable webcamHeight
. My webcam supports 720p at 60fps, so I’ve set the default value to 720
.
Create webcamWidth
Variable
Next, create a variable for the camera’s width and name it webcamWidth
. I’ve set the default value to 1280
.
Create webcamFPS
Variable
We’ll also add a variable to set the frame rate for the camera and name it webcamFPS
. Set the default value to 60
.
Add Video Resolution Variables
Next, we need to create a couple of private int
variables to store the dimensions of the video source. Name the variables videoHeight
and videoWidth
.
Update Start()
Method
We’ll make some modifications to the Start()
method to prepare for input with different aspect ratios.
Get Reference to Video Player
Create a new GameObject
variable to access the Video Player
object. Name the variable videoPlayer
and place it at the top of the Start()
method. We’ll also move the Transform videoScreen
variable outside of the if (useWebcam)
statement.
Get Webcam Resolution
We can’t access the webcam resolution until after the webcam has started. Therefore, we’ll update the values for videoHeight
and videoWidth
after webcamTexture.Play()
has been called.
Get Video Clip Dimensions
We need to get the video resolution from the Video Player
object when we’re not using a webcam. We can get the height and width by accessing the VideoPlayer
component.
Replace videoTexture
The dimensions for videoTexture
can’t be updated once it’s been created. We’ll need to replace it with a new RenderTexture
that uses the new videoHeight
and videoWidth
values. We also need to set the new videoTexture
as the targetTexture
for the Video Player
object.
Update VideoScreen
Next, we need to apply the new videoTexture
to the VideoScreen
object. We’ll also resize and reposition the VideoScreen
to account for any changes in the videoTexture
dimensions.
Adjust Main Camera
We’ll also resize and reposition the Main Camera
object to account for changes to the VideoScreen
.
Update ProcessOutput()
Method
Finally, we need to modify how we calculate the key point locations.
Update Scaling Calculations
We’ll account for different aspect ratios by first determining whether the height or width of videoTexture
is larger. We then use that information to calculate the scale
and unsqueezeScale
values.
Update Key Point Calculations
We should only scale the xPos
value by the unsqueezeScale
if the width larger than the height. Remove that part from the initial xPos
calculation. We also need to move the if (useWebcam)
statement.
We’ll scale either xPos
or yPos
by unsqueezeScale
depending on whether the width or height value for videoTexture
is larger. Place the if (useWebcam)
statement after this calculation.
Try It Out
Now we can test the modifications on some vertical videos from Pexels.
Summary
Now we can use video sources with different aspect ratios without needing to manually change any parameters.