Barracuda PoseNet Tutorial Pt. 8 (Outdated)

unity
tutorial
This post covers how to handle video input with different aspect ratios.
Author

Christian Mills

Published

November 20, 2020

Version 2: Part 1

Last Updated: Dec 1, 2020

Previous: Part 7

Introduction

So far, we’ve only worked with video sources that have 16:9 aspect ratios. You might need to work with input that has a taller or wider aspect ratio depending on your application and equipment. In this post, we’ll cover how to adjust to the current source resolution at runtime.

Add UnityEngine.Video Namespace

Open the PoseNet script and add a new using directive at the top. We need the UnityEngine.Video namespace to work with the Video Player object.

Add Webcam Variables

Unity defaults to a resolution of 640 x 480 for webcams. There is no built-in method that returns the frame rate. However, the default does not appear to be over 30fps. We can request a resolution and frame rate when initializing the webcamTexture. Unity should accept the requested settings as long as the camera supports them. Unity defaults to its closest preset if the requested settings are not supported.

Create webcamHeight Variable

Add a new public int variable so we can adjust the camera height from the Inspector tab. Name the variable webcamHeight. My webcam supports 720p at 60fps, so I’ve set the default value to 720.

Create webcamWidth Variable

Next, create a variable for the camera’s width and name it webcamWidth. I’ve set the default value to 1280.

Create webcamFPS Variable

We’ll also add a variable to set the frame rate for the camera and name it webcamFPS. Set the default value to 60.

Add Video Resolution Variables

Next, we need to create a couple of private int variables to store the dimensions of the video source. Name the variables videoHeight and videoWidth.

Update Start() Method

We’ll make some modifications to the Start() method to prepare for input with different aspect ratios.

Get Reference to Video Player

Create a new GameObject variable to access the Video Player object. Name the variable videoPlayer and place it at the top of the Start() method. We’ll also move the Transform videoScreen variable outside of the if (useWebcam) statement.

Get Webcam Resolution

We can’t access the webcam resolution until after the webcam has started. Therefore, we’ll update the values for videoHeight and videoWidth after webcamTexture.Play() has been called.

Get Video Clip Dimensions

We need to get the video resolution from the Video Player object when we’re not using a webcam. We can get the height and width by accessing the VideoPlayer component.

Replace videoTexture

The dimensions for videoTexture can’t be updated once it’s been created. We’ll need to replace it with a new RenderTexture that uses the new videoHeight and videoWidth values. We also need to set the new videoTexture as the targetTexture for the Video Player object.

Update VideoScreen

Next, we need to apply the new videoTexture to the VideoScreen object. We’ll also resize and reposition the VideoScreen to account for any changes in the videoTexture dimensions.

Adjust Main Camera

We’ll also resize and reposition the Main Camera object to account for changes to the VideoScreen.

Update ProcessOutput() Method

Finally, we need to modify how we calculate the key point locations.

Update Scaling Calculations

We’ll account for different aspect ratios by first determining whether the height or width of videoTexture is larger. We then use that information to calculate the scale and unsqueezeScale values.

Update Key Point Calculations

We should only scale the xPos value by the unsqueezeScale if the width larger than the height. Remove that part from the initial xPos calculation. We also need to move the if (useWebcam) statement.

We’ll scale either xPos or yPos by unsqueezeScale depending on whether the width or height value for videoTexture is larger. Place the if (useWebcam) statement after this calculation.

Try It Out

Now we can test the modifications on some vertical videos from Pexels.

  1. Woman Doing a Jump Rope Exercise
  2. Man Dancing Hip-Hop

Summary

Now we can use video sources with different aspect ratios without needing to manually change any parameters.

GitHub Repository - Version 1