Version 2: Part 1
Last Updated: Nov 24, 2020
- Create a New Project
- Import Video Files
- Create the Video Player
- Create the Video Screen
- Test the Video Player
This tutorial series provides step-by-step instructions for how to perform human pose estimation in Unity with the Barracuda inference library. We’ll be using a pretrained PoseNet model to estimate the 2D locations of key points on an individual’s body.
This post demonstrates how to play and view videos inside Unity. We’ll later perform pose estimation on individual frames while the video is playing. We can gauge the model’s accuracy by comparing the estimated key point locations to the source video.
I recommend checking the following prerequisites if you want to follow along on your own computer.
This tutorial assumes that you have Unity installed. You can get acquainted with Unity by clicking on one of the tutorials listed below.
How to Make a Game - Unity Beginner Tutorial
Note: You can download the exact version of Unity used for this tutorial by clicking the link below.
We’ll be performing inference on the GPU for this series. If possible, use a graphics card from a recent generation.
Create a New Project
First, we need to create a new Unity project. We’ll select the 2D template since the PoseNet model only estimates 2D poses.
Import Video Files
We’ll be using these two videos available on Pexels, a free stock photos & videos site. The first one is easier for the PoseNet model. The second has some more challenging sections. Download the videos in
Full HD resolution.
Note: Renamed to
Note: Renamed to
Create the Videos Folder
Assets window, right-click an empty space, select the
Create option, and click
Folder. Name the folder
Videos folder to open it.
Add Video Files
Drag and drop the two video files from the file explorer into the
Create the Video Player
Hierarchy tab, right-click an empty area, select the
Video section, and click
Video Player. This will create a new
Video Player. The default name works well enough so we’ll leave it as is.
Set Video Clip
Video Player object in the
Hierarchy tab. Then, drag and drop the
pexels_boardslides file into the
Video Clip parameter in the
Make the Video Loop
Loop checkbox in the
Inspector tab to make the video repeat when the project is running.
Create the Video Screen
We need to make a “screen” in Unity to watch the video. We’ll use a
Render Texture to store the data for the current frame and attach it to the surface of a
Create a Render Texture
Create a new folder in the
Assets window and name it
Open the folder and right-click an empty space. Select
Render Texture in the
Create submenu and name it
Resize the Render Texture
video_texture asset and set the
1920 x 1080 in the
Inspector tab. This will match the
video_texture to the resolution of our videos.
Assign the Render Texture
With the resolution set, select the
Video Player object in the
Hierarchy tab again. Drag and drop the
video_texture object into the
Target Texture parameter in the
Create the Screen GameObject
Now, we need to create the screen itself. We’ll use a
Quad object for the screen. Right click an empty space in the
Hierarchy tab, select the
3D Object section and click
Quad. We can just name it
Resize the Screen
VideoScreen object selected, we need to adjust the
Scale parameter in the
Inspector tab. Set the
X value to 1920 and the
Y value to 1080. Leave the
Z value at 1.
Set the Screen Position
Next, we’ll move
VideoScreen to make things easier when processing output from the model. We want the bottom left corner to be at the origin. Set the
X value for
Position to half the
X value for the
Scale parameter. Do the same for the
Y value. The new
Position values should be
X: 960 Y: 540 Z: 0.
Reset the Scene Perspective
We should center our perspective on the
VideoScreen. We can do so by selecting the
VideoScreen object and pressing the
F key on our keyboard. You can zoom back in by scrolling up with your mouse wheel.
Apply the Render Texture to the Screen
Drag and drop the
video_texture asset onto the
VideoScreen in the
Scene tab. The
VideoScreen object should turn completely black.
Make Video Screen Unlit
VideoScreen object selected, click the
Shader dropdown in the
Inspector tab. Select the Unlit option and click
Texture. This removes the need for a separate light source. The videos would look extremely dim with the
Before playing the video, we need to reposition and resize the
Main Camera object.
Set Camera Position
Main Camera object in the
Hierarchy tab and set the
Position to same
X: 960 Y: 540 as the
VideoScreen object. Next, we need to set the
Z value for the
Position to the opposite of the
Resize the Camera
Finally, we need to adjust the Size parameter to 540 in the
Test the Video Player
Now we can finally click the play button and watch the video.
We now have a video player that we can use to feed input to the PoseNet model. The next post covers how to prepare input for the model on the GPU.