Barracuda PoseNet Tutorial Pt. 2.5 (Outdated)
Version 2: Part 1
Last Updated: Nov 30, 2020
Previous: Part 2
Introduction
Examining the preprocessed input images can be a useful step when debugging. We can see how much the source image gets squished when resized to a square aspect ratio. The model can have a hard time accurately determining key point locations if the individual gets squished too much. We can also get a visual sense of how the preprocessing operations change the pixel values.
Make a New Screen
We can make a second screen to see what the preprocessed images look like before they get fed into the model.
Make a Quad
In the Hierarchy tab, make an new Quad and name it InputScreen.
Set the Scale
We’ll set both the X and Y scale values to 360 to match our current input resolution for the model.
Set the Position
We also need to set the X and Y position values to 180. Set the Z position value to -1 so that it’s in front of the VideoScreen.
Make a RenderTexture
Make another RenderTexture and name it input_texture.
Set the Size
Set the Size to 360 x 360.
Apply the input_texture
Drag and drop the input_texture onto the InputScreen in the Scene tab.
Make the Shader Unlit
Set the Shader for the InputScreen to Unlit/Texture just like the VideoScreen.
Update the PoseNet Script
Next, we need to create a few new public variables in the PoseNet script.
Add displayInput variable
Create a new public bool variable called displayInput. This will add a checkbox in the Inpsector tab that we can use to turn the InputScreen on and off.
Add inputScreen variable
Create a new public GameObject variable called inputScreen. We need to access the InputScreen object to activate and deactivate it.
Add inputTexture variable
Create a new public RenderTexture variable called inputTexture. We’ll assign the input_texture asset to this variable in the Unity Editor.
Modify the Update() Method
We can use the Graphics.Blit() method to copy the processedImage data to the inputTexture variable. We’ll use the inputScreen.SetActive() method to activate and deactivate the InputScreen.
Assign Inspector Variables
With the PoseEstimator selected in the Hierarchy tab, drag and drop the InputScreen and input_texture into their respective variables in the Inspector tab.
Test the New Screen
Make sure the Display Input checkbox is ticked in the Inspector tab. It will be easier to see the changes to the preprocessed images if we use a full color video. We can set the Video Clip for the Video Player to the pexels_woman_dancing file that we downloaded in Part 1.
Rescale Pixel Values
The current result may not be the most accurate representation as pixel values are still in the the range of [0, 255] instead of Unity’s [0.0, 1.0]. We can rescale the pixel values by adding another function to our PoseNetShader.
Update PoseNetShader
We’ll create another function called ScaleInputImage() that divides the RGB channel values by 255.0.
Note: Rescaling the pixel values will have a small impact on performance. Make sure to untick the Display Input checkbox when you’re not examining the input image.
Create ScaleInputImage Method
We need to create a new method in the PoseNet script to execute the function just like for the PreprocessResNet function.
Call ScaleInputImage Method
Finally, we’ll call the new method in the if (displayInput) statement.
Result
Summary
We now have a little screen that we can use to view the processed input image before it gets fed to the model.