Barracuda PoseNet Tutorial 2nd Edition Pt. 1

unity
barracuda
tutorial
This tutorial series provides step-by-step instructions for how to perform human pose estimation in Unity with the Barracuda inference library.
Author

Christian Mills

Published

July 20, 2021

Introduction

This tutorial series provides step-by-step instructions for how to perform human pose estimation in Unity with the Barracuda inference library. We will be using a pretrained PoseNet model to estimate the 2D locations of key points on the bodies of one or more individuals in a video frame. We will then use the output from the model to control the locations of GameObjects in a scene.

Single Pose Demo

Multi-Pose Demo

Overview

This post covers the process for installing the Barracuda package as well as importing the required video files and PoseNet models into the project.

Prerequisites

The following prerequisites are required to complete this tutorial.

Unity

This tutorial assumes that Unity is already installed on the system. We will be using Unity 2020, and the exact version can be downloaded from the link below.

Anyone who has never used Unity before can become acquainted with the basics by following the tutorial below. It will walk through the installation process all the way to making an Angry Birds clone.

Hardware

There appears to be a known issue with playing videos in Unity on AMD GPUs. Therefore, an Intel or Nvidia GPU is recommended. However, webcams seem to work fine on AMD GPUs.

Create a New Project

First, we need to create a new Unity project. We can use the default 3D template.

Note: There appears to currently be an issues with the 2D template where Barracuda does not work when the project is built.

Install Barracuda Package

We will start by installing the Barracuda package. This will allow us to import the PoseNet models into the project. Open the Window menu at the top of the Unity Editor and select Package Manager.

There might be a message in the console indicating that there is a new version of the Visual Studio Editor package.

Feel free to update the package by selecting it in the Package Manager and clicking the Update button.

We will be using version 2.1.0 of the Barracuda package. Unity has this version marked as preview, so we will need to enable preview packages to install it. Click the small gear icon and select the Advanced Project Settings option.

Tick the Enable Preview Packages checkbox so that we can install the latest version of Barracuda.

A popup window will appear, warning us that preview packages might not be ready for production. However, the latest version of Barracuda contains bug fixes that are not present in the Verified version, so click I understand in the popup window.

Even though there is a verified version of Barracuda, it is not available in the package manager by default. We need to either install a package that has it as a dependency (e.g. ML Agents) or add it directly with a git URL. Click on the + icon in the upper-left corner and select Add package from git URL....

Enter com.unity.barracuda into the search box and click Add. This will install the latest Verified version of the package. Unfortunately, there is a bug with this version that causes an error when performing inference on the CPU. This is resolved in later versions.

Note: The version of Barracuda that we will be using in this tutorial is not available through the package manager in Unity 2021 at the time of writing. You will need to manually update the value for "com.unity.barracuda" in the Project_Folder/Packages/manifest.json file from "1.0.4" to "2.1.0-preview" as shown below. The package will be marked as Experimental in the editor.

"dependencies": {
    "com.unity.barracuda": "2.1.0-preview",
    "com.unity.collab-proxy": "1.5.7",
    "com.unity.ide.rider": "2.0.7",
    "com.unity.ide.visualstudio": "2.0.11",
    "com.unity.ide.vscode": "1.2.3",

We can view more recent versions of the package by clicking See other versions.

Scroll all the way up to version 2.1.0-preview and click the Update to 2.1.0-preview button in the bottom-right corner.

During the installation process a popup window will appear indicating that the version of the Burst compiler has changed. Click OK to close the window. Once the installation process has finished, close Unity and then reopen the project.

Unity seems to be concerned that anyone who jumps through the multiple hoops to install a preview package might forget that they are indeed using a preview package. To eliminate this possibility, they have added a reminder at the top of the editor that can not be permanently removed.

Import Video Files

We will be using these two videos available on Pexels, a free stock photos & videos site. The first one is for testing single pose estimation and only has one person in frame at a time. The second video is meant for testing multipose estimation and has several individuals in frame at varying distances from the camera. Download the videos in Full HD resolution.

  1. Two Young Men Doing a Boardslide Over a Railing

    Note: Renamed to pexels_boardslides

  2. Teens Riding Skateboard Doing Grind Rail

    Note: Renamed to pexels_teens_riding_skateboard_doing_grind_rail

Add Files to Assets

In the Assets section, right-click an empty space, select the Create option, and click Folder. Name the folder Videos. Double-click the Videos folder to open it.

Drag and drop the two video files from the File Explorer into the Videos folder.

Import ONNX Models

We will cover how to use two different versions of the PoseNet model. The MobileNet version is optimized to run efficiently on CPUs at the cost of some accuracy. The ResNet50 version is noticeably more accurate, but is more computationally demanding.

Download Files

The model files used in this tutorial series can be downloaded from the links below.

Add Files to Assets

Back in the Assets section, create a new folder called Models. Drag and drop the ONNX files from the File Explorer into the Models folder.

Summary

That takes care of the preliminary setup for the project. The next post will cover how to play and view videos inside Unity from both video files and a webcam.

Next: Part 2

Project Resources: GitHub Repository

About Me:

I’m Christian Mills, a deep learning consultant specializing in practical AI implementations. I help clients leverage cutting-edge AI technologies to solve real-world problems.

Interested in working together? Fill out my Quick AI Project Assessment form or learn more about me.