A Step-by-Step Guide to Object Detection in Unity with IceVision and OpenVINO Pt. 3
- Overview
- Create New Project
- Import Assets
- Allow Unsafe Code
- Create Processing Shader
- Create Object Detector Script
- Set up Unity Scene
- Test in Editor
- Summary
Tutorial Links
- Part 1: Train a YOLOX model using IceVision and export it to OpenVINO.
- Part 2: Create a dynamic link library (DLL) file in Visual Studio to perform object detection with a YOLOX model using OpenVINO.
- Part 3: Perform object detection in a Unity project with OpenVINO.
- GitHub Repository
Overview
In part 3 of this tutorial series, we will integrate the trained YOLOX model into a Unity project to perform real-time object detection. We will begin by creating a new Unity project and importing the necessary assets. Then, we will allow unsafe code in our project to share input data with the DLL file. Next, we will create a processing shader and an object detector script to handle object detection in our Unity project. Lastly, we will set up the Unity scene and test our object detection model in the editor. By the end of this post, you will know how to add object detection functionality to a Unity project.
Important: This post assumes you already have Unity Hub on your system. Check out this section from a previous tutorial if this is not the case (link).
Create New Project
Open the Unity Hub and click New Project.
Select the target editor version from the Editor Version dropdown menu. We’ll use Unity 2022 for this post, but the current LTS release should also work fine.
Select the 2D Core
template.
Pick a name for the project and a location for the project folder.
Finally, click Create Project
in the lower right-hand corner.
Import Assets
Once the project loads, we’ll store the DLL files from part 2 in a new folder called Plugins
. Right-click a space in the Assets section and select Create → Folder
from the popup menu.
The DLL targets 64-bit x86 architectures, so we need to place the DLL files in a subfolder named x86_64
.
Note: You can place the
Plugins
folder inside another folder if needed.
Copy all the DLL files and the plugins.xml
file into the Assets/Plugins/x86_64
folder. We then need to close and reopen the project for Unity to load the plugin files.
Back in the Unity Editor, create a new folder called Colormaps
to store the JSON file from part 1.
We place any test images into a new folder called Images
.
Next, we’ll create a folder to store the OpenVINO IR models. We need to place the XML and BIN files for the IR models in a StreamingAssets folder to include them in project builds. Create a new folder named StreamingAssets
. We’ll place files for each model in a separate folder and put those in a new subfolder called OpenVINOModels
to keep things organized.
The plugins.xml file included with the DLL files contains locations for the DLL files needed for using different types of devices.
plugins.xml
content:
ie>
<plugins>
<plugin name="AUTO" location="openvino_auto_plugin.dll">
<properties>
<property key="MULTI_WORK_MODE_AS_AUTO" value="YES"/>
<properties>
</plugin>
</plugin name="BATCH" location="openvino_auto_batch_plugin.dll">
<plugin>
</plugin name="CPU" location="openvino_intel_cpu_plugin.dll">
<plugin>
</plugin name="GNA" location="openvino_intel_gna_plugin.dll">
<plugin>
</plugin name="GPU" location="openvino_intel_gpu_plugin.dll">
<plugin>
</plugin name="HETERO" location="openvino_hetero_plugin.dll">
<plugin>
</plugin name="MULTI" location="openvino_auto_plugin.dll">
<plugin>
</plugin name="MYRIAD" location="openvino_intel_myriad_plugin.dll">
<plugin>
</plugin name="HDDL" location="openvino_intel_hddl_plugin.dll">
<plugin>
</plugin name="VPUX" location="openvino_intel_vpux_plugin.dll">
<plugin>
</plugins>
</ie> </
It needs to be in the same folder as the DLL files for the plugin to work. However, Unity does not include XML files in the Plugins folder when building the project. We need to store a copy of the plugins.xml file in the StreamingAssets
folder and then copy it back to the Plugins/x86_64
folder when first running the built project. We can handle both steps automatically in code.
Allow Unsafe Code
Rather than copying the input image from Unity to the OpenVINO plugin, we’ll pass a pointer to the pixel data. First, we need to allow unsafe code for the Unity project. Select Edit → Project Settings...
from the top menu.
Open the Player → Other Settings
dropdown and scroll down to the Allow 'unsafe' Code
checkbox. Enable the setting and close the Project Settings window.
Now we can start coding.
Create Processing Shader
The input image gets flipped upside down when we send it to the plugin. We can pre-flip the image in a Compute Shader. We’ll add the Compute Shader in a new folder called Shaders
. Right-click a space in the Shaders
folder and select Create → Shader → Compute Shader
.
Name the Compute Shader ProcessingShader
and open it in the code editor.
Default Compute Shader Code
// Each #kernel tells which function to compile; you can have many kernels
#pragma kernel CSMain
// Create a RenderTexture with enableRandomWrite flag and set it
// with cs.SetTexture
<float4> Result;
RWTexture2D
[numthreads(8,8,1)]
void CSMain (uint3 id : SV_DispatchThreadID)
{
// TODO: insert actual code here!
[id.xy] = float4(id.x & id.y, (id.x & 15)/15.0, (id.y & 15)/15.0, 0.0);
Result}
We need to add a new Texture2D
variable to store the pixel data for the input image. We’ll remove the default method and create a new one called FlipXAxis
. Replace the default method name in the #pragma kernel
line at the top.
We need the input image height for the flip operation, which we can access with the Texture2D::GetDimensions function.
// Each #kernel tells which function to compile; you can have many kernels
#pragma kernel FlipXAxis
// The pixel data for the input image
<float4> InputImage;
Texture2D// The pixel data for the processed image
<float4> Result;
RWTexture2D
// Flip the image around the x-axis
[numthreads(8, 8, 1)]
void FlipXAxis(uint3 id : SV_DispatchThreadID)
{
// Stores the InputImage width
uint width;
// Stores the InputImage height
uint height;
// Get the dimensions of the InputImage
.GetDimensions(width, height);
InputImage
// Update the y value for the pixel coordinates
= int2(id.x, height - id.y);
int2 coords [id.xy] = float4(InputImage[coords].x, InputImage[coords].y, InputImage[coords].z, 1.0f);
Result}
Create Object Detector Script
We’ll store the C# script that interacts with the OpenVINO plugin in a new Scripts
folder. Right-click a space inside it and select Create → C# Script
.
Name the script ObjectDetector
and open it in the code editor.
Default script code
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
public class ObjectDetector : MonoBehaviour
{
// Start is called before the first frame update
void Start()
{
}
// Update is called once per frame
void Update()
{
}
}
Add required namespaces
- System: Contains fundamental classes and base classes that define commonly-used value and reference data types, events and event handlers, interfaces, attributes, and processing exceptions.
- UnityEngine.UI: Provides access to UI elements.
- UnityEngine.Rendering: Provides access to the elements of the rendering pipeline.
- System.Runtime.InteropServices: Provides a wide variety of members that support COM interop and platform invoke services.
- System.IO: Allows reading and writing to files and data streams.
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.Rendering;
using System;
using UnityEngine.UI;
using System.Runtime.InteropServices;
using System.IO;
Add code to copy plugins.xml
file to StreamingAssets
folder
Unity provides an InitializeOnLoad
attribute to run code in the Unity Editor without requiring action from the user. This attribute requires the UnityEditor
namespace. We can only use this while in the Editor, so we need to wrap the code in Conditional compilation preprocessor directives. We’ll place this code right below the namespaces.
#if UNITY_EDITOR
using UnityEditor;
[InitializeOnLoad]
public class Startup
{
static Startup()
{
// Get all files named "plugins.xml"
string[] files = Directory.GetFiles("./Assets/", "plugins.xml", SearchOption.AllDirectories);
// Iterate through each found file
foreach (string file in files)
{
// Check if the file is in the "x86_64" folder
if (file.Contains("x86_64"))
{
// Define file path for StreamingAssets folder
string targetPath = $"{Application.streamingAssetsPath}/plugins.xml";
// Print the source file path
.Log(file);
Debug// Only copy the file to the StreamingAssets folder if it is not already present
if (!File.Exists(targetPath)) File.Copy(file, targetPath);
}
}
}
}
#endif
We use the UNITY_EDITOR
scripting symbol to check whether we are in the Unity Editor. We are in the Editor, so it returns true, and the code executes.
If we check if we are not in the Unity Editor, it returns false, and the code block does not execute.
We can verify the code works by saving the script and going to the StreamingAssets
folder in the Editor. The plugins.xml file should be present.
Define public variables
We’ll add the required public variables above the Start method. We will be able to access these variables in the Inspector tab. We can add Header attributes to organize the public variables in the Inspector tab and use Tooltip attributes to provide information about variables.
Define scene object variables
First, we need a variable to access the screen object that displays either a test image or webcam input. We may or may not want to mirror the screen based on whether a webcam is facing the user.
[Header("Scene Objects")]
[Tooltip("The Screen object for the scene")]
public Transform screen;
[Tooltip("Mirror the in-game screen.")]
public bool mirrorScreen = true;
Define data processing variables
Next, we’ll define the variables for processing model input. We can set the default target input resolution to 224
and use it to scale the source resolution while maintaining the original aspect ratio.
We’ll also add a public ComputeShader
variable to access the ProcessingShader
we made earlier.
We need to download the pixel data for the input image from the GPU to the CPU before passing it to the plugin. This step can cause a significant performance bottleneck, so we’ll add the option to read the model output asynchronously at the cost of a few frames of latency. This latency might cause the bounding box to trail slightly behind a fast-moving object on the screen. The effect should be minimal, provided the frame rate is high enough.
[Header("Data Processing")]
[Tooltip("The target minimum model input dimensions")]
public int targetDim = 224;
[Tooltip("The compute shader for GPU processing")]
public ComputeShader processingShader;
[Tooltip("Asynchronously download input image from the GPU to the CPU.")]
public bool useAsyncGPUReadback = true;
Define output processing variables
We pass in the JSON file containing the class labels as a TextAsset.
[Header("Output Processing")]
[Tooltip("A json file containing the colormaps for object classes")]
public TextAsset colormapFile;
[Tooltip("Minimum confidence score for keeping detected objects")]
[Range(0,1f)]
public float minConfidence = 0.5f;
Define variables for debugging
Next, we’ll add a Boolean variable to toggle printing debug messages to the console.
[Header("Debugging")]
[Tooltip("Print debugging messages to the console")]
public bool printDebugMessages = true;
Define webcam variables
We need to specify a desired resolution and framerate when using a webcam as input.
[Header("Webcam")]
[Tooltip("Use a webcam as input")]
public bool useWebcam = false;
[Tooltip("The requested webcam dimensions")]
public Vector2Int webcamDims = new Vector2Int(1280, 720);
[Tooltip("The requested webcam framerate")]
[Range(0, 60)]
public int webcamFPS = 60;
Define variables for user interface
We’ll make a simple GUI that displays the predicted class, the current framerate, and controls for selecting webcam devices, models, and compute devices.
[Header("GUI")]
[Tooltip("Display predicted class")]
public bool displayBoundingBoxes = true;
[Tooltip("Display number of detected objects")]
public bool displayProposalCount = true;
[Tooltip("Display fps")]
public bool displayFPS = true;
[Tooltip("The on-screen text color")]
public Color textColor = Color.red;
[Tooltip("The scale value for the on-screen font size")]
[Range(0, 99)]
public int fontScale = 50;
[Tooltip("The number of seconds to wait between refreshing the fps value")]
[Range(0.01f, 1.0f)]
public float fpsRefreshRate = 0.1f;
[Tooltip("The toggle for using a webcam as the input source")]
public Toggle useWebcamToggle;
[Tooltip("The dropdown menu that lists available webcam devices")]
public Dropdown webcamDropdown;
[Tooltip("The dropdown menu that lists available OpenVINO models")]
public Dropdown modelDropdown;
[Tooltip("The dropdown menu that lists available OpenVINO devices")]
public Dropdown deviceDropdown;
Define public variables for the OpenVINO plugin
[Header("OpenVINO")]
[Tooltip("The name of the openvino models folder")]
public string openvinoModelsDir = "OpenVINOModels";
Define private variables
We’ll add the required private variables right below the public variables.
Define private webcam variables
We’ll keep a list of available webcam devices so users can switch between them. Unity renders webcam input to a WebcamTexture.
// List of available webcam devices
private WebCamDevice[] webcamDevices;
// Live video input from a webcam
private WebCamTexture webcamTexture;
// The name of the current webcam device
private string currentWebcam;
Define input variables
We’ll update the dimensions and content of the screen object based on the test image or webcam.
When using asynchronous GPU readback, we need one Texture that stores data on the GPU and one that stores data on the CPU.
// The test image dimensions
private Vector2Int imageDims;
// The test image texture
private Texture imageTexture;
// The current screen object dimensions
private Vector2Int screenDims;
// The model GPU input texture
private RenderTexture inputTextureGPU;
// The model CPU input texture
private Texture2D inputTextureCPU;
Define variable for tracking the current number of detected objects
// Stores the number of detected objects
private int numObjects;
Define variables for storing colormaps
We need to create a couple of classes to parse the JSON content.
// A class for parsing in colormaps from a JSON file
[System.Serializable]
class ColorMap { public string label; public float[] color; }
// A class for reading in a list of colormaps from a JSON file
[System.Serializable]
class ColorMapList { public List<ColorMap> items; }
// Stores a list of colormaps from a JSON file
private ColorMapList colormapList;
// A list of colors that map to class labels
private Color[] colors;
// A list of single pixel textures that map to class labels
private Texture2D[] colorTextures;
Define variables for tracking the framerate
We’ll define some variables to track the frame rate.
// The current frame rate value
private int fps = 0;
// Controls when the frame rate value updates
private float fpsTimer = 0f;
Define private variables for the OpenVINO plugin
// File paths for the available OpenVINO models
private List<string> modelPaths = new List<string>();
// Names of the available OpenVINO models
private List<string> modelNames = new List<string>();
// Names of the available OpenVINO devices
private List<string> openvinoDevices = new List<string>();
Define a struct for reading object information from the OpenVINO plugin
We need to create an Object
struct for Unity to match the one we defined for the OpenVINO code, along with an array of Object
structs that we’ll update with the PopulateObjectsArray()
function.
// Indicate that the members of the struct are laid out sequentially
[StructLayout(LayoutKind.Sequential)]
/// <summary>
/// Stores the information for a single object
/// </summary>
public struct Object
{
// The X coordinate for the top left bounding box corner
public float x0;
// The Y coordinate for the top left bounding box cornder
public float y0;
// The width of the bounding box
public float width;
// The height of the bounding box
public float height;
// The object class index for the detected object
public int label;
// The model confidence score for the object
public float prob;
public Object(float x0, float y0, float width, float height, int label, float prob)
{
this.x0 = x0;
this.y0 = y0;
this.width = width;
this.height = height;
this.label = label;
this.prob = prob;
}
}
// Stores information for the current list of detected objects
private Object[] objectInfoArray;
Import functions from the OpenVINO plugin
We pass the pointer to the input pixel data as an IntPtr.
// Name of the DLL file
const string dll = "OpenVINO_YOLOX_DLL";
[DllImport(dll)]
private static extern int GetDeviceCount();
[DllImport(dll)]
private static extern IntPtr GetDeviceName(int index);
[DllImport(dll)]
private static extern void SetConfidenceThreshold(float minConfidence);
[DllImport(dll)]
private static extern int LoadModel(string model, int index, int[] inputDims);
[DllImport(dll)]
private static extern int PerformInference(IntPtr inputData);
[DllImport(dll)]
private static extern void PopulateObjectsArray(IntPtr objects);
[DllImport(dll)]
private static extern void FreeResources();
Define Initialization Methods
We first need to define some methods to initialize webcams, the screen object, any GUI dropdown menus, and the in-game camera.
Define method to initialize a webcam device
/// <summary>
/// Initialize the selected webcam device
/// </summary>
/// <param name="deviceName">The name of the selected webcam device</param>
private void InitializeWebcam(string deviceName)
{
// Stop any webcams already playing
if (webcamTexture && webcamTexture.isPlaying) webcamTexture.Stop();
// Create a new WebCamTexture
= new WebCamTexture(deviceName, webcamDims.x, webcamDims.y, webcamFPS);
webcamTexture
// Start the webcam
.Play();
webcamTexture// Check if webcam is playing
= webcamTexture.isPlaying;
useWebcam // Update toggle value
.SetIsOnWithoutNotify(useWebcam);
useWebcamToggle
.Log(useWebcam ? "Webcam is playing" : "Webcam not playing, option disabled");
Debug}
Define method to initialize the in-scene screen object
/// <summary>
/// Resize and position an in-scene screen object
/// </summary>
private void InitializeScreen()
{
// Set the texture for the screen object
.gameObject.GetComponent<MeshRenderer>().material.mainTexture = useWebcam ? webcamTexture : imageTexture;
screen// Set the screen dimensions
= useWebcam ? new Vector2Int(webcamTexture.width, webcamTexture.height) : imageDims;
screenDims
// Flip the screen around the Y-Axis when using webcam
float yRotation = useWebcam && mirrorScreen ? 180f : 0f;
// Invert the scale value for the Z-Axis when using webcam
float zScale = useWebcam && mirrorScreen ? -1f : 1f;
// Set screen rotation
.rotation = Quaternion.Euler(0, yRotation, 0);
screen// Adjust the screen dimensions
.localScale = new Vector3(screenDims.x, screenDims.y, zScale);
screen
// Adjust the screen position
.position = new Vector3(screenDims.x / 2, screenDims.y / 2, 1);
screen}
Define method to get the available OpenVINO models
/// <summary>
/// Get the file paths for available OpenVION models
/// </summary>
private void GetOpenVINOModels()
{
// Get the paths for each model folder
foreach (string dir in System.IO.Directory.GetDirectories($"{Application.streamingAssetsPath}/{openvinoModelsDir}"))
{
string modelName = dir.Split('\\')[1];
.Add(modelName.Substring(0, modelName.Length));
modelNames
// Get the paths for the XML file for each model
foreach (string file in System.IO.Directory.GetFiles(dir))
{
if (file.EndsWith(".xml"))
{
.Add(file);
modelPaths}
}
}
}
Define method to get the names of available OpenVINO devices
/// <summary>
/// Get the names of the available OpenVINO devices
/// </summary>
private void GetOpenVINODevices()
{
// Get the number of available OpenVINO devices
int deviceCount = GetDeviceCount();
for (int i = 0; i < deviceCount; i++)
{
.Add(Marshal.PtrToStringAnsi(GetDeviceName(i)));
openvinoDevices}
}
Define method to initialize GUI dropdown menu options
/// <summary>
/// Initialize the GUI dropdown list
/// </summary>
private void InitializeDropdown()
{
// Create list of webcam device names
<string> webcamNames = new List<string>();
Listforeach(WebCamDevice device in webcamDevices) webcamNames.Add(device.name);
// Remove default dropdown options
.ClearOptions();
webcamDropdown// Add webcam device names to dropdown menu
.AddOptions(webcamNames);
webcamDropdown// Set the value for the dropdown to the current webcam device
.SetValueWithoutNotify(webcamNames.IndexOf(currentWebcam));
webcamDropdown
// Remove default dropdown options
.ClearOptions();
modelDropdown// Add OpenVINO model names to menu
.AddOptions(modelNames);
modelDropdown// Select the first option in the dropdown
.SetValueWithoutNotify(0);
modelDropdown
// Remove default dropdown options
.ClearOptions();
deviceDropdown// Add OpenVINO device names to menu
.AddOptions(openvinoDevices);
deviceDropdown// Select the first option in the dropdown
.SetValueWithoutNotify(0);
deviceDropdown}
Define method to initialize the in-scene camera object
/// <summary>
/// Resize and position the main camera based on an in-scene screen object
/// </summary>
/// <param name="screenDims">The dimensions of an in-scene screen object</param>
private void InitializeCamera(Vector2Int screenDims, string cameraName = "Main Camera")
{
// Get a reference to the Main Camera GameObject
= GameObject.Find(cameraName);
GameObject camera // Adjust the camera position to account for updates to the screenDims
.transform.position = new Vector3(screenDims.x / 2, screenDims.y / 2, -10f);
camera// Render objects with no perspective (i.e. 2D)
.GetComponent<Camera>().orthographic = true;
camera// Adjust the camera size to account for updates to the screenDims
.GetComponent<Camera>().orthographicSize = screenDims.y / 2;
camera}
Define method to update the selected OpenVINO model
/// <summary>
/// Update the selected OpenVINO model
/// </summary>
public void UpdateOpenVINOModel()
{
// Reset objectInfoArray
= new Object[0];
objectInfoArray
int[] inputDims = new int[] {
.width,
inputTextureCPU.height
inputTextureCPU};
.Log($"Selected Device: {openvinoDevices[deviceDropdown.value]}");
Debug
// Load the specified OpenVINO model
int return_msg = LoadModel(modelPaths[modelDropdown.value], deviceDropdown.value, inputDims);
SetConfidenceThreshold(minConfidence);
string[] return_messages = {
"Model loaded and reshaped successfully",
"Failed to load model",
"Failed to reshape model input",
};
.Log($"Updated input dims: {inputDims[0]} x {inputDims[1]}");
Debug.Log($"Return message: {return_messages[return_msg]}");
Debug}
Define Awake Method
We’ll implement the code to copy the plugins.xml file from the StreamingAssets
folder to the Plugins/x86_64
folder in the build folder in the Awake() method. The code should be inactive since we are in the Editor.
// Awake is called when the script instance is being loaded
private void Awake()
{
#if !UNITY_EDITOR
// Define the path for the plugins.xml file in the StreamingAssets folder
string sourcePath = $"{Application.streamingAssetsPath}/plugins.xml";
// Define the destination path for the plugins.xml file
string targetPath = $"{Application.dataPath}/Plugins/x86_64/plugins.xml";
// Only copy the file if it is not already present at the destination
if (!File.Exists(targetPath)) File.Copy(sourcePath, targetPath);
#endif
}
Define Start Method
The Start method is called once before the first frame update, so we’ll perform any required setup steps here.
// Start is called before the first frame update
void Start()
{
// Get the source image texture
= screen.gameObject.GetComponent<MeshRenderer>().material.mainTexture;
imageTexture // Get the source image dimensions as a Vector2Int
= new Vector2Int(imageTexture.width, imageTexture.height);
imageDims
// Initialize list of available webcam devices
= WebCamTexture.devices;
webcamDevices foreach (WebCamDevice device in webcamDevices) Debug.Log(device.name);
= webcamDevices[0].name;
currentWebcam = webcamDevices.Length > 0 ? useWebcam : false;
useWebcam // Initialize webcam
if (useWebcam) InitializeWebcam(currentWebcam);
// Resize and position the screen object using the source image dimensions
InitializeScreen();
// Resize and position the main camera using the source image dimensions
InitializeCamera(screenDims);
// Initialize list of color maps from JSON file
= JsonUtility.FromJson<ColorMapList>(colormapFile.text);
colormapList // Initialize the list of colors
= new Color[colormapList.items.Count];
colors // Initialize the list of color textures
= new Texture2D[colormapList.items.Count];
colorTextures
// Populate the color and color texture arrays
for (int i = 0; i < colors.Length; i++)
{
// Create a new color object
[i] = new Color(
colors.items[i].color[0],
colormapList.items[i].color[1],
colormapList.items[i].color[2]);
colormapList// Create a single-pixel texture
[i] = new Texture2D(1, 1);
colorTextures[i].SetPixel(0, 0, colors[i]);
colorTextures[i].Apply();
colorTextures
}
// Get the file paths for available OpenVINO models
GetOpenVINOModels();
// Get the names of available OpenVINO devices
GetOpenVINODevices();
// Initialize the webcam dropdown list
InitializeDropdown();
}
Define Processing Methods
Next, we need to define methods to process images using the Compute Shader, calculate the input resolution, handle asynchronous GPU readback, and scale the bounding box information.
Define method to process images using a compute shader
/// <summary>
/// Process the provided image using the specified function on the GPU
/// </summary>
/// <param name="image">The target image RenderTexture</param>
/// <param name="computeShader">The target ComputerShader</param>
/// <param name="functionName">The target ComputeShader function</param>
/// <returns></returns>
private void ProcessImageGPU(RenderTexture image, ComputeShader computeShader, string functionName)
{
// Specify the number of threads on the GPU
int numthreads = 8;
// Get the index for the specified function in the ComputeShader
int kernelHandle = computeShader.FindKernel(functionName);
// Define a temporary HDR RenderTexture
= new RenderTexture(image.width, image.height, 24, RenderTextureFormat.ARGBHalf);
RenderTexture result // Enable random write access
.enableRandomWrite = true;
result// Create the HDR RenderTexture
.Create();
result
// Set the value for the Result variable in the ComputeShader
.SetTexture(kernelHandle, "Result", result);
computeShader// Set the value for the InputImage variable in the ComputeShader
.SetTexture(kernelHandle, "InputImage", image);
computeShader
// Execute the ComputeShader
.Dispatch(kernelHandle, result.width / numthreads, result.height / numthreads, 1);
computeShader
// Copy the result into the source RenderTexture
.Blit(result, image);
Graphics
// Release RenderTexture
.Release();
result}
Define method to calculate input resolution
/// <summary>
/// Scale the source image resolution to the target input dimensions
/// while maintaing the source aspect ratio.
/// </summary>
/// <param name="imageDims"></param>
/// <param name="targetDims"></param>
/// <returns></returns>
private Vector2Int CalculateInputDims(Vector2Int imageDims, int targetDim)
{
= new Vector2Int();
Vector2Int inputDims
// Calculate the input dimensions using the target minimum dimension
if (imageDims.x >= imageDims.y)
{
[0] = (int)(imageDims.x / ((float)imageDims.y / (float)targetDim));
inputDims[1] = targetDim;
inputDims}
else
{
[0] = targetDim;
inputDims[1] = (int)(imageDims.y / ((float)imageDims.x / (float)targetDim));
inputDims}
return inputDims;
}
Define method to handle asynchronous GPU readback
/// <summary>
/// Called once AsyncGPUReadback has been completed
/// </summary>
/// <param name="request"></param>
private void OnCompleteReadback(AsyncGPUReadbackRequest request)
{
if (request.hasError)
{
.Log("GPU readback error detected.");
Debugreturn;
}
// Make sure the Texture2D is not null
if (inputTextureCPU)
{
// Fill Texture2D with raw data from the AsyncGPUReadbackRequest
.LoadRawTextureData(request.GetData<uint>());
inputTextureCPU// Apply changes to Textur2D
.Apply();
inputTextureCPU}
}
Define method to send the input texture data to the plugin
/// <summary>
/// Pin memory for the input data and pass a reference to the plugin for inference
/// </summary>
/// <param name="texture">The input texture</param>
/// <returns></returns>
public unsafe int UploadTexture(Texture2D texture)
{
//Pin Memory
fixed (byte* p = texture.GetRawTextureData())
{
// Perform inference and get the number of detected objects
= PerformInference((IntPtr)p);
numObjects }
// Initialize the array
= new Object[numObjects];
objectInfoArray
// Pin memory
fixed (Object* o = objectInfoArray)
{
// Get the detected objects
PopulateObjectsArray((IntPtr)o);
}
return numObjects;
}
Define method to scale bounding boxes to the display resolution
/// <summary>
/// Scale the latest bounding boxes to the display resolution
/// </summary>
public void ScaleBoundingBoxes()
{
// Process new detected objects
for (int i = 0; i < objectInfoArray.Length; i++)
{
// The smallest dimension of the screen
float minScreenDim = Mathf.Min(screen.transform.localScale.x, screen.transform.localScale.y);
// The smallest input dimension
int minInputDim = Mathf.Min(inputTextureCPU.width, inputTextureCPU.height);
// Calculate the scale value between the in-game screen and input dimensions
float minImgScale = minScreenDim / minInputDim;
// Calculate the scale value between the in-game screen and display
float displayScale = Screen.height / screen.transform.localScale.y;
// Scale bounding box to in-game screen resolution and flip the bbox coordinates vertically
float x0 = objectInfoArray[i].x0 * minImgScale;
float y0 = (inputTextureCPU.height - objectInfoArray[i].y0) * minImgScale;
float width = objectInfoArray[i].width * minImgScale;
float height = objectInfoArray[i].height * minImgScale;
// Mirror bounding box across screen
if (mirrorScreen && useWebcam) x0 = screen.transform.localScale.x - x0 - width;
// Scale bounding boxes to display resolution
[i].x0 = x0 * displayScale;
objectInfoArray[i].y0 = y0 * displayScale;
objectInfoArray[i].width = width * displayScale;
objectInfoArray[i].height = height * displayScale;
objectInfoArray
// Offset the bounding box coordinates based on the difference between the in-game screen and display
[i].x0 += (Screen.width - screen.transform.localScale.x * displayScale) / 2;
objectInfoArray}
}
Define Update method
We’ll place anything we want to run every frame in the Update method.
// Update is called once per frame
void Update()
{
= webcamDevices.Length > 0 ? useWebcam : false;
useWebcam if (useWebcam)
{
// Initialize webcam if it is not already playing
if (!webcamTexture || !webcamTexture.isPlaying) InitializeWebcam(currentWebcam);
// Skip the rest of the method if the webcam is not initialized
if (webcamTexture.width <= 16) return;
// Make sure screen dimensions match webcam resolution when using webcam
if (screenDims.x != webcamTexture.width)
{
// Resize and position the screen object using the source image dimensions
InitializeScreen();
// Resize and position the main camera using the source image dimensions
InitializeCamera(screenDims);
}
}
else if (webcamTexture && webcamTexture.isPlaying)
{
// Stop the current webcam
.Stop();
webcamTexture
// Resize and position the screen object using the source image dimensions
InitializeScreen();
// Resize and position the main camera using the source image dimensions
InitializeCamera(screenDims);
}
// Scale the source image resolution
= CalculateInputDims(screenDims, targetDim);
Vector2Int inputDims if (printDebugMessages) Debug.Log($"Input Dims: {inputDims.x} x {inputDims.y}");
// Initialize the input texture with the calculated input dimensions
= RenderTexture.GetTemporary(inputDims.x, inputDims.y, 24, RenderTextureFormat.ARGBHalf);
inputTextureGPU
if (!inputTextureCPU || inputTextureCPU.width != inputTextureGPU.width)
{
= new Texture2D(inputDims.x, inputDims.y, TextureFormat.RGBA32, false);
inputTextureCPU // Update the selected OpenVINO model
UpdateOpenVINOModel();
}
// Copy the source texture into model input texture
.Blit((useWebcam ? webcamTexture : imageTexture), inputTextureGPU);
Graphics
// Flip image before sending to DLL
ProcessImageGPU(inputTextureGPU, processingShader, "FlipXAxis");
// Download pixel data from GPU to CPU
if (useAsyncGPUReadback)
{
.Request(inputTextureGPU, 0, TextureFormat.RGBA32, OnCompleteReadback);
AsyncGPUReadback}
else
{
.active = inputTextureGPU;
RenderTexture.ReadPixels(new Rect(0, 0, inputTextureGPU.width, inputTextureGPU.height), 0, 0);
inputTextureCPU.Apply();
inputTextureCPU}
// Send reference to inputData to DLL
= UploadTexture(inputTextureCPU);
numObjects if (printDebugMessages) Debug.Log($"Detected {numObjects} objects");
// Scale bounding boxes
ScaleBoundingBoxes();
// Release the input texture
.ReleaseTemporary(inputTextureGPU);
RenderTexture}
Define GUI Methods
We need some methods to handle user interactions with the GUI and display the bounding boxes and current framerate.
Define method to update webcam usage from GUI
/// <summary>
/// This method is called when the value for the webcam toggle changes
/// </summary>
/// <param name="useWebcam"></param>
public void UpdateWebcamToggle(bool useWebcam)
{
this.useWebcam = useWebcam;
}
Define method to update webcam device from GUI
/// <summary>
/// The method is called when the selected value for the webcam dropdown changes
/// </summary>
public void UpdateWebcamDevice()
{
= webcamDevices[webcamDropdown.value].name;
currentWebcam .Log($"Selected Webcam: {currentWebcam}");
Debug// Initialize webcam if it is not already playing
if (useWebcam) InitializeWebcam(currentWebcam);
// Resize and position the screen object using the source image dimensions
InitializeScreen();
// Resize and position the main camera using the source image dimensions
InitializeCamera(screenDims);
}
Define method to update the minimum confidence value
/// <summary>
/// Update the minimum confidence score for keeping bounding box proposals
/// </summary>
/// <param name="slider"></param>
public void UpdateConfidenceThreshold(Slider slider)
{
= slider.value;
minConfidence SetConfidenceThreshold(minConfidence);
}
Define OnGUI method
We’ll display the predicted bounding boxes and current frame rate in the OnGUI method.
// OnGUI is called for rendering and handling GUI events.
public void OnGUI()
{
// Initialize a rectangle for label text
= new Rect();
Rect labelRect // Initialize a rectangle for bounding boxes
= new Rect();
Rect boxRect
= new GUIStyle
GUIStyle labelStyle {
= (int)(Screen.width * 11e-3)
fontSize };
.alignment = TextAnchor.MiddleLeft;
labelStyle
foreach (Object objectInfo in objectInfoArray)
{
if (!displayBoundingBoxes) break;
// Skip object if label index is out of bounds
if (objectInfo.label > colors.Length - 1) continue;
// Get color for current class index
= colors[objectInfo.label];
Color color // Get label for current class index
string name = colormapList.items[objectInfo.label].label;
// Set bounding box coordinates
.x = objectInfo.x0;
boxRect.y = Screen.height - objectInfo.y0;
boxRect// Set bounding box dimensions
.width = objectInfo.width;
boxRect.height = objectInfo.height;
boxRect
// Scale bounding box line width based on display resolution
int lineWidth = (int)(Screen.width * 1.75e-3);
// Render bounding box
.DrawTexture(
GUI: boxRect,
position: Texture2D.whiteTexture,
image: ScaleMode.StretchToFill,
scaleMode: true,
alphaBlend: 0,
imageAspect: color,
color: lineWidth,
borderWidth: 0);
borderRadius
// Include class label and confidence score in label text
string labelText = $" {name}: {(objectInfo.prob * 100).ToString("0.##")}%";
// Initialize label GUI content
= new GUIContent(labelText);
GUIContent labelContent
// Calculate the text size.
= labelStyle.CalcSize(labelContent);
Vector2 textSize
// Set label text coordinates
.x = objectInfo.x0;
labelRect.y = Screen.height - objectInfo.y0 - textSize.y + lineWidth;
labelRect
// Set label text dimensions
.width = Mathf.Max(textSize.x, objectInfo.width);
labelRect.height = textSize.y;
labelRect// Set label text and backgound color
.normal.textColor = color.grayscale > 0.5 ? Color.black : Color.white;
labelStyle.normal.background = colorTextures[objectInfo.label];
labelStyle// Render label
.Label(labelRect, labelContent, labelStyle);
GUI
= new Rect();
Rect objectDot .height = lineWidth * 5;
objectDot.width = lineWidth * 5;
objectDotfloat radius = objectDot.width / 2;
.x = (boxRect.x + boxRect.width / 2) - radius;
objectDot.y = (boxRect.y + boxRect.height / 2) - radius;
objectDot
.DrawTexture(
GUI: objectDot,
position: Texture2D.whiteTexture,
image: ScaleMode.StretchToFill,
scaleMode: true,
alphaBlend: 0,
imageAspect: color,
color: radius,
borderWidth: radius);
borderRadius
}
// Define styling information for GUI elements
= new GUIStyle
GUIStyle style {
= (int)(Screen.width * (1f / (100f - fontScale)))
fontSize };
.normal.textColor = textColor;
style
// Define screen spaces for GUI elements
= new Rect(10, 10, 500, 500);
Rect slot1 = new Rect(10, style.fontSize * 1.5f, 500, 500);
Rect slot2
string content = $"Objects Detected: {numObjects}";
if (displayProposalCount) GUI.Label(slot1, new GUIContent(content), style);
// Update framerate value
if (Time.unscaledTime > fpsTimer)
{
= (int)(1f / Time.unscaledDeltaTime);
fps = Time.unscaledTime + fpsRefreshRate;
fpsTimer }
// Adjust screen position when not showing predicted class
= displayProposalCount ? slot2 : slot1;
Rect fpsRect if (displayFPS) GUI.Label(fpsRect, new GUIContent($"FPS: {fps}"), style);
}
Define OnDisable Method
We’ll perform any clean-up steps in the OnDisablemethod.
private void OnDisable()
{
FreeResources();
}
Set up Unity Scene
Now we can start setting up our Unity scene. We need a screen to display the webcam feed, an empty object to attach the object detector script, dropdown menus for selecting webcams, models, and compute devices, a toggle to activate a webcam feed, and a slider to update the confidence threshold.
Create Screen object
Right-click a space in the Hierarchy tab and select 3D Object → Quad
. We can name the new object Screen.
Next, drag and drop a test image from the Assets → Images
folder onto the Screen object in the Scene view. Note that the Screen looks a bit dim. We need to change the shader for the Screen’s Material so that it does not require an external light source.
Select the Screen in the Hierarchy tab and open the Shader
dropdown menu in the Inspector tab. Type Unlit/Texture
into the search box and press enter.
Create Inference Manager object
Right-click a space in the Hierarchy tab and select Create Empty
. Name the empty object InferenceManager
.
With the InferenceManager
object selected, drag the ObjectDetector
script into the Inspector tab.
Now we can assign the screen object, compute shader, and colormap file in the Inspector tab by dragging them into their respective fields.
Add GUI prefab
We still need to create the GUI controls. To save time, I made a Prefab that we can drop into the Scene.
- Google Drive: Canvas Prefab
Drag and drop the Canvas prefab into a new folder called Prefabs.
From there, drag the prefab into the Hierarchy tab. We can see the GUI by switching to the Game view.
Configure Webcam Toggle On Value Changed function
Next, we need to pair the WebcamToggle
with the UpdateWebcamToggle
function in the ObjectDetector
script. Expand the Canvas object and select the WebcamToggle
.
Click and drag the InferenceManager
into the On Value Changed
field.
Open the No Function
dropdown menu and select ObjectDetector → UpdateWebcamToggle
.
Configure Webcam Dropdown On Value Changed function
We can follow the same steps to pair the WebcamDropdown
with the UpdateWebcamDevice
function in the ObjectDetector
script.
This time select ObjectDetector → UpdateWebcamDevice
.
Configure OpenVINOModelDropdown
On Value Changed Event
Configure OpenVINODeviceDropdown
On Value Changed Event
Configure ConfidenceThresholdSlider
On Value Changed Event
Assign GUI objects to Inference Manager
We can now assign the GUI objects to their respective fields for the ObjectDetector
script.
Add Event System
Before we can use the GUI, we need to add an Event System. Right-click a space in the Hierarchy tab and select UI → Event System
.
Test in Editor
Click the play button in the top-middle of the Editor window to test the project.
There should be a bounding box for the call sign and one for the idle hand.
Summary
In this three-part tutorial series, we learned how to perform end-to-end object detection in Unity using IceVision and OpenVINO. In part 1, we trained a YOLOX model using IceVision and exported it to OpenVINO. In part 2, we created a dynamic link library (DLL) file in Visual Studio to perform object detection with the YOLOX model using OpenVINO. Finally, in this post, we integrated the trained model into a Unity project to perform real-time object detection. You now have a template to perform object detection in Unity that you can adapt to other projects.
Previous: End-to-End Object Detection for Unity With IceVision and OpenVINO Pt. 2
Project Resources: GitHub Repository