Real-Time Object Detection in Unity With ONNX and DirectML Pt. 2
- Overview
- Create New Project
- Import Assets
- Allow Unsafe Code
- Create Processing Shader
- Create Object Detector Script
- Set up Unity Scene
- Test in Editor
- Summary
Tutorial Links
- Training Tutorial: Train a hand gesture classifier using fastai and export it to TensorFlow.js.
- Part 1: Create a dynamic link library (DLL) file in Visual Studio to perform object detection with a YOLOX model using ONNX Runtime and DirectML.
- Part 2: Perform object detection in a Unity project with ONNX Runtime and DirectML.
- GitHub Repository
Overview
In Part 2 of this tutorial series, we will integrate our DLL file into a Unity project to perform real-time object detection. We will start by creating a new Unity project and importing the necessary assets. Then, we will allow unsafe code in the project to share input data with the DLL. Next, we will create a processing shader and an object detector script to handle object detection in our Unity project. Finally, we will set up the Unity scene and test the object detection model in the Unity editor. By the end of this post, you will have a working real-time object detection system in Unity for Windows.
Important: This post assumes you already have Unity Hub on your system. Check out this section from a previous tutorial if this is not the case (link).
Create New Project
Open the Unity Hub and click New Project.
Select the target editor version from the Editor Version dropdown menu. We’ll use Unity 2022 for this post, but the current LTS release should also work fine.
Select the 2D Core
template.
Pick a name for the project and a location for the project folder before clicking Create Project
in the lower right-hand corner.
Import Assets
Once the project loads, we’ll store the DLL files from part 2 in a new folder called Plugins
. Right-click a space in the Assets section and select Create → Folder
from the popup menu.
The DLL targets 64-bit x86 architectures, so we need to place the DLL files in a subfolder named x86_64
.
Note: You can place the
Plugins
folder inside another folder if needed.
Copy all the DLL files into the Assets/Plugins/x86_64
folder. We then need to close and reopen the Unity Editor to load the plugin files.
After restarting the Unity Editor, create a new folder called Colormaps
to store the JSON file from the training tutorial.
We’ll place any test images into a new folder called Images
.
Next, we’ll create a folder to store the ONNX models. We need to place the .onnx
files in a StreamingAssets folder to include them in project builds. Create a new folder named StreamingAssets
. We’ll place each model file in a separate folder and put those in a new subfolder called ONNXModels
to keep things organized.
Allow Unsafe Code
Rather than copying the input image from Unity to the plugin, we’ll pass a pointer to the pixel data. First, we need to allow unsafe code for the Unity project. Select Edit → Project Settings...
from the top menu.
Open the Player → Other Settings
dropdown and scroll down to the Allow 'unsafe' Code
checkbox. Enable the setting and close the Project Settings window.
Now we can start coding.
Create Processing Shader
The input image gets flipped upside down when we send it to the plugin. We can pre-flip the image in a Compute Shader. We’ll add the Compute Shader in a new folder called Shaders
. Right-click a space in the Shaders
folder and select Create → Shader → Compute Shader
.
Name the Compute Shader ProcessingShader
and open it in the code editor.
Default Compute Shader Code
// Each #kernel tells which function to compile; you can have many kernels
#pragma kernel CSMain
// Create a RenderTexture with enableRandomWrite flag and set it
// with cs.SetTexture
<float4> Result;
RWTexture2D
[numthreads(8,8,1)]
void CSMain (uint3 id : SV_DispatchThreadID)
{
// TODO: insert actual code here!
[id.xy] = float4(id.x & id.y, (id.x & 15)/15.0, (id.y & 15)/15.0, 0.0);
Result}
We need to add a new Texture2D
variable to store the pixel data for the input image. We’ll remove the default method and create a new one called FlipXAxis
. Replace the default method name in the #pragma kernel
line at the top.
We need the input image height for the flip operation, which we can access with the Texture2D::GetDimensions function.
// Each #kernel tells which function to compile; you can have many kernels
#pragma kernel FlipXAxis
// The pixel data for the input image
<float4> InputImage;
Texture2D// The pixel data for the processed image
<float4> Result;
RWTexture2D
// Flip the image around the x-axis
[numthreads(8, 8, 1)]
void FlipXAxis(uint3 id : SV_DispatchThreadID)
{
// Stores the InputImage width
uint width;
// Stores the InputImage height
uint height;
// Get the dimensions of the InputImage
.GetDimensions(width, height);
InputImage
// Update the y value for the pixel coordinates
= int2(id.x, height - id.y);
int2 coords [id.xy] = float4(InputImage[coords].x, InputImage[coords].y, InputImage[coords].z, 1.0f);
Result}
Create Object Detector Script
We’ll store the C# script that interacts with the plugin in a new Scripts
folder. Right-click a space inside it and select Create → C# Script
.
Name the script ObjectDetector
and open it in the code editor.
Default script code
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
public class ObjectDetector : MonoBehaviour
{
// Start is called before the first frame update
void Start()
{
}
// Update is called once per frame
void Update()
{
}
}
Add required namespaces
- System: Contains fundamental classes and base classes that define commonly-used value and reference data types, events and event handlers, interfaces, attributes, and processing exceptions.
- UnityEngine.UI: Provides access to UI elements.
- UnityEngine.Rendering: Provides access to the elements of the rendering pipeline.
- System.Runtime.InteropServices: Provides a wide variety of members that support COM interop and platform invoke services.
- System.IO: Allows reading and writing to files and data streams.
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.Rendering;
using System;
using UnityEngine.UI;
using System.Runtime.InteropServices;
using System.IO;
Add code to copy DirectML.dll
file to editor folder
We must copy the DirectML.dll
file from the Plugins folder to the parent folder for the Unity Editor application to use DirectML in the Editor. We’ll also need to copy that file to the build folder when building the Unity project. We can handle both steps automatically in code.
We can obtain the path to the current Unity Editor from the EditorApplication.applicationpath
variable.
Unity provides an InitializeOnLoad
attribute to run code in the Unity Editor without requiring action from the user. This attribute requires the UnityEditor
namespace. We can only use this while in the Editor, so we need to wrap the code in Conditional compilation preprocessor directives. We’ll place this code right below the namespaces.
#if UNITY_EDITOR
using UnityEditor;
[InitializeOnLoad]
public class Startup
{
static Startup()
{
// Get all files named "DirectML.dll" in the Assets directory
string[] files = Directory.GetFiles("./Assets/", "DirectML.dll", SearchOption.AllDirectories);
// Iterate through each found file
foreach (string file in files)
{
// Check if the file is in the "x86_64" folder
if (file.Contains("x86_64"))
{
// Get the file path for the Editor application
string editorPath = EditorApplication.applicationPath;
// Extract the parent folder for the Editor application
string editorDir = Directory.GetParent(editorPath).ToString();
// Define target file path
string targetPath = $"{editorDir}/DirectML.dll";
// Only copy the file to the Editor application folder if it is not already present
if (!File.Exists(targetPath)) File.Copy(file, targetPath);
}
}
}
}
#endif
We use the UNITY_EDITOR
scripting symbol to check whether we are in the Unity Editor. We are in the Editor, so it returns true, and the code executes.
If we check if we are not in the Unity Editor, it returns false, and the code block does not execute.
We can verify the code works by saving the script and going to the parent folder for Editor application. The DirectML.dll
file should be present.
Note: I install Unity editors to a location that does not require Administrator access. You might need to manually copy the file if this is not the case for you.
Define public variables
We’ll add the required public variables above the Start method. We will be able to access these variables in the Inspector tab. We can add Header attributes to organize the public variables in the Inspector tab and use Tooltip attributes to provide information about variables.
Define scene object variables
First, we need a variable to access the screen object that displays either a test image or webcam input. We may or may not want to mirror the screen based on whether a webcam is facing the user.
[Header("Scene Objects")]
[Tooltip("The Screen object for the scene")]
public Transform screen;
[Tooltip("Mirror the in-game screen.")]
public bool mirrorScreen = true;
Define data processing variables
Next, we’ll define the variables for processing model input. We can set the default target input resolution to 224
and use it to scale the source resolution while maintaining the original aspect ratio.
We’ll also add a public ComputeShader
variable to access the ProcessingShader
we made earlier.
We need to download the pixel data for the input image from the GPU to the CPU before passing it to the plugin. This step can cause a significant performance bottleneck, so we’ll add the option to read the model output asynchronously at the cost of a few frames of latency. This latency might cause the bounding box to trail slightly behind a fast-moving object on the screen. The effect should be minimal, provided the frame rate is high enough.
[Header("Data Processing")]
[Tooltip("The target minimum model input dimensions")]
public int targetDim = 224;
[Tooltip("The compute shader for GPU processing")]
public ComputeShader processingShader;
[Tooltip("Asynchronously download input image from the GPU to the CPU.")]
public bool useAsyncGPUReadback = true;
Define output processing variables
We pass in the JSON file containing the class labels as a TextAsset.
[Header("Output Processing")]
[Tooltip("A json file containing the colormaps for object classes")]
public TextAsset colormapFile;
[Tooltip("Minimum confidence score for keeping detected objects")]
[Range(0,1f)]
public float minConfidence = 0.5f;
Define variables for debugging
Next, we’ll add a Boolean variable to toggle printing debug messages to the console.
[Header("Debugging")]
[Tooltip("Print debugging messages to the console")]
public bool printDebugMessages = true;
Define webcam variables
We need to specify a desired resolution and framerate when using a webcam as input.
[Header("Webcam")]
[Tooltip("Use a webcam as input")]
public bool useWebcam = false;
[Tooltip("The requested webcam dimensions")]
public Vector2Int webcamDims = new Vector2Int(1280, 720);
[Tooltip("The requested webcam framerate")]
[Range(0, 60)]
public int webcamFPS = 60;
Define variables for user interface
We’ll make a simple GUI that displays the predicted class, the current framerate, and controls for selecting webcam devices, models, and execution providers.
[Header("GUI")]
[Tooltip("Display predicted class")]
public bool displayBoundingBoxes = true;
[Tooltip("Display number of detected objects")]
public bool displayProposalCount = true;
[Tooltip("Display fps")]
public bool displayFPS = true;
[Tooltip("The on-screen text color")]
public Color textColor = Color.red;
[Tooltip("The scale value for the on-screen font size")]
[Range(0, 99)]
public int fontScale = 50;
[Tooltip("The number of seconds to wait between refreshing the fps value")]
[Range(0.01f, 1.0f)]
public float fpsRefreshRate = 0.1f;
[Tooltip("The toggle for using a webcam as the input source")]
public Toggle useWebcamToggle;
[Tooltip("The dropdown menu that lists available webcam devices")]
public Dropdown webcamDropdown;
[Tooltip("The dropdown menu that lists available ONNX models")]
public Dropdown modelDropdown;
[Tooltip("The dropdown menu that lists available ONNX execution providers")]
public Dropdown executionProviderDropdown;
Define public variables for the ONNX plugin
[Header("ONNX")]
[Tooltip("The name of the ONNX models folder")]
public string onnxModelsDir = "ONNXModels";
Define private variables
We’ll add the required private variables right below the public variables.
Define private webcam variables
We’ll keep a list of available webcam devices so users can switch between them. Unity renders webcam input to a WebcamTexture.
// List of available webcam devices
private WebCamDevice[] webcamDevices;
// Live video input from a webcam
private WebCamTexture webcamTexture;
// The name of the current webcam device
private string currentWebcam;
Define input variables
We’ll update the dimensions and content of the screen object based on the test image or webcam.
When using asynchronous GPU readback, we need one Texture that stores data on the GPU and one that stores data on the CPU.
// The test image dimensions
private Vector2Int imageDims;
// The test image texture
private Texture imageTexture;
// The current screen object dimensions
private Vector2Int screenDims;
// The model GPU input texture
private RenderTexture inputTextureGPU;
// The model CPU input texture
private Texture2D inputTextureCPU;
Define variable for tracking the current number of detected objects
// Stores the number of detected objects
private int numObjects;
Define variables for storing colormaps
We need to create a couple of classes to parse the JSON content.
// A class for parsing in colormaps from a JSON file
[System.Serializable]
class ColorMap { public string label; public float[] color; }
// A class for reading in a list of colormaps from a JSON file
[System.Serializable]
class ColorMapList { public List<ColorMap> items; }
// Stores a list of colormaps from a JSON file
private ColorMapList colormapList;
// A list of colors that map to class labels
private Color[] colors;
// A list of single pixel textures that map to class labels
private Texture2D[] colorTextures;
Define variables for tracking the framerate
We’ll define some variables to track the frame rate.
// The current frame rate value
private int fps = 0;
// Controls when the frame rate value updates
private float fpsTimer = 0f;
Define private variables for the plugin
// File paths for the available ONNX models
private List<string> modelPaths = new List<string>();
// Names of the available ONNX models
private List<string> modelNames = new List<string>();
// Names of the available ONNX execution providers
private List<string> onnxExecutionProviders = new List<string>();
Define a struct for reading object information from the plugin
We need to create an Object
struct for Unity to match the one we defined for the ONNX Runtime code, along with an array of Object
structs that we’ll update with the PopulateObjectsArray()
function.
// Indicate that the members of the struct are laid out sequentially
[StructLayout(LayoutKind.Sequential)]
/// <summary>
/// Stores the information for a single object
/// </summary>
public struct Object
{
// The X coordinate for the top left bounding box corner
public float x0;
// The Y coordinate for the top left bounding box cornder
public float y0;
// The width of the bounding box
public float width;
// The height of the bounding box
public float height;
// The object class index for the detected object
public int label;
// The model confidence score for the object
public float prob;
public Object(float x0, float y0, float width, float height, int label, float prob)
{
this.x0 = x0;
this.y0 = y0;
this.width = width;
this.height = height;
this.label = label;
this.prob = prob;
}
}
// Stores information for the current list of detected objects
private Object[] objectInfoArray;
Import functions from the plugin
We pass the pointer to the input pixel data as an IntPtr.
// Name of the DLL file
const string dll = "ONNX_YOLOX_DLL";
[DllImport(dll)]
private static extern int InitOrtAPI();
[DllImport(dll)]
private static extern int GetProviderCount();
[DllImport(dll)]
private static extern IntPtr GetProviderName(int index);
[DllImport(dll)]
private static extern void SetConfidenceThreshold(float minConfidence);
[DllImport(dll)]
private static extern void RefreshMemory();
[DllImport(dll)]
private static extern int LoadModel(string model, string execution_provider, int[] inputDims);
[DllImport(dll)]
private static extern int PerformInference(IntPtr inputData);
[DllImport(dll)]
private static extern void PopulateObjectsArray(IntPtr objects);
[DllImport(dll)]
private static extern void FreeResources();
Define Initialization Methods
We first need to define some methods to initialize webcams, the screen object, any GUI dropdown menus, and the in-game camera.
Define method to initialize a webcam device
/// <summary>
/// Initialize the selected webcam device
/// </summary>
/// <param name="deviceName">The name of the selected webcam device</param>
private void InitializeWebcam(string deviceName)
{
// Stop any webcams already playing
if (webcamTexture && webcamTexture.isPlaying) webcamTexture.Stop();
// Create a new WebCamTexture
= new WebCamTexture(deviceName, webcamDims.x, webcamDims.y, webcamFPS);
webcamTexture
// Start the webcam
.Play();
webcamTexture// Check if webcam is playing
= webcamTexture.isPlaying;
useWebcam // Update toggle value
.SetIsOnWithoutNotify(useWebcam);
useWebcamToggle
.Log(useWebcam ? "Webcam is playing" : "Webcam not playing, option disabled");
Debug}
Define method to initialize the in-scene screen object
/// <summary>
/// Resize and position an in-scene screen object
/// </summary>
private void InitializeScreen()
{
// Set the texture for the screen object
.gameObject.GetComponent<MeshRenderer>().material.mainTexture = useWebcam ? webcamTexture : imageTexture;
screen// Set the screen dimensions
= useWebcam ? new Vector2Int(webcamTexture.width, webcamTexture.height) : imageDims;
screenDims
// Flip the screen around the Y-Axis when using webcam
float yRotation = useWebcam && mirrorScreen ? 180f : 0f;
// Invert the scale value for the Z-Axis when using webcam
float zScale = useWebcam && mirrorScreen ? -1f : 1f;
// Set screen rotation
.rotation = Quaternion.Euler(0, yRotation, 0);
screen// Adjust the screen dimensions
.localScale = new Vector3(screenDims.x, screenDims.y, zScale);
screen
// Adjust the screen position
.position = new Vector3(screenDims.x / 2, screenDims.y / 2, 1);
screen}
Define method to get the available ONNX models
/// <summary>
/// Get the file paths for available ONNX models
/// </summary>
private void GetONNXModels()
{
// Get the paths for each model folder
foreach (string dir in System.IO.Directory.GetDirectories($"{Application.streamingAssetsPath}/{onnxModelsDir}"))
{
// Extract the model folder name
string modelName = dir.Split('\\')[1];
// Add name to list of model names
.Add(modelName);
modelNames
// Get the paths for the ONNX file for each model
foreach (string file in System.IO.Directory.GetFiles(dir))
{
if (file.EndsWith(".onnx"))
{
.Add(file);
modelPaths}
}
}
}
Define method to get the names of available execution providers
/// <summary>
/// Get the names of the available ONNX execution providers
/// </summary>
private void GetONNXExecutionProviders()
{
// Get the number of available ONNX execution providers
int providerCount = GetProviderCount();
.Log($"Provider Count: {providerCount}");
Debug
for (int i = 0; i < providerCount; i++)
{
string providerName = Marshal.PtrToStringAnsi(GetProviderName(i));
.Log(providerName);
Debug= providerName.Replace("ExecutionProvider", "");
providerName .Add(providerName);
onnxExecutionProviders}
.Reverse();
onnxExecutionProviders}
Define method to initialize GUI dropdown menu options
/// <summary>
/// Initialize the GUI dropdown list
/// </summary>
private void InitializeDropdown()
{
// Create list of webcam device names
<string> webcamNames = new List<string>();
Listforeach (WebCamDevice device in webcamDevices) webcamNames.Add(device.name);
// Remove default dropdown options
.ClearOptions();
webcamDropdown// Add webcam device names to dropdown menu
.AddOptions(webcamNames);
webcamDropdown// Set the value for the dropdown to the current webcam device
.SetValueWithoutNotify(webcamNames.IndexOf(currentWebcam));
webcamDropdown
// Remove default dropdown options
.ClearOptions();
modelDropdown// Add ONNX model names to menu
.AddOptions(modelNames);
modelDropdown// Select the first option in the dropdown
.SetValueWithoutNotify(0);
modelDropdown
// Remove default dropdown options
.ClearOptions();
executionProviderDropdown// Add ONNX provider names to menu
.AddOptions(onnxExecutionProviders);
executionProviderDropdown// Select the first option in the dropdown
.SetValueWithoutNotify(0);
executionProviderDropdown}
Define method to initialize the in-scene camera object
/// <summary>
/// Resize and position the main camera based on an in-scene screen object
/// </summary>
/// <param name="screenDims">The dimensions of an in-scene screen object</param>
private void InitializeCamera(Vector2Int screenDims, string cameraName = "Main Camera")
{
// Get a reference to the Main Camera GameObject
= GameObject.Find(cameraName);
GameObject camera // Adjust the camera position to account for updates to the screenDims
.transform.position = new Vector3(screenDims.x / 2, screenDims.y / 2, -10f);
camera// Render objects with no perspective (i.e. 2D)
.GetComponent<Camera>().orthographic = true;
camera// Adjust the camera size to account for updates to the screenDims
.GetComponent<Camera>().orthographicSize = screenDims.y / 2;
camera}
Define method to update the selected ONNX model
/// <summary>
/// Update the selected ONNX model
/// </summary>
public void UpdateONNXModel()
{
// Reset objectInfoArray
= new Object[0];
objectInfoArray
int[] inputDims = new int[] {
.width,
inputTextureCPU.height
inputTextureCPU};
.Log($"Source input dims: {inputDims[0]} x {inputDims[1]}");
Debug
// Load the specified ONNX model
int return_msg = LoadModel(
[modelDropdown.value],
modelPaths[executionProviderDropdown.value],
onnxExecutionProviders);
inputDims
SetConfidenceThreshold(minConfidence);
string[] return_messages = {
"Using DirectML",
"Using CPU",
};
.Log($"Updated input dims: {inputDims[0]} x {inputDims[1]}");
Debug.Log($"Return message: {return_messages[return_msg]}");
Debug}
Define Awake Method
We’ll implement the code to copy the DirectML.dll
file from the Plugins/x86_64
folder to the root of the build folder in the Awake() method. The code should be inactive since we are in the Editor.
// Awake runs when the script instance is being loaded
private void Awake()
{
#if !UNITY_EDITOR
// Define the path for the DirectML.dll file in the StreamingAssets folder
string sourcePath = $"{Application.dataPath}/Plugins/x86_64/DirectML.dll";
string dataPath = Application.dataPath;
string buildDir = Directory.GetParent(dataPath).ToString();
// Define the destination path for the DirectML.dll file
string targetPath = $"{buildDir}/DirectML.dll";
// Only copy the file if it is not already present at the destination
if (!File.Exists(targetPath)) File.Copy(sourcePath, targetPath);
#endif
}
Define Start Method
The Start method is called once before the first frame update, so we’ll perform any required setup steps here.
// Start runs before the first frame update
void Start()
{
// Get the source image texture
= screen.gameObject.GetComponent<MeshRenderer>().material.mainTexture;
imageTexture // Get the source image dimensions as a Vector2Int
= new Vector2Int(imageTexture.width, imageTexture.height);
imageDims
// Initialize list of available webcam devices
= WebCamTexture.devices;
webcamDevices foreach (WebCamDevice device in webcamDevices) Debug.Log(device.name);
= webcamDevices[0].name;
currentWebcam = webcamDevices.Length > 0 ? useWebcam : false;
useWebcam // Initialize webcam
if (useWebcam) InitializeWebcam(currentWebcam);
// Resize and position the screen object using the source image dimensions
InitializeScreen();
// Resize and position the main camera using the source image dimensions
InitializeCamera(screenDims);
// Initialize list of color maps from JSON file
= JsonUtility.FromJson<ColorMapList>(colormapFile.text);
colormapList // Initialize the list of colors
= new Color[colormapList.items.Count];
colors // Initialize the list of color textures
= new Texture2D[colormapList.items.Count];
colorTextures
// Populate the color and color texture arrays
for (int i = 0; i < colors.Length; i++)
{
// Create a new color object
[i] = new Color(
colors.items[i].color[0],
colormapList.items[i].color[1],
colormapList.items[i].color[2]);
colormapList// Create a single-pixel texture
[i] = new Texture2D(1, 1);
colorTextures[i].SetPixel(0, 0, colors[i]);
colorTextures[i].Apply();
colorTextures
}
// Get the file paths for available ONNX models
GetONNXModels();
// Initialize the ONNX Runtime API
InitOrtAPI();
// Get the names of available ONNX execution providers
GetONNXExecutionProviders();
// Initialize the webcam dropdown list
InitializeDropdown();
}
Define Processing Methods
Next, we need to define methods to process images using the Compute Shader, calculate the input resolution, handle asynchronous GPU readback, and scale the bounding box information.
Define method to process images using a compute shader
/// <summary>
/// Process the provided image using the specified function on the GPU
/// </summary>
/// <param name="image">The target image RenderTexture</param>
/// <param name="computeShader">The target ComputerShader</param>
/// <param name="functionName">The target ComputeShader function</param>
/// <returns></returns>
private void ProcessImageGPU(RenderTexture image, ComputeShader computeShader, string functionName)
{
// Specify the number of threads on the GPU
int numthreads = 8;
// Get the index for the specified function in the ComputeShader
int kernelHandle = computeShader.FindKernel(functionName);
// Define a temporary HDR RenderTexture
= new RenderTexture(image.width, image.height, 24, RenderTextureFormat.ARGBHalf);
RenderTexture result // Enable random write access
.enableRandomWrite = true;
result// Create the HDR RenderTexture
.Create();
result
// Set the value for the Result variable in the ComputeShader
.SetTexture(kernelHandle, "Result", result);
computeShader// Set the value for the InputImage variable in the ComputeShader
.SetTexture(kernelHandle, "InputImage", image);
computeShader
// Execute the ComputeShader
.Dispatch(kernelHandle, result.width / numthreads, result.height / numthreads, 1);
computeShader
// Copy the result into the source RenderTexture
.Blit(result, image);
Graphics
// Release RenderTexture
.Release();
result}
Define method to calculate input resolution
/// <summary>
/// Scale the source image resolution to the target input dimensions
/// while maintaing the source aspect ratio.
/// </summary>
/// <param name="imageDims"></param>
/// <param name="targetDims"></param>
/// <returns></returns>
private Vector2Int CalculateInputDims(Vector2Int imageDims, int targetDim)
{
= new Vector2Int();
Vector2Int inputDims
// Calculate the input dimensions using the target minimum dimension
if (imageDims.x >= imageDims.y)
{
[0] = (int)(imageDims.x / ((float)imageDims.y / (float)targetDim));
inputDims[1] = targetDim;
inputDims}
else
{
[0] = targetDim;
inputDims[1] = (int)(imageDims.y / ((float)imageDims.x / (float)targetDim));
inputDims}
return inputDims;
}
Define method to handle asynchronous GPU readback
/// <summary>
/// Called once AsyncGPUReadback has been completed
/// </summary>
/// <param name="request"></param>
private void OnCompleteReadback(AsyncGPUReadbackRequest request)
{
if (request.hasError)
{
.Log("GPU readback error detected.");
Debugreturn;
}
// Make sure the Texture2D is not null
if (inputTextureCPU)
{
// Fill Texture2D with raw data from the AsyncGPUReadbackRequest
.LoadRawTextureData(request.GetData<uint>());
inputTextureCPU// Apply changes to Textur2D
.Apply();
inputTextureCPU}
}
Define method to send the input texture data to the plugin
/// <summary>
/// Pin memory for the input data and pass a reference to the plugin for inference
/// </summary>
/// <param name="texture">The input texture</param>
/// <returns></returns>
public unsafe int UploadTexture(Texture2D texture)
{
//Pin Memory
fixed (byte* p = texture.GetRawTextureData())
{
// Perform inference and get the number of detected objects
= PerformInference((IntPtr)p);
numObjects }
// Initialize the array
= new Object[numObjects];
objectInfoArray
// Pin memory
fixed (Object* o = objectInfoArray)
{
// Get the detected objects
PopulateObjectsArray((IntPtr)o);
}
return numObjects;
}
Define method to scale bounding boxes to the display resolution
/// <summary>
/// Scale the latest bounding boxes to the display resolution
/// </summary>
public void ScaleBoundingBoxes()
{
// Process new detected objects
for (int i = 0; i < objectInfoArray.Length; i++)
{
// The smallest dimension of the screen
float minScreenDim = Mathf.Min(screen.transform.localScale.x, screen.transform.localScale.y);
// The smallest input dimension
int minInputDim = Mathf.Min(inputTextureCPU.width, inputTextureCPU.height);
// Calculate the scale value between the in-game screen and input dimensions
float minImgScale = minScreenDim / minInputDim;
// Calculate the scale value between the in-game screen and display
float displayScale = Screen.height / screen.transform.localScale.y;
// Scale bounding box to in-game screen resolution and flip the bbox coordinates vertically
float x0 = objectInfoArray[i].x0 * minImgScale;
float y0 = (inputTextureCPU.height - objectInfoArray[i].y0) * minImgScale;
float width = objectInfoArray[i].width * minImgScale;
float height = objectInfoArray[i].height * minImgScale;
// Mirror bounding box across screen
if (mirrorScreen && useWebcam) x0 = screen.transform.localScale.x - x0 - width;
// Scale bounding boxes to display resolution
[i].x0 = x0 * displayScale;
objectInfoArray[i].y0 = y0 * displayScale;
objectInfoArray[i].width = width * displayScale;
objectInfoArray[i].height = height * displayScale;
objectInfoArray
// Offset the bounding box coordinates based on the difference between the in-game screen and display
[i].x0 += (Screen.width - screen.transform.localScale.x * displayScale) / 2;
objectInfoArray}
}
Define Update method
We’ll place anything we want to run every frame in the Update method.
// Update runs once per frame
void Update()
{
= webcamDevices.Length > 0 ? useWebcam : false;
useWebcam if (useWebcam)
{
// Initialize webcam if it is not already playing
if (!webcamTexture || !webcamTexture.isPlaying) InitializeWebcam(currentWebcam);
// Skip the rest of the method if the webcam is not initialized
if (webcamTexture.width <= 16) return;
// Make sure screen dimensions match webcam resolution when using webcam
if (screenDims.x != webcamTexture.width)
{
// Resize and position the screen object using the source image dimensions
InitializeScreen();
// Resize and position the main camera using the source image dimensions
InitializeCamera(screenDims);
}
}
else if (webcamTexture && webcamTexture.isPlaying)
{
// Stop the current webcam
.Stop();
webcamTexture
// Resize and position the screen object using the source image dimensions
InitializeScreen();
// Resize and position the main camera using the source image dimensions
InitializeCamera(screenDims);
}
// Scale the source image resolution
= CalculateInputDims(screenDims, targetDim);
Vector2Int inputDims if (printDebugMessages) Debug.Log($"Input Dims: {inputDims.x} x {inputDims.y}");
// Initialize the input texture with the calculated input dimensions
= RenderTexture.GetTemporary(inputDims.x, inputDims.y, 24, RenderTextureFormat.ARGBHalf);
inputTextureGPU
if (!inputTextureCPU || inputTextureCPU.width != inputTextureGPU.width)
{
= new Texture2D(inputDims.x, inputDims.y, TextureFormat.RGBA32, false);
inputTextureCPU // Update the selected ONNX model
UpdateONNXModel();
}
// Copy the source texture into model input texture
.Blit((useWebcam ? webcamTexture : imageTexture), inputTextureGPU);
Graphics
// Flip image before sending to DLL
ProcessImageGPU(inputTextureGPU, processingShader, "FlipXAxis");
// Download pixel data from GPU to CPU
if (useAsyncGPUReadback)
{
.Request(inputTextureGPU, 0, TextureFormat.RGBA32, OnCompleteReadback);
AsyncGPUReadback}
else
{
.active = inputTextureGPU;
RenderTexture.ReadPixels(new Rect(0, 0, inputTextureGPU.width, inputTextureGPU.height), 0, 0);
inputTextureCPU.Apply();
inputTextureCPU}
// Send reference to inputData to DLL
= UploadTexture(inputTextureCPU);
numObjects if (printDebugMessages) Debug.Log($"Detected {numObjects} objects");
// Scale bounding boxes
ScaleBoundingBoxes();
// Release the input texture
.ReleaseTemporary(inputTextureGPU);
RenderTexture}
Define GUI Methods
We need some methods to handle user interactions with the GUI and display the bounding boxes and current framerate.
Define method that to handle switching ONNX models and execution providers
/// <summary>
/// This method runs when the value for an ONNX dropdown changes
/// </summary>
public void ONNXDropdownUpdate()
{
// Only call plugin methods after initializing the input texture
if (inputTextureCPU)
{
RefreshMemory();
UpdateONNXModel();
}
}
Define method to update webcam usage from GUI
/// <summary>
/// This method is called when the value for the webcam toggle changes
/// </summary>
/// <param name="useWebcam"></param>
public void UpdateWebcamToggle(bool useWebcam)
{
this.useWebcam = useWebcam;
}
Define method to update webcam device from GUI
/// <summary>
/// The method is called when the selected value for the webcam dropdown changes
/// </summary>
public void UpdateWebcamDevice()
{
= webcamDevices[webcamDropdown.value].name;
currentWebcam .Log($"Selected Webcam: {currentWebcam}");
Debug// Initialize webcam if it is not already playing
if (useWebcam) InitializeWebcam(currentWebcam);
// Resize and position the screen object using the source image dimensions
InitializeScreen();
// Resize and position the main camera using the source image dimensions
InitializeCamera(screenDims);
}
Define method to update the minimum confidence value
/// <summary>
/// Update the minimum confidence score for keeping bounding box proposals
/// </summary>
/// <param name="slider"></param>
public void UpdateConfidenceThreshold(Slider slider)
{
= slider.value;
minConfidence SetConfidenceThreshold(minConfidence);
}
Define OnGUI method
We’ll display the predicted bounding boxes and current frame rate in the OnGUI method.
// OnGUI handles and renders GUI events.
public void OnGUI()
{
// Initialize a rectangle for label text
= new Rect();
Rect labelRect // Initialize a rectangle for bounding boxes
= new Rect();
Rect boxRect
= new GUIStyle
GUIStyle labelStyle {
= (int)(Screen.width * 11e-3)
fontSize };
.alignment = TextAnchor.MiddleLeft;
labelStyle
foreach (Object objectInfo in objectInfoArray)
{
if (!displayBoundingBoxes) break;
// Skip object if label index is out of bounds
if (objectInfo.label > colors.Length - 1) continue;
// Get color for current class index
= colors[objectInfo.label];
Color color // Get label for current class index
string name = colormapList.items[objectInfo.label].label;
// Set bounding box coordinates
.x = objectInfo.x0;
boxRect.y = Screen.height - objectInfo.y0;
boxRect// Set bounding box dimensions
.width = objectInfo.width;
boxRect.height = objectInfo.height;
boxRect
// Scale bounding box line width based on display resolution
int lineWidth = (int)(Screen.width * 1.75e-3);
// Render bounding box
.DrawTexture(
GUI: boxRect,
position: Texture2D.whiteTexture,
image: ScaleMode.StretchToFill,
scaleMode: true,
alphaBlend: 0,
imageAspect: color,
color: lineWidth,
borderWidth: 0);
borderRadius
// Include class label and confidence score in label text
string labelText = $" {name}: {(objectInfo.prob * 100).ToString("0.##")}%";
// Initialize label GUI content
= new GUIContent(labelText);
GUIContent labelContent
// Calculate the text size.
= labelStyle.CalcSize(labelContent);
Vector2 textSize
// Set label text coordinates
.x = objectInfo.x0;
labelRect.y = Screen.height - objectInfo.y0 - textSize.y + lineWidth;
labelRect
// Set label text dimensions
.width = Mathf.Max(textSize.x, objectInfo.width);
labelRect.height = textSize.y;
labelRect// Set label text and backgound color
.normal.textColor = color.grayscale > 0.5 ? Color.black : Color.white;
labelStyle.normal.background = colorTextures[objectInfo.label];
labelStyle// Render label
.Label(labelRect, labelContent, labelStyle);
GUI
= new Rect();
Rect objectDot .height = lineWidth * 5;
objectDot.width = lineWidth * 5;
objectDotfloat radius = objectDot.width / 2;
.x = (boxRect.x + boxRect.width / 2) - radius;
objectDot.y = (boxRect.y + boxRect.height / 2) - radius;
objectDot
.DrawTexture(
GUI: objectDot,
position: Texture2D.whiteTexture,
image: ScaleMode.StretchToFill,
scaleMode: true,
alphaBlend: 0,
imageAspect: color,
color: radius,
borderWidth: radius);
borderRadius
}
// Define styling information for GUI elements
= new GUIStyle
GUIStyle style {
= (int)(Screen.width * (1f / (100f - fontScale)))
fontSize };
.normal.textColor = textColor;
style
// Define screen spaces for GUI elements
= new Rect(10, 10, 500, 500);
Rect slot1 = new Rect(10, style.fontSize * 1.5f, 500, 500);
Rect slot2
string content = $"Objects Detected: {numObjects}";
if (displayProposalCount) GUI.Label(slot1, new GUIContent(content), style);
// Update framerate value
if (Time.unscaledTime > fpsTimer)
{
= (int)(1f / Time.unscaledDeltaTime);
fps = Time.unscaledTime + fpsRefreshRate;
fpsTimer }
// Adjust screen position when not showing predicted class
= displayProposalCount ? slot2 : slot1;
Rect fpsRect if (displayFPS) GUI.Label(fpsRect, new GUIContent($"FPS: {fps}"), style);
}
Define method to exit the application using the GUI
/// <summary>
/// This method runs when the user clicks the GUI Quit button
/// </summary>
public void Quit()
{
.Quit();
Application}
Define OnApplicationQuit Method
We’ll perform any clean-up steps in the OnApplicationQuitmethod.
private void OnApplicationQuit()
{
FreeResources();
}
Set up Unity Scene
Now we can start setting up our Unity scene. We need a screen to display the webcam feed, an empty object to attach the object detector script, dropdown menus for selecting webcams, models, and execution providers, a toggle to activate a webcam feed, and a slider to update the confidence threshold.
Create Screen object
Right-click a space in the Hierarchy tab and select 3D Object → Quad
. We can name the new object Screen.
Next, drag and drop a test image from the Assets → Images
folder onto the Screen object in the Scene view. Note that the Screen looks a bit dim. We need to change the shader for the Screen’s Material so that it does not require an external light source.
Select the Screen in the Hierarchy tab and open the Shader
dropdown menu in the Inspector tab. Type Unlit/Texture
into the search box and press enter.
Create Inference Manager object
Right-click a space in the Hierarchy tab and select Create Empty
. Name the empty object InferenceManager
.
With the InferenceManager
object selected, drag the ObjectDetector
script into the Inspector tab.
Now we can assign the screen object, compute shader, and colormap file in the Inspector tab by dragging them into their respective fields.
Add GUI prefab
We still need to create the GUI controls. To save time, I made a Prefab that we can drop into the Scene.
- Google Drive: Canvas Prefab
Drag and drop the Canvas prefab into a new folder called Prefabs.
From there, drag the prefab into the Hierarchy tab. We can see the GUI by switching to the Game
view.
Configure Webcam Toggle On Value Changed function
Next, we need to pair the WebcamToggle
with the UpdateWebcamToggle
function in the ObjectDetector
script. Expand the Canvas object and select the WebcamToggle
.
Click and drag the InferenceManager
into the On Value Changed
field.
Open the No Function
dropdown menu and select ObjectDetector → UpdateWebcamToggle
.
Configure Webcam Dropdown On Value Changed function
We can follow the same steps to pair the WebcamDropdown
with the UpdateWebcamDevice
function in the ObjectDetector
script.
This time select ObjectDetector → UpdateWebcamDevice
.
Configure ONNXModelDropdown
On Value Changed Event
Configure ONNXExecutionProviderDropdown
On Value Changed Event
Configure ConfidenceThresholdSlider
On Value Changed Event
Configure QuitButton
On Click Event
Assign GUI objects to Inference Manager
We can now assign the GUI objects to their respective fields for the ObjectDetector
script.
Add Event System
Before we can use the GUI, we need to add an Event System. Right-click a space in the Hierarchy tab and select UI → Event System
.
Test in Editor
Click the play button in the top-middle of the Editor window to test the project.
There should be a bounding box for the call sign and one for the idle hand.
CPU Execution Provider
DirectML Execution Provider
Summary
In this two-part tutorial series, you learned how to implement real-time object detection in Unity using ONNX Runtime and DirectML. In Part 1, we created a dynamic link library (DLL) file in Visual Studio to perform object detection with ONNX Runtime and DirectML. In this post, we integrated this DLL file into a Unity project and performed real-time object detection. You should now have the skills and knowledge to leverage ONNX Runtime and DirectML in your Unity projects.
Previous: Object Detection for Unity With ONNX Runtime and DirectML Pt. 1
Project Resources: GitHub Repository