Unity Native Plugins: A wrapper, for a wrapper, for a wrapper

Punny photo credit: Saigon by Regime Management

Unity is an awesome way to make an affordable game available on many platforms quickly.  It also has an amazing ecosystem of developer tools (ie: the Unity Asset Store).  Unity assets are a great way for middleware providers to sell to developers.  If these assets are files natively supported by multiple platforms (ie: videos, models, mono scripts, and images) than little to no work is required to sell them for all platforms.  Native plugins (ie: libraries), however, are not so easy.  

The Unity team has created documentation and a video that makes native plugins appear to be easy to create.  Mike Geig did a good job with the video, but it’s far more complicated than he can cover in an hour (and that means it’s probably too complicated for me to cover in a blog as well).  In this article I’m going to describe the issues (like this and this) I ran into and how to solve them.  I’ll also give you a more real-world example of how to create a native plugin.

My experience with writing native plugins is from my work at Affectiva to create an emotion-sensing Unity plugin.  Briefly, the plugin allows Unity games to detect emotions based on facial expressions via a webcam.  We already had SDKs for various platforms and had to decide if we wanted to port the core science code to mono or write a wrapper.  For various reasons, we decided to write a wrapper.

Affectiva already had a C# wrapper, so I naively thought I could use that.  Unity uses C#, so why wouldn’t it work?  Well, the version of Mono that Unity uses is equivalent to .NET 2.0.  Our C# SDK was built for .NET 4.5.  This meant we had to write a wrapper for our native libraries.  

The video tutorial makes this appear trivial, and it is, if you are only calling functions from native libraries.  Classes, however, are not so easy.  Here is what just the constructor and destructor look like for the Windows wrapper:

CAffdexNativeWrapper::CAffdexNativeWrapper()
{
   detector = new FrameDetector(30);
   faceFoundListener = NULL;
   faceLostListener = NULL;
   imageResultsListener = NULL;
   return;
}

CAffdexNativeWrapper::~CAffdexNativeWrapper()
{
   detector->stop();
   delete detector;
}

To be clear, this is a wrapper for the SDK (which is already a wrapper).  This wrapper exposes the SDK functionality to Unity.  If you are as naive as I was, you probably think this code could easily be used for other platforms.  Here is the OS X version of these methods:

CAffdexNativeWrapper::CAffdexNativeWrapper()
{
   detector = std::make_shared<FrameDetector>(30);
   faceFoundListener = NULL;
   faceLostListener = NULL;
   imageResultsListener = NULL;
   return;
}

CAffdexNativeWrapper::~CAffdexNativeWrapper()
{
   detector->stop();
   detector = NULL;
}

The Windows code is in a cpp file that is part of a Visual Studio project that builds a dll.  The OS X wrapper is in an mm file that is part of an XCode project that builds a bundle.  There is some common code for the Windows and OS X wrapper, which is encompassed in a 139 line header file.  For perspective, the mm file is 312 lines and the cpp file is 468 lines.  

For Windows, the library is in the form of a DLL file.  The Windows wrapper is also a DLL file.  To support 32 and 64 bit games on Windows, you need corresponding 64 and 32 bit DLLs.  

For OS X, the library must be a dylib file.  You cannot use an OS X static library or framework!  The OS X wrapper is a bundle that contains the dylib.  For OS X, you need a universal bundle that preferably supports i386, x86_64 and x86.  Here is what the file structure looks like:

For each platform, I have a native platform script.  Here is an example method for Windows from Assets / Affdex / Plugins / Scripts / WindowsNativePlatform.cs:

       [UnmanagedFunctionPointer(CallingConvention.Cdecl)]
       delegate void ImageResults(IntPtr i);

       [UnmanagedFunctionPointer(CallingConvention.Cdecl)]
       delegate void FaceResults(Int32 i);

       [DllImport("AffdexNativeWrapper")]
       private static extern int initialize(int discrete, string affdexDataPath);

       public void Initialize(Detector detector, int discrete)
       {
           WindowsNativePlatform.detector = detector;

           //load our lib!
           String adP = Application.streamingAssetsPath;
           
           String affdexDataPath = Path.Combine(adP, "affdex-data"); // Application.streamingAssetsPath + "/affdex-data";
           //String affdexDataPath = Application.dataPath + "/affdex-data";
           affdexDataPath = affdexDataPath.Replace('/', '\\');
           int status = initialize(discrete, affdexDataPath);
           Debug.Log("Initialized detector: " + status);

           FaceResults faceFound = new FaceResults(this.onFaceFound);
           FaceResults faceLost = new FaceResults(this.onFaceLost);
           ImageResults imageResults = new ImageResults(this.onImageResults);

           h1 = GCHandle.Alloc(faceFound, GCHandleType.Pinned);
           h2 = GCHandle.Alloc(faceLost, GCHandleType.Pinned);
           h3 = GCHandle.Alloc(imageResults, GCHandleType.Pinned);

           status = registerListeners(imageResults, faceFound, faceLost);
           Debug.Log("Registered listeners: " + status);
       }

Here is how it looks for OS X in Assets / Affdex / Plugins / Scripts / OSXNativePlatform.cs :

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
delegate void ImageResults(IntPtr i);

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
delegate void FaceResults(Int32 i);

[DllImport("affdex-native")]
private static extern IntPtr initialize(int discrete, string affdexDataPath);

public void Initialize(Detector detector, int discrete)
{
     OSXNativePlatform.detector = detector;
     String adP = Application.streamingAssetsPath;
     String affdexDataPath = Path.Combine(adP, "affdex-data-osx");
     int status = 0;

     try
     {
          cWrapperHandle = initialize(discrete, affdexDataPath);
     }
     catch (Exception e)
     {
          Debug.LogException(e);
     }

     Debug.Log("Initialized detector: " + status);

     FaceResults faceFound = new FaceResults(this.onFaceFound);
     FaceResults faceLost = new FaceResults(this.onFaceLost);
     ImageResults imageResults = new ImageResults(this.onImageResults);

     h1 = GCHandle.Alloc(faceFound, GCHandleType.Pinned);
     h2 = GCHandle.Alloc(faceLost, GCHandleType.Pinned);
     h3 = GCHandle.Alloc(imageResults, GCHandleType.Pinned);

     status = registerListeners(cWrapperHandle, imageResults, faceFound, faceLost);
     Debug.Log("Registered listeners: " + status);
}

As you can see, there is some redundancy here, which I might be able to consolidate.  An interface (INativePlatform.cs) is used to make it so that people can use the asset without specifying a platform:

using System;

namespace Affdex
{
   internal enum NativeEventType
   {
       ImageResults,
       FaceFound,
       FaceLost
   }

   internal struct NativeEvent
   {
       public NativeEventType type;
       public object eventData;

       public NativeEvent(NativeEventType t, object data)
       {
           type = t;
           eventData = data;
       }
   }

   internal interface INativePlatform
   {
       /// <summary>
       /// Initialize the detector.  Creates the instance for later calls
       /// </summary>
       /// <param name="discrete"></param>
       /// <param name="detector">Core detector object.  Handles all communicatoin with the native APIs.</param>
       void Initialize(Detector detector, int discrete);

       /// <summary>
       /// Start the detector
       /// </summary>
       /// <returns>Non-zero error code</returns>
       int Start();

       void Stop();

       /// <summary>
       /// Enable or disable an expression
       /// </summary>
       /// <param name="expression">ID of the expression to set the state of</param>
       /// <param name="state">ON/OFF state for the expression</param>
       void SetExpressionState(int expression, bool state);

       /// <summary>
       /// Get the ON/OFF state of the expression
       /// </summary>
       /// <param name="expression">ID of the expression</param>
       /// <returns>0/1 for OFF/ON state</returns>
       bool GetExpressionState(int expression);

       /// <summary>
       /// Enable or disable an emotion
       /// </summary>
       /// <param name="emotion">ID of the emotion to set the state of</param>
       /// <param name="state">ON/OFF state for the emotion</param>
       void SetEmotionState(int emotion, bool state);

       /// <summary>
       /// Get the ON/OFF state of the emotion
       /// </summary>
       /// <param name="emotion">emotion id to get the state of</param>
       /// <returns>0/1 for OFF/ON state</returns>
       bool GetEmotionState(int emotion);

       /// <summary>
       /// Process a single frame of data
       /// </summary>
       /// <param name="rgba">Representation of RGBA colors in 32 bit format.</param>
       /// <param name="width">Width of the frame. Value has to be greater than zero</param>
       /// <param name="height">Height of the frame. Value has to be greater than zero</param>
       /// <param name="timestamp">The timestamp of the frame (in seconds). Can be used as an identifier of the frame.  If you use Time.timeScale to pause and use the same time units then you will not be able to process frames while paused.</param>
       void ProcessFrame(byte[] rgba, int width, int height, float timestamp);

       /// <summary>
       /// Notify the native plugin to release memory and cleanup
       /// </summary>
       void Release();
   }
}

The top-level script (Detector.cs) is 447 lines of code.  Here is a snippet showing how it actually loads the libraries:

       /// <summary>
       /// Pointer to loaded library if it doesn't exist already
       /// </summary>
       private static IntPtr lib;

       internal static bool LoadNativeDll (string FileName)
       {
           if (lib != IntPtr.Zero) {
               return true;
           }
           lib = NativeMethods.LoadLibrary (FileName);
           if (lib == IntPtr.Zero) {
               Debug.LogError("Failed to load native library!");
               return false;
           }
           return true;
       }

#if UNITY_EDITOR_WIN || UNITY_STANDALONE_WIN
   /// <summary>
   /// DLL loader helper
   /// </summary>
   internal static class NativeMethods
   {
       [DllImport("kernel32", SetLastError = true, CharSet = CharSet.Unicode)]
       internal static extern IntPtr LoadLibrary(
           string lpFileName
           );
       
       [DllImport("kernel32", SetLastError = true, CharSet = CharSet.Unicode)]
       internal static extern int FreeLibrary(
           string lpFileName
           );
   }
   
#else
   /// <summary>
   /// DLL loader helper
   /// </summary>
   internal class NativeMethods
   {
       public static IntPtr LoadLibrary (string fileName)
       {
           IntPtr retVal = dlopen (fileName, RTLD_NOW);
           var errPtr = dlerror ();
           if (errPtr != IntPtr.Zero) {
               Debug.LogError (Marshal.PtrToStringAnsi (errPtr));
           }
           return retVal;
       }

       public static void FreeLibrary (IntPtr handle)
       {
           dlclose (handle);
       }

       const int RTLD_NOW = 2;

       [DllImport ("libdl.dylib")]
       private static extern IntPtr dlopen (String fileName, int flags);

       [DllImport ("libdl.dylib")]
       private static extern IntPtr dlsym (IntPtr handle, String symbol);

       [DllImport ("libdl.dylib")]
       private static extern int dlclose (IntPtr handle);

       [DllImport ("libdl.dylib")]
       private static extern IntPtr dlerror ();
   }
#endif

As you can see, just to open the libraries I need to use other libraries (libdl.dylib for OS X and kernel32.dll for Windows).  When I first created the OS X bundle I got the following error from the Unity Console:

As shown in the Unity Questions forum, here are all the steps I needed to go through to resolve this error:

Here is a diagram of what the methods and wrappers look like (for just two platforms):

That’s five layers of wrappers!  It is my hope that this helps illuminate the process of creating an asset using native libraries.  Is it worth it?  There are performance reasons to go this route.  Native code targetted to a given platform is a far more performant option.  For Affectiva's Unity asset, performance is critical.  Beyond performance, there is also the intent to reduce duplicate code.  Our hope is that, once written, the wrappers will need very little work to support future versions of the SDK.