Sponsored: Integrating Cortana with Unity in Windows Universal games (Unite 2015)

Author: by Kevin Ashley, Microsoft Game Evangelist

This article brought to you by Microsoft

IMPORTANT: This demo shows a Windows 10 Universal app built with Unity and demonstrating Cortana integration. At Unite 2015, Unity made available its public roadmap which also includes Windows Universal apps support here. Making this demo was so much fun, because Cortana naturally blends with games, and in some scenarios you can significantly improve the user experience with your game by adding intelligent voice features.

This is the sample I presented at Unite Europe 2015 event to demonstrate how to integrate Unity with Cortana in Windows 10, among other cool things, such as using Unity mecanim system for animation. Fortunately, it’s very easy to do: if you are a Unity developer, integrating Unity with Cortana can certainly add flavor to your Windows version of the app. And for users it just means more fun! So let’s get started…

The Idea: Mecanim and Cortana

I’ve been thinking of combining Unity mecanim animation with Cortana to make my animated character react to human voice commands. Cortana, besides being a personal assistant and just a friendly artificial intelligence inside my Windows devices can also help with basic speech things: parsing human voice commands and as it turns out, with Unity games. Here’s what my app looks like:

Cortana integrated app built with Unity

I’m using a free asset from Mixamo for my animation character (in fact you can use any character with animations from Unity Asset Store). I want to activate Cortana with standard Windows “Hey, Cortana” and then make my character dance, walk, run etc, by speaking a voice command. Easy?

Dancing avatar

You can find the code and also instructions for this example at the bottom of this post. Instructions:

1. After loading the Unity project, open the MainScene scene.

2. Now let’s build a Windows specific project. Open Build Settings. Since we will use Cortana in our project, click Player Settings and make sure Microphone is checked.

3. Now, select Windows Store output, then choose Universal 10 as a target. If you do not see this target, then you don’t have a Windows 10 integrated Unity. Make sure you select XAML as an output project type.

4. Click Build…, Unity will prompt you for output folder, this is where a native Visual Studio project will be generated.

5. Make sure to make the following changes to App.xaml.cs

Make AppCallbacks public instead of private:

public AppCallbacks appCallbacks;

Change OnLaunched event as follows:

protectedasyncoverridevoid OnLaunched(LaunchActivatedEventArgs args)

{

splashScreen = args.SplashScreen;

try

{

StorageFile vcdStorageFile = awaitPackage.Current.InstalledLocation.GetFileAsync(@”CortanaCommands.xml”);

await Windows.ApplicationModel.VoiceCommands.VoiceCommandDefinitionManager.InstallCommandDefinitionsFromStorageFileAsync(vcdStorageFile);

awaitSpeechHelper.InitialiseSpeechRecognition();

}

catch (Exception ex)

{

System.Diagnostics.Debug.WriteLine(“Installing Voice Commands Failed: “ + ex.ToString());

}

InitializeUnity(args.Arguments);

}

Then, update OnActivated event to include voice command activation:

protectedoverridevoid OnActivated(IActivatedEventArgs args)

{

string appArgs = “”;

switch (args.Kind)

{

caseActivationKind.Protocol:

ProtocolActivatedEventArgs eventArgs = args asProtocolActivatedEventArgs;

splashScreen = eventArgs.SplashScreen;

appArgs += string.Format(“Uri={0}”, eventArgs.Uri.AbsoluteUri);

break;

//Add VoiceCommand start detection and use the SpeechHelper handler

caseActivationKind.VoiceCommand:

SpeechHelper.HandleSpeechCommand(args);

break;

}

InitializeUnity(appArgs);

}

6. Now, drop CortanaCommands.xml, SpeechHelper.cs, ToastNotificationHelper.cs from WindowsSpecificCode folder to your project, build and run it. For your convenience I also added App.xaml.cs file.

7. When the project starts, you can use Windows 10 Cortana to give it commands such as “Avatar dance” or “Avatar run”.

Next steps:  download the code, install the Windows 10 SDK and check out our updated Windows 10 DevCenter for Game Developers.

And if you are a Unity beginner, then check out this Microsoft Virtual Academy course done by some of my friends at Microsoft for some starter files and another neat tutorial.