Next-Gen Cel Shading in Unity 5

Introduction

With the arrival of Unity 5 we find that it’s never been so easy to get high quality visuals in our game, but we don’t always want realistic graphics for our project.

Cel Shading is a rendering technique used to simulate the lighting in cartoons and comics. In a more traditional diffuse shading, light creates a gradient on an object’s surface using its normals; Cel Shading polarizes the gradient, making a pixel be lit or not lit by light, with no interpolations or interpolated values.

In the previous image we can see the difference between a diffuse lighting and cel shading. The dot product between the light direction and the surface normal (which we’ll refer to as ‘NdotL’) defines the amount of light a pixel receives. Rounding this value to 0 or 1, or defining different ‘grades’ or ‘cuts’ will result in different cel shading styles.

There's plenty of ways to do cel shading in a game, some of them with incredible results, but today we are going to learn a very fast way to implement our own cel shading light model in Unity 5.

There's different ways to achieve this depending on the rendering pipeline (the rendering path) used. You can set your desired rendering path in your Camera component or your Player Settings.

Forward Rendering

With Forward Rendering, the engine goes over every vertex, every pixel and every light in the scene. The rendering complexity is huge when you’ve got multiple light sources in your scene, but when you don’t use many lights it’s a very lightweight rendering pipeline.

The best way to achieve Cel Shading with Forward Rendering is using a Custom Lighting Model in a Surface Shader.

Shader "Custom/CelShadingForward" {
	Properties {
		_Color("Color", Color) = (1, 1, 1, 1)
		_MainTex("Albedo (RGB)", 2D) = "white" {}
	}
	SubShader {
		Tags {
			"RenderType" = "Opaque"
		}
		LOD 200

		CGPROGRAM#pragma surface surf CelShadingForward#pragma target 3.0

		half4 LightingCelShadingForward(SurfaceOutput s, half3 lightDir, half atten) {
			half NdotL = dot(s.Normal, lightDir);
			if (NdotL <= 0.0) NdotL = 0;
			else NdotL = 1;
			half4 c;
			c.rgb = s.Albedo * _LightColor0.rgb * (NdotL * atten * 2);
			c.a = s.Alpha;
			return c;
		}

		sampler2D _MainTex;
		fixed4 _Color;

		struct Input {
			float2 uv_MainTex;
		};

		void surf(Input IN, inout SurfaceOutput o) {
			// Albedo comes from a texture tinted by color
			fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
			o.Albedo = c.rgb;
			o.Alpha = c.a;
		}
		ENDCG
	}
	FallBack "Diffuse"
}

The ‘LightingCelShadingForward’ function calculates the NdotL and polarizes it to 0 or 1. We then use this value as a factor to decide how much light the pixel receives, thus getting a cel shading effect.

There are better ways to do the 'branching', but for the sake of simplicity we'll leave the 'if else' that way. A possible alternative would be to use:

or if you want to soften the edge you could use:

That same ‘hack’ can be used with other custom shaders that use specular, normal mapping, rim lighting etc.

Light Pre-Pass (Legacy Deferred)

With a Deferred Pipeline the engine goes over every light on the screen just once and stores that info in a buffer. Light Pre-Pass stores only the light info, while Unity 5′s new Deferred pipeline stores info about all of the geometry in the scene. This approach is much more efficient when you’ve got a big quantity of light sources in the scene, but requires a more complex hardware architecture.

With Light Pre-Pass we can’t use a Custom Lighting model to modify lighting info, as the lighting value is already calculated in the Deferred Lighting Pass. What we need to do is modify the internal deferred lighting pass. In order to do this we must download Unity’s Built-In Shaders from the official website.

We extract the files and look for "Internal-PrePassLighting.shader” inside the “DefaultResourcesExtra” folder. We copy this file to our Unity project, preferably in the Resources folder (create it if you haven’t). If we are using Unity 5 we can change its filename.

Now let’s edit the file and look for the line:

and modify it to:

We save the file and it’s done. We now need to tell Unity to use this file instead of the default Pre-Pass Lighting pipeline.

Go to Edit -> Project Settings -> Graphics and, in Legacy Deferred, change the Built-in shader and select our file. If you are using Unity 4, Unity should already recognize the file (if you haven’t changed the filename) and use it with no user action. Please check that your Camera is using the Player Settings or is using Legacy Deferred (Light Pre-Pass), and that our materials use a Legacy shader. If you did everything correctly you should get cel shading working in your game.

Deferred Rendering

But what’s the point of having Unity 5 if we can’t take advantage of its new features and the powerful Standard Shader?

The solution is to use Unity’s new Deferred Rendering pipeline, which stores all scene geometry along lighting in buffers to calculate the final shading of the object.

Following the same steps as Light Pre-Pass, we download the Built-In Shaders but this time we copy into our project the file named “Internal-DeferredShading.shader”. We edit this file looking for the function ‘CalculateLight’ and modify the next line:

And add these lines:

Like the last time, we go to Edit -> Project Settings -> Graphics and in Deferred we choose our modified file. Check that the Camera component has Rendering Path = Deferred, and that you are using a model with a Standard shader in its material. You should now notice that your model has all the advantages of the Standard Shader, but with Cel Shading applied.

Toon Outline and Other Effects

You’ve got yourself Cel Shading in Unity 5, but we are still not done to get our next-gen toon look. We need a toon outline.

Let’s import the Camera Image Effects by going to Assets -> Import Package -> Effects.

We now go to our camera and add the Edge Detection component. We configure it at our liking (I usually use Robert Cross Depth Normals). We can now add other image effects like Anti Aliasing to soften the lines, or some more advanced effects like Ambient Occlusion, Bloom or Depth of Field to get that AAA look we were going for.

Conclusions

As you can see, in a small amount of time we ‘hacked’ the new Unity Deferred pipeline to completely change our game visual style. 

In Twin Souls: The Path of Shadows (the game I’m working on) we are using this approach with a few other additions to get our unique look:
 

I hope this article was useful to some of you, and remember to contact me at [email protected] if you have any question regarding this article.

[UPDATE] As of March of 2016 de Deferred hack doesn't seem to work. I modified the UnityStandardBRDF library as a working fix. Save this code as "UnityStandardBRDF.cginc" in your Assets/Resources/ folder: https://gist.github.com/xDavidLeon/38b392700fbec56162ba If that doesn't work either, check this post that explains a better solution.

[UPDATE 2] An update of this article for Unity 5.6 is available here: http://www.gamasutra.com/blogs/DavidLeon/20170519/298374/NextGen_Cel_Shading_in_Unity_56.php