Toon Shading in Unity Pt. 1

May 28, 2015

I have spent the past several years of my career working as a rendering engineer on the Lockheed Martin Prepar3D flight simulator, but I've never really done a deep dive into writing custom shaders for Unity. When a recent project we were working on called for a mobile-ready toon-shader I thought I'd take the opportunity to really dig into Unity's graphics pipeline rather than just grabbing one of the excellent offerings already available on the asset store.

Unity has two very distinct paths available when writing shaders, and so I'm going to break this post up into two pieces. The first will cover the simple cube-map based lighting that we're using in our mobile game using traditional vertex and fragment shaders. The second will cover implementing a toon-shader using Unity's very cool surface shader system, which allows you to write a portion of the fragment shader that is then automatically expanded into full shaders that handle the wide variety of rendering paths and options that Unity offers.

Toon shading, also known as cel shading, is a non-photorealistic rendering technique where lighting is limited to a small number of values, resulting in large bands of shading rather than realistic, smooth gradients. It is often used to give graphics a comic book or hand drawn feel. Additionally, toon shading is often accompanied by some form of outline process, further reinforcing the illustrated appearance. In this post, we'll go through a very simple technique where a texture is used to provide the lighting environment instead of virtual lights. We're also going to include a flag which allows us to switch between to different approaches to orienting our environment. In a separate pass, the model is then expanded along its normals with only its back faces rendered to create an outline.

When creating a shader in Unity, the Shader declaration indicates what category the shader will appear under in the editor.

Shader "Toon/UnlitToonBase";

It is typically followed immediately by the properties block. The properties block for this example shader includes three properties:

//The properties block is automatically populated into a nice little editor for your material in the Unity Inspector.
Properties 
{
    // The environment cubemap acts as the lighting environment in unlit toon shading.
    // Modifying this cubemap will change the appearance of lighting
    _ENVIRONMENT( "Environment Texture", CUBE ) = "" {} //0

    // Realistically the lighting environment is independent of the position of the camera, but it can provide a more
    // cartoonish effect to have the lighting environment change with the orientation of the camera
    [MaterialToggle( _CAMERA_BASED_ENVIRONMENT_ON )] _CAMERA_ENVIRONMENT( "Use Camera Based Environment", float ) = 0 //1

    // The diffuse texture defines the basic color of the surface that will then be multiplied by the environment to get a final color.
    // The color in the diffuse texture represents the color of the surface under full, direct light.
    _DIFFUSE_TEXTURE( "Diffuse Texture", 2D ) = "white" {} //2
}

Even when writing your own vertex and fragment shaders, Unity automatically handles a huge amount of configuration and set up for you. The Subshader and Pass blocks each have a fairly large variety of configuration options that can be set with various tags. These blocks of ShaderLab tags give Unity the information it needs to configure the rendering pipeline for the shaders to execute as intended. The configuration for this toon shader is very simple:

SubShader 
{
    // This technique is intended for opaque models, we'd need to make some adjustments to handle transparency
    Tags { "RenderType" = "Opaque" }
    // Cull faces we can't see
    Cull Back
    // We don't need any lighting, so don't even bother to give us the information
    Lighting Off

    Pass
    {
        Name "BASE"
        CGPROGRAM
            // tell Unity the names of our vertex and fragment shaders
            #pragma vertex vertex
            #pragma fragment fragment

            // Pull in the Unity CG shader include file
            #include "UnityCG.cginc"

            // This tells Unity to compile 2 versions of this shader
            #pragma multi_compile _CAMERA_BASED_ENVIRONMENT_OFF _CAMERA_BASED_ENVIRONMENT_ON

            // We need 2 samplers, one for our environment cube map, and one for our diffuse texture
            samplerCUBE _ENVIRONMENT;
            sampler2D _DIFFUSE_TEXTURE;
            //..

With most of the setup taken care of, the last step before writing our shader programs is to declare the data that we'll need at each step of the process. Unity is again very helpful here, and it automatically recognizes the data being asked for to make it available to your first shader program, the vertex shader. Our shader involves 2 steps. First, we'll run a small program called a vertex shader on each vertex in the models we're drawing to determine where they should appear on screen, and then we'll run a second program called a fragment shader ( usually pixel shader in DirectX ) to compute the final appearance of each pixel covered by the model. We'll look at the input to the vertex shader first:

// We need the position, normal, and texture coordinate for each vertex.
struct VertexIn 
{
    // The position semantic gives us the position of the vertex in the model.
    // If the model is animated, that is already taken into account here, so we don't have to worry about it.
    float4 pos : POSITION;
    // The normal gives us the direction direction of the surface at each vertex
    fixed3 normal : NORMAL;
    // Each vertex has a texture coordinate that is used to map from the 3D model to its 2D texture
    fixed4 texcoord : TEXCOORD0;
};

The VertexIn structure is populated for us by the engine. We're then responsible for taking that data and transforming it into what we need for the fragment shader:

// The fragment shader gets interpolated versions of the positions, normals and texture
// coordinates calculated in the vertex shader.
struct FragmentIn 
{
    // The SV_POSITION semantic indicates the position of this pixel in screenspace.
    float4 pos : SV_POSITION;
    // Each vertex has a texture coordinate that is used to map from the 3D model to its 2D texture
    half2 uv : TEXCOORD0;
    // The normal gives us the direction direction of the surface at each vertex
    fixed3 normal : TEXCOORD1;
};

Now that we've got the input and output structures defined for the vertex shader, lets take a look at it:

// The vertex shader runs once for each vertex in a model
FragmentIn vertex( VertexIn v )
{
    FragmentIn o;
    // multiply the vertex position into camera space.
    o.pos = mul( UNITY_MATRIX_MVP, v.pos );

    // We don't need to do anything with the UV coordinates, they're correct in the model.
    o.uv = v.texcoord;
    
    // This if else block is used to create 2 versions of this shader
    #if _CAMERA_BASED_ENVIRONMENT_ON
    // If we want lighting based on the camera, we multiply the normal into view space
    o.normal = mul( UNITY_MATRIX_MV, float4( v.normal, 0 ));
    #else
    // Otherwise, we only multiply the normal into world space
    o.normal = mul( _Object2World, float4( v.normal, 0 ));
    #endif
    return o;
}

The purpose of the vertex shader is to transform the information in the models vertices so that it is relative to the camera, screen, or virtual world, as needed. For example, a basic 3D model is stored as a series of vertex positions relative to the origin of that model. To draw it to a screen, there are three steps that must be taken. First, every vertex must be transformed so that it is positioned accurately in the world. This transformation is refered to as the world or model transform. It then must be positioned relative to the camera. This transform is generally known as the view transform. Finally, the objects in 3D space in front of the camera are flattened to a 2D image with a transform known as the projection. Each of these transformations is represented with a 4x4 matrix. Because each of these matrices are the same for every vertex in a model, they are generally multiplied together ahead of time, to produce a single matrix which represents a transform whose result would be the same as doing each step in order. In the shader above, we use 3 different transforms. We use the ModelViewProjection matrix for the position of each vertex, because we want to calculate where they should appear on screen. In the #if/#else block, we use two different transform matrices. If we want the environmental lighting to move with the camera, we need to transform the vertex normal into the world, and then relative to the camera, but we don't want to flatten it to 2 dimensions with the projection. For the more traditional model, where the lighting doesn't move with the camera, we only need to multiply the vertex normal into the world.

The final step in the process is the fragment shader, which calculates and outputs the final color for each pixel covered by a model.

fixed4 fragment( FragmentIn i ) : COLOR
{
    // We start by loading a base diffuse color from the texture
    fixed4 finalColor = tex2D( _DIFFUSE_TEXTURE, i.uv );

    // Since the normals are interpolated, we re-normalize here to make sure it's a unit vector
    float3 normal = normalize( i.normal );

    // Now we sample the environment cube map using our normal.
    fixed4 environment = texCUBE( _ENVIRONMENT, normal );

    // We multiply the lighting from our cube map with our diffuse color to get our final appearance.
    finalColor *= environment;

    return finalColor;
}

The fragment shader is resposible for calculating the final color of a pixel in a model, and this one is actually very simple. It involves almost no calculation, and just uses the values calculated in the vertex shader to read from the input textures and calculate a final color. First, it uses the UV coordinates to sample the diffuse texture, which contains the basic color information for the model. Then it normalizes the input normal because the values calculated in the vertex shader are automatically interpolated between vertices, and this means that the normal passed to the pixel shader might not be a unit length vector anymore. We then use that corrected normal to sample our cubemap, and get the pixel in the cubemap which the normal is pointing towards. Finally, we multiply the value we loaded from the environment map with our diffuse color to get our final output color for the pixel.

The second pass will take a lot less time to cover. Now that we've created our toon shading model, the last step is to draw an outline around the character. There are a number of approaches for achieving this effect, but we're going to stick to a simple technique where we draw the model again slightly larger, and only draw the back faces, giving us an outline around our character. We do this by expanding each vertex slightly along its normal, with a pixel shader that simply returns black. When a users selects the outline shader for a material, the outline shader will make use of the base shader to provide the toon shading effect, then draw the outline as a second pass. We'll start out by taking a look at all of the setup code for the outline shader:

Shader "Toon/UnlitToonOutline" 
{
	// The outline shader includes all of the properties of the base shader so that they can be passed on to that shader when we invoke it
	// as a first pass from this one.
	Properties 
	{
		// The environment cubemap acts as the lighting environment in unlit toon shading.
		// Modifying this cubemap will change the appearance of lighting
		_ENVIRONMENT( "Environment Texture", CUBE ) = "" {} //0

		// Realistically the lighting environment is independent of the position of the camera, but it can preovide a more
		// cartoonish effect to have the lighting environment change with the orientation of the camera
		[MaterialToggle( _CAMERA_BASED_ENVIRONMENT_ON )] _CAMERA_ENVIRONMENT( "Use Camera Based Environment", float ) = 0 //1

		// The diffuse texture defines the basic color of the surface that will then be multiplied by the environment to get a final color.
		// The color in the diffuse texture represents the color of the surface under full, direct light.
		_DIFFUSE_TEXTURE( "Diffuse Texture", 2D ) = "white" {} //2

		// The outline thickness lets us adjust how much we expand along the models normals to draw the outline
		_OUTLINE_THICKNESS( "Outline Thickness", Range( 0, 1 )) = 0.1 //3
	}

	SubShader 
	{
		// This technique is intended for opaque models, we'd need to make some adjustments to handle transparency
		Tags { "RenderType"="Opaque" }
		// We don't need any lighting
		Lighting Off

		// Use the base pass from the base toon shader to draw the toon shaded model first
		UsePass "Toon/UnlitToonBase/BASE"

		Pass
		{
			// Since we're drawing an outline, we're only going to draw faces on the back side of the model
			Cull Front
			Name "OUTLINE"
			CGPROGRAM

				// tell Unity the names of our vertex and fragment shaders
				#pragma vertex vertex
				#pragma fragment fragment

				#include "UnityCG.cginc"

				// The only piece of info we need here is how much to expand the model.
				float _OUTLINE_THICKNESS;
				
				// The only information we need from vertices is their position and normal
				struct VertexIn 
				{
					float4 pos : POSITION;
					float3 normal : NORMAL;
				};
				
				// All we need for the pixel shading is the screen position.
				struct FragmentIn 
				{
					float4 pos : SV_POSITION;
				};
                //....
                

You'll notice that the properties block contains all of the properties of the base shader plus the additional property to allow customization of the outline thickness. It needs the properties of the base shader because it will actually use the base shader as a first pass, and needs to be able to provide the information needed by the base shader. The next difference is where the outline shader actually uses the base pass from the first shader:

// Use the base pass from the base toon shader to draw the toon shaded model first
		UsePass "Toon/UnlitToonBase/BASE"

The last few differences are that the pass defined in this shader culls front facing polygons and draws ones facing away from the camera, the additional variable for how much to expand the mesh, and less information for both the vertex and fragment shaders, because they are much simpler. For the vertex shader, the only information required is the vertex's position and it's normal. The fragment shader only needs the final position, because it simply returns black:

FragmentIn vertex( VertexIn v )
{
    FragmentIn o;
    // Here, we take each vertex and expand it along its normal, effectively enlarging the mesh.
    // This technique develops artifacts along hard edges, but it's very inexpensive
    float4 pos = v.pos +  float4( v.normal.xyz * _OUTLINE_THICKNESS * 0.01, 0);
    // Once we've expanded the vertex along its normal in model space, we multiply it by the model view projection matrix to put it into screen space.
    o.pos = mul( UNITY_MATRIX_MVP, pos );
    return o;
}

fixed4 fragment( FragmentIn i ) : COLOR
{
    // All that's left is to return black for pixels in our outline.
    return fixed4( 0, 0, 0, 1 );
}

The full source for each shader can be found here:
Base Shader
Outline Shader
A fully working demo project is availabe here:
Demo: http://www.voidstarsolutions.com/unity/toonblog