Unreal Engine 4 Cel Shading Tutorial

Tommy Tran

Unreal Engine 4 Cel Shading Tutorial

Developing realistic-looking games is Unreal Engine 4 is easy due to physically based rendering. This model approximates how light interacts with materials, resulting in realistic images. However, if you want to develop a game with a stylized look, you need to explore other techniques.

One of the ways to achieve a stylized look is to use cel shading (also known as toon shading). This technique mimics the shading typically used in cartoons and anime. You can see examples of this in games such as Jet Set Radio, The Legend of Zelda: The Wind Waker and Gravity Rush.

In this tutorial, you will learn how to:

  • Create and use a post process material
  • Create a cel shader
  • Isolate the cel shader to specific meshes
  • Control the color bands using lookup tables
Note: This tutorial assumes you already know the basics of using Unreal Engine. If you are new to Unreal Engine, you should go through our 10-part Unreal Engine for Beginners tutorial series first.
Note: This tutorial is part of a 6-part tutorial series on Diving Deeper into Unreal Engine:

Getting Started

Download the starter project and unzip it. Navigate to the project folder and open CelShader.uproject. You will see the following scene:

unreal engine 4 cel shading

This is the character you will be cel shading. Before you start, you should know what cel shading is.

What is Cel Shading?

Cel shading is when you render something using multiple bands of color rather than a continuous gradient.

unreal engine 4 cel shading

Below is an example of cel shading in The Legend of Zelda: Breath of the Wild. Note that only the character has cel shading while the background does not.

unreal engine 4 cel shading

In this image, there are three bands. One for shadows, one for midtones and one for highlights.

A common misconception is that if something has outlines, it is cel shaded. An example of this is Borderlands. Although this game has a stylized look, it is not cel shaded. You can see this in the image below. Notice that the character’s shading does not utilize bands of color.

unreal engine 4 cel shading

Even though outlines are not cel shading, it is common to combine them together. This helps an image look like it was drawn or inked. You will see this a lot in anime-styled games such as Guilty Gear Xrd and Dragon Ball FighterZ.

unreal engine 4 cel shading

In the next section, you will learn how you can implement cel shading.

Cel Shading Method

The most common method is to compare the surface direction (known as the normal) and the light direction. By calculating the dot product between the normal and light direction, you will get a value between -1 and 1.

A value of -1 means the surface and light are facing opposite directions. 0 means they are perpendicular to each other. 1 means they are facing the same direction.

unreal engine 4 cel shading

By thresholding the dot product, you can create multiple bands. For example, you can assign a darker color if the dot product is higher than -0.8. If the dot product is lower than -0.8, assign a light color. This will create a two band cel shader.

unreal engine 4 cel shading

The limitation with this method is that other lights cannot affect cel shaded objects. Also, objects can not cast shadows on cel shaded objects.

unreal engine 4 cel shading

To fix this, you need to use a different method. Instead of calculating the dot product, you calculate how lit a surface is. You can then use this value during thresholding instead of the dot product.

unreal engine 4 cel shading

Now that you know what a cel shader is and how it works, it’s time to create one.

Creating the Cel Shader

The cel shading in this tutorial is a post process effect. Post processing allows you to alter the image after the engine has finished rendering it. Common uses for post processing are depth of field, motion blur and bloom.

To create your own post process effect, you need to use a post process material. Navigate to the Materials folder and create a new Material. Rename it to PP_CelShader and then open it.

To convert a material to a post process material, you need to change its domain. Go to the Details panel and change Material Domain to Post Process.

unreal engine 4 cel shading

The first step in creating the cel shader is to calculate how lit each pixel is. We’ll call this the lighting buffer.

Calculating the Lighting Buffer

When Unreal renders an image to the screen, it stores passes to buffers. To calculate the lighting buffer, you will need to access two of these buffers:

  1. Post Process Input: Once Unreal has performed lighting and post processing, it will store the image to this buffer. This is what Unreal will display to the player if you do not perform further post processing.
  2. Diffuse Color: This is the scene without any lighting and post processing. It will contain the diffuse colors of everything on-screen.

unreal engine 4 cel shading

To access these buffers, you need to use the SceneTexture node. Create one and with it selected, go to the Details panel. To access the Post Process Input buffer, change Scene Texture Id to PostProcessInput0.

unreal engine 4 cel shading

To access Diffuse Color, create another SceneTexture node. Change its Scene Texture Id to DiffuseColor.

unreal engine 4 cel shading

The lighting buffer should only contain grayscale values (which describe how lit something is). This means you do not need the color information from both buffers. To discard the color, connect the Color output of both SceneTexture nodes to a Desaturation. This will completely desaturate both buffers.

unreal engine 4 cel shading

To calculate the lighting buffer, simply divide SceneTexture:PostProcessInput0 by SceneTexture:DiffuseColor. Order is important here!

unreal engine 4 cel shading

Afterwards, use a Clamp so the values stay within the 0 to 1 range. This makes it easier to perform thresholding since you know your possible values.

unreal engine 4 cel shading

Here is a visualization of the lighting buffer:

unreal engine 4 cel shading

As you can see, the lit areas are closer to white and the unlit areas are closer to black.

Next, you will use the lighting buffer to create a threshold.

Creating a Threshold

For this cel shader, any pixel with a value greater than 0.5 will use the normal diffuse color. Pixels with a value less than 0.5 will use the diffuse color at half brightness.

First, create an If node. This will let you compare two values. You can then specify different outputs depending on the comparison result.

unreal engine 4 cel shading

Next, connect the Clamp to the A input. Afterwards, create a Constant with a value of 0.5 and plug it into the B input.

unreal engine 4 cel shading

Note: You can change the value of the B input to change the threshold.

To get the colors, create a SceneTexture and set its Scene Texture Id to Diffuse Color. Afterwards, multiply Color by 0.5 to get the diffuse at half brightness.

unreal engine 4 cel shading

Finally, connect everything like so:

unreal engine 4 cel shading

Summary:

  1. Desaturation will convert Post Process Input and Diffuse Color to grayscale images
  2. Divide will divide Post Process Input by Diffuse Color. This will give you the lighting buffer.
  3. The Clamp will keep the values within the 0 to 1 range
  4. The If will output the normal diffuse color if the lighting value is greater than 0.5. If it is less than 0.5 it will output the diffuse color at half brightness.

Now that you have your cel shader, you need to apply it to the scene.

Using Post Process Materials

To use post process materials, you need to create a Post Process Volume. These are commonly used to control post process effects such as white balance, saturation and contrast.

Click Apply and then go back to the main editor. To create a Post Process Volume, go to the Modes panel and select the Volumes category. Afterwards, drag a Post Process Volume into the Viewport to create one.

unreal engine 4 cel shading

Now you need to tell the Post Process Volume to use the cel shader. With the Post Process Volume selected, go to the Details panel. Afterwards, locate Rendering Features\Post Process Materials and click the + icon. This will add a new entry to the array.

unreal engine 4 cel shading

Next, click the Choose drop-down and select Asset Reference.

unreal engine 4 cel shading

This will allow you to select a material. Click the None drop-down and select PP_CelShader.

unreal engine 4 cel shading

By default, Post Process Volumes will only take affect if you are inside of it. However, in this case, you want it to affect the entire world. To do this, scroll down to Post Process Volume Settings and enable Infinite Extent (Unbound).

unreal engine 4 cel shading

Now that the cel shader is being applied to the entire image, you will see this:

unreal engine 4 cel shading

"Hang on, that doesn’t look like the cel shader you showed earlier!"

The main reason it looks different is because the engine is applying the cel shader after tonemapping. To fix this, you need to tell the engine to apply the cel shader before tonemapping.

Applying Cel Shading Before Tonemapping

Before displaying an image to the player, Unreal performs a process called tonemapping. One of the reasons for tonemapping is to make the image look more natural. It does this by taking an input color and then uses a curve to shift it to a new value.

Below are two images of the Post Process Input buffer. One before tonemapping and one after tonemapping:

unreal engine 4 cel shading

As you can see, the highlights before tonemapping are too bright. However, after tonemapping, the bright areas are softer.

Tonemapping is fine in most cases. However, since you are using the Post Process Input buffer in a calculation, you need its original values.

Open PP_CelShader and make sure you do not have anything selected. Afterwards, go to the Details panel and locate the Post Process Material section. Set Blendable Location to Before Tonemapping.

unreal engine 4 cel shading

Click Apply and then go back to the main editor. The colors are looking a lot better now!

unreal engine 4 cel shading

In the next section, you will learn how to only apply cel shading to specific objects.

Isolating the Cel Shader

To isolate post process effects, you need to use a feature called Custom Depth. Like Post Process Input and Diffuse Color, this is a buffer you can use in post process materials.

Before you learn what Custom Depth is, you should know what the Scene Depth buffer is. Scene Depth stores how far each pixel is from the camera plane. Here is a visualization of Scene Depth:

unreal engine 4 cel shading

Custom Depth stores the same information but only for meshes you specify. Here is a visualization of it with the viking rendered to Custom Depth:

unreal engine 4 cel shading

By comparing Scene Depth against Custom Depth, you can isolate effects. If Scene Depth is less than Custom Depth, use the normal image. If Scene Depth is greater than Custom Depth, use the cel shaded image.

The first step is to render the viking to Custom Depth.

Using Custom Depth

Go to the World Outliner and select SK_Viking. Afterwards, go to the Details panel and locate the Rendering section. Next, enable Render CustomDepth Pass.

unreal engine 4 cel shading

Next, you need to perform the depth comparison. Open PP_CelShader and create the following setup:

unreal engine 4 cel shading

Note: The Mask (R) nodes are Component Masks. These allow you to convert multi-channel data into scalars. The reason you need to mask Scene Depth and Custom Depth is because the If node only accepts scalars for the A and B inputs.

Afterwards, connect the output of the cel shading network to A > B. Finally connect the output of the If you just made into Emissive Color.

unreal engine 4 cel shading

Now, only meshes that rendered to Custom Depth will have cel shading.

Click Apply and then go back to the main editor. You will see that only the viking has cel shading now.

unreal engine 4 cel shading

The cel shader is working great but it’s a bit simple. What if you want to have more than two bands? What if you want to create a softer transition between bands? You can accomplish this by using lookup tables (LUT).

What is a Lookup Table?

When you were young, you learned what multiplication is. However, a young mind might not have been able to perform these calculations. Instead of calculating, you might have used a multiplication table to "look up" the answers.

unreal engine 4 cel shading

This is essentially what a LUT is. It is an array of values (usually precomputed) which you can access with inputs. In the case with the multiplication table, the inputs are the multiplier and multiplicand.

In the context of this cel shader, a LUT is a texture with some sort of gradient. Here are four examples of what a LUT could look like:

unreal engine 4 cel shading

Currently, the way you are calculating the shadow color is by multiplying the diffuse color by 0.5. Instead of multiplying by a constant 0.5, you will use a value from the LUT. By doing this, you can control the number of bands and their transitions. You can get an idea of how the shading would look by the appearance of the LUT.

Before you use a LUT, you need to change some of its texture settings.

Changing LUT Settings

Navigate to the Textures folder and open T_Lut_01. This is what the LUT looks like:

unreal engine 4 cel shading

The first setting you need to change is sRGB. When rendering, Unreal will convert any textures with sRGB enabled to linear color. Basically, this makes it easier for Unreal to perform rendering calculations.

The sRGB setting is good for textures which describe appearance. However, textures such as normal maps and LUTs hold values meant for math calculations. As such, Unreal should assume their values are already correct. By disabling sRGB, Unreal will not perform the conversion to linear color.

To do this, uncheck the sRGB checkbox. You can find this setting under the Texture section.

unreal engine 4 cel shading

The next setting you need to change is how the texture tiles. Since you are not displaying this texture, there is no need to tile it. Furthermore, leaving tiling enabled will introduce issues when sampling from the edges. For example, if you were to sample a pixel from the left edge, it will try to blend to the right edge because of tiling.

To disable tiling, change X-axis Tiling Method to Clamp. Do the same for Y-axis Tiling Method.

unreal engine 4 cel shading

That’s all for the settings. Now you need to use the LUT in the post process material.

Using LUTs

Close T_Lut_01 and open PP_CelShader. First, delete the highlighted nodes:

unreal engine 4 cel shading

Next, create a Texture Sample and change its Texture to T_Lut_01. This LUT will create three bands with a soft transition.

unreal engine 4 cel shading

If you remember, LUTs take inputs to determine which value to output. In this case, you will use the lighting buffer as the input.

To do this, connect the Clamp to the UVs of the Texture Sample.

unreal engine 4 cel shading

This works because the lighting buffer and texture coordinates are in the range of 0 to 1. For example, if a pixel from the lighting buffer is 0.5, the LUT will output the pixel value from the middle of the texture.

Next, you need to multiply the diffuse color with the LUT. To do this, recreate the following setup:

unreal engine 4 cel shading

The reason for the Append is to convert the Texture Sample’s output to a four-channel vector. You need to do this because you cannot multiply a three-channel vector by a four-channel vector (SceneTexture).

Finally, connect everything like so:

unreal engine 4 cel shading

Now, instead of multiplying diffuse color by a constant, you are multiplying by a value from the LUT. This controls how many bands there are and their transitions (depending on the LUT). The lighting buffer determines which value the LUT will output.

Click Apply and then close PP_CelShader. The shading will now have three bands with a softer transition between bands.

unreal engine 4 cel shading

Below is a comparison of what alternative LUTs would look like. These LUTs are also included in the project.

unreal engine 4 cel shading

Where to Go From Here?

You can download the completed project here.

As you can see, post process materials are very powerful. They allow you to create a lot of realistic and stylized effects. If you’d like to learn more about post processing, check out the Post Process Effects documentation.

If there are any effects you’d like to me cover, let me know in the comments below!

Tommy Tran

When Tommy is not doing work, he likes to lose in Super Smash Bros. You can reach out to him through email.

Other Items of Interest

Save time.
Learn more with our video courses.

raywenderlich.com Weekly

Sign up to receive the latest tutorials from raywenderlich.com each week, and receive a free epic-length tutorial as a bonus!

Advertise with Us!

PragmaConf 2016 Come check out Alt U

Our Books

Our Team

Video Team

... 27 total!

iOS Team

... 84 total!

Android Team

... 44 total!

Unity Team

... 16 total!

Articles Team

... 4 total!

Resident Authors Team

... 32 total!

Podcast Team

... 8 total!

Recruitment Team

... 8 total!

Illustration Team

... 4 total!