Walking in a Painting: Interior Daytime Lighting in Elsinore

Right now, our Unity adventure game Elsinore looks like this:

In this post, I'll be talking about how we ended up with this style, as well as our process for making it happen.

A Living Painting

When we were developing Elsinore's art style, we had only one main goal: we wanted the game to feel like a living painting.

Unlike a photograph, a painting is not always a perfect reflection of reality.  Some areas may be more detailed than others, and the colors and lighting might ignore optics in favor of expression.  Artists often do unrealistic things to make their paintings look better.  However, most modern rendering techniques focus on emulating the same optics used by cameras.  Since we're trying to achieve a different effect, we couldn't use them directly.

When you're working on this kind of problem, there are two fundamentally different approaches you can take.  The first is to rely on actual artists to hand-paint as much as possible.  Sure enough, when you want something to look like a painting, the easiest way to get the effect is by using an actual 2D painting of the scene as a backdrop.  This way you can get the most organic effects, but you're limited in how dynamic you can make your scenes in terms of camera movement, lighting and animation.

On the other end of the spectrum, you have the fully computational approach.  You create a full 3D scene and turn it into a painting using the rendering pipeline.  This way, you can make your scenes as dynamic as you want.  The downside is that it's very difficult to create organic effects with a shader.  Fancy shaders can get very expensive to compute, and you end up having to do a lot of modeling and texturing work anyway.

Having worked with both approaches, I like looking for creative ways to combine them.  That's exactly what we ended up doing for Elsinore.

3D Models on 2D Backdrops

As a small team, we didn't want to commit to major art style decisions until the rest of the game started coming together.  We did know that it would be important for our world to feel interesting and intuitive to explore, and for our characters to feel like a natural part of it.  With that in mind, we decided to model our characters in 3D for practical reasons: animations would be easier to make, change and reuse, and we wouldn't have to worry about camera angles or resolution until much later in development.

There was a lot of back-and-forth regarding the environments, but we had always been leaning towards 2D backdrops.  High-fidelity models of everything in the game were outside the scope of the project, and we didn't want to limit our target platforms by using expensive shaders.  Unfortunately, the 3D characters added a complication.  When we rendered them on top of 2D backdrops, it became obvious that they were very different in terms of style and level of detail.

We tried different lighting on our characters, but the contrast was still jarring.

The question became one of making 3D characters look like they belong in a 2D world.  We decided to do this with dynamic lighting for several reasons.  First of all, even if we couldn't get our characters to look exactly like our backgrounds, having both affected by the same light sources would have a unifying effect.  Second, dynamic lighting would make the world itself look less flat and more alive.  But the deciding factor was the importance of time in our game: if the direction of the light changed throughout the in-game day, it would serve as a valuable visual cue to the player.

What we ended up doing was creating low-poly models of the scene.  We allowed Unity to calculate a shadow map based on that geometry and hijacked it with a custom shader.  Using Unity's own shadow calculations, we interpolated between two hand-painted images to give the impression of dynamic, hand-painted lighting.  Read on for the technical details.

Our Process

Our art process starts with a 2D paintover of the room.  Once the team is happy with the look, we make a rough model of the room and project the UVs from the angle of our in-game camera.  The paintover is then applied to the model as a texture so that the model looks identical to the paintover when viewed from that camera angle.

Though things aren't so pretty from any other angle.

We then make two versions of the paintover texture - a "light" version and a "shadow" version.  In Unity, we have a directional light that changes in color and rotation relative to the game clock, simulating the sun.  We can access the shadow map data for this light inside a surface shader using the LIGHT_ATTENUATION(i) macro.  Based on this data, we linearly interpolate between the two textures.  Pixels that are completely lit sample from the "light" texture, and pixels that are completely occluded use the "shadow" texture.  Pixels inside the border of a soft shadow sample from both and split the difference based on the intensity of the light there.

Finally, we multiply the result by the color of the light itself.  What we end up with is a dynamic simulation of direct sunlight that looks hand-painted.

The shader itself is pretty basic.  The real key lies in the way the two textures are set up.  It isn't enough to make one a lighter version of the other.  The "light" texture is a painting of how the scene would look if it were completely lit from the primary light source.  The "shadow" version depicts the scene as though it were almost completely in shadow with subtle hints of ambient light.  In places where light would always be occluded, the two textures are identical.  By planning out smooth transitions between potentially lit and shadowed areas, we can make the dynamic shadows look much softer than they are computed to be.

By making the textures do the heavy lifting, we can afford to simplify the models.  Since the lighting is technically flat, it's enough if the model matches the general plane of the painted object.  The only places where we need a higher polygon count are those where the shadows need to follow a curve, such as the backlit curtains.  As long as the general shape of the object casting the shadow is correct, we only need to model the side that faces the camera.  And objects placed where they don't need to cast or receive shadows don't even need to be modeled.

Because of our isometric perspective, the relative positions of objects don't change during camera pans, so the illusion doesn't break down.  However, a perfectly orthographic projection has a lot of parallel lines that can look sterile, and the lack of foreshortening can seem unnatural.  Since this is antithetical to the organic, painterly look we want, we paint the scene before modeling it.  Imperfections in the paintover lead to distorted models, which we sometimes keep to make the scene look more "hand-painted."

Of course we fix the more blatant distortions like this table.

For objects that characters can stand behind, we overlay planes with an alpha channel.  These are multiplied by the color of the light to match the rest of the scene, but they are not affected by lighting.  Using these, we can also achieve other interesting effects, such as floating brush strokes.

Secondary Light

With this set-up, our scenes look good when there is a lot of light, but unfortunately, that isn't always the case.  When there are no windows in the path of the light, we're back to square one.  To maintain the depth of our environments throughout the day, we needed a secondary light source.

When you look at actual sunlit interiors, direct sunlight doesn't always play a big role.  The light outside gets diffused by clouds and reflected by most surfaces.  As a result, the indirect sunlight that enters through a window doesn't seem like it's coming from a particular direction.  Though the intensity of the light varies, most windows actually act like area lights.  In rooms with a lot of windows, light can seem like it's coming from everywhere, resulting in a lot of very soft, subtle shadows.  These shadows overlap at the point where two surfaces touch.

Obviously, modeling all of that dynamically would be unreasonable, but there is a useful takeaway.  Because our feet touch the floor, the overlapping shadows right around them are relatively dark.  We can simulate this effect with a second, low-intensity directional light that shines from above.

With this combination of lights, we still get our dramatic sunlight, and characters look like they belong in the scene throughout the day.  As a finishing touch, we place non-shadowing point lights that only affect characters at each window.

Which Objects Cast Shadows?

Because we're using lighting to unify the scene, most visible objects both cast and receive shadows.  The only exceptions are objects that will never be in the path of the sunlight and objects that will never cast a shadow on a character from either light.  But what about invisible objects?

When the player enters a room in Elsinore, we cut away the walls to show the inside of that room.  The walls crumble as a visual metaphor of tragedy - the game's core theme.  Of course the castle isn't actually being destroyed or rebuilt every time the player walks through a door.  The missing architecture is still technically there, and it should affect the lighting regardless of its own visibility.  The invisible parts, however, shouldn't be affected by lighting.

To accomplish this, we actuallymake two sets of wall models.  First, the fully intact walls that are visible when the player is in front of them react fully to lighting.  When the player walks into the room behind them, we make these walls invisible, but they still cast shadows.  We also enable the "crumbled" walls, which also react to lighting.  These are positioned slightly in front of the first walls so as not to be shadowed by them.

Inside and outside the bedroom during sunset.

We treat ceilings more or less the same way, with a caveat.  Since the secondary light from above approximates light already inside a room, ceilings need to occlude sunlight but not secondary light.  In Unity, we handle this case by putting ceilings on a special layer that the secondary light ignores.

Conclusion

Using these methods, we were able to unify 3D characters and 2D environments, add a level of depth to our world, and all while maintaining a hand-painted look.  We're well aware that our "lighting model" isn't, strictly speaking, correct - and we're perfectly okay with that.  By compromising on "correctness," we were able to benefit from both hand painting and simulated lighting with only a small amount of extra work.

These screenshots are by no means final.  We are actively working to improve both the lighting and the paintovers themselves.  From here, we'll also be looking at exterior and nighttime lighting, both of which will likely involve variations on the same approach.

If you're interested in seeing more progress updates on Elsinore, visit our site or follow us on Twitter!