Intel View in Mafia III

Introduction
Mafia III is an action-adventure video game set in the fictional city of New Bordeaux. The protagonist Lincoln Clay fights to take power away from the Italian mob. Mafia III: Developer-Narrated Gameplay: https://youtu.be/_Xtq18Vmh3s

"Intel View" is one of the tools the player has, that helps them survive this hostile world. It highlights certain types of characters and objects in the scene, e.g. enemies, cops, medicine cabinets, weapon lockers, etc. Thus, making it easier for the player to find/track them and take appropriate action. Demo of Intel View in Mafia III: https://youtu.be/VDSwBzKSb1M

In this article we will describe the way Intel View works, and how it was implemented (mostly about the rendering).

Feature requirements
Any object in the game world can be tagged to be highlighted in this mode, and un-tagged when not necessary anymore. When the mode is enabled, all tagged objects in a certain radius from the player should be rendered un-occluded (visible through other objects) and colorized/ highlighted/ silhouetted to make them pop out of the scene.

The whole screen should be desaturated and darkened, except these objects. Different types of objects should show different colored outlines. (e.g. cops are blue, enemies are red, etc.). The intensity of highlight should decrease as the object gets farther away from the player.

Gameplay
Gameplay code maintains lists of "special objects" that it needs to manage (tag and un-tag) as per scenarios in game. It sets the category for an object:
0: Default (i.e. not highlighted)
1,2,3: Intel View objects. Color of highlight can be set for each, by artists.

Rendering
Here is a screenshot in normal game view. 

In this scene we will mark 4 objects for Intel View: the player's red car, the cop and 2 mobsters on the street. Below we describe the steps that happen in 1 frame of the game.
(Input: List of Intel View objects, along with their categories)

1. Calculate the screen-space AABB and distance from player for each object. 
2. Render the AABBs, with the color's intensity representing the distance. (we don't use per-pixel depth because we want a uniform color for the entire object.) This will be used to fade the highlight with distance.

(Note: Code Samples are provided for extra detail. You can skip them if you just want to understand the high level logic)

Code Sample 1: Render highlight object AABBs

// Fill Vertex buffer of Highlight Objects' screen space AABBs
for each object
{
    outlineColorDepthFadeFactor = (maxHighlightDistance - object.m_DistanceFromCamera) / (maxHighlightDistance - minHighlightDistance); // between 0 and 1
    for each corner of object's ScreenSpace AABB
    {
        vertexData.WritePosition2D(corner);
        vertexData.WriteColor(outlineColorDepthFadeFactor);
    }
}
 
// Render the screen space AABBs of objects with object depth as pixel color (just 1 channel: red). (Figure-1):
RenderScreenSpaceMaterial(vertexColorMaterial, vertexData);

Object AABB & Distance Texture (Figure-1):

3. Render the objects to G-Buffer, and set a different stencil value for each category of objects. This will be used to associate each pixel in the image with its Intel View category.

Code Sample 2: Render highlight objects and silhouettes

primaryColors[NUM_HIGHLIGHT_OBJECT_CATEGORIES]  = { Red, Green, Blue };
stencilBitValues[NUM_HIGHLIGHT_OBJECT_CATEGORIES]   = { 01000000, 10000000, 11000000 };
 
for each category
{
    // Regular object shading
    Set stencil buffer value to stencilBitValues[category], for each pixel written (Figure-3)
    Render objects in category to G Buffer (Figure-2)
    
    // Render object mask
    SetObjectColor(primaryColors[category])
    Set stencil logic so rendering will update only pixels that have stencil value stencilBitValues[category] (i.e. selective drawing of color on objects of this category)
    Enable blend, to add pixel values of any previous existing color (to show intel objects even when overlapping)
    RenderScreenSpaceMaterial using singleColorMaterial to Highlight Object Mask (Figure-4)
}
 
// singleColorMaterial pixel shader
HighlightObjectColorFadeDepthFactor = Figure 1
FinalShading = HighlightObjectColorFadeDepthFactor * ShaderParam_objectColor; // (Figure-4)

G-Buffer rendered scene (Figure-2):

Stencil Buffer (Figure-3):

4. We generate a texture combining the info from Figure 1 and 3. Primary colors (R,G,B) denote each object category.
Highlight Object Mask (Figure-4):

Post Process
5. Blur Highlight Object Mask to get a slightly larger silhouette of each object.
Blurred Highlight Object Mask (Figure-5):

6. Subtract the textures (5-4) to get the silhouette edge (Figure-6): 

7. Decrease saturation and brightness of the G-Buffer image (Figure-2)
8. Replace the primary colors (RGB) in Figure-6 with the artist defined colors per category, and add it to get the final output: Intel View (Figure-7):

Code Sample 3: Post process pseudcode

//These textures are RGBA8 at half the game resolution. RT = Render Target
DownSampleRT(innerHighlightMaskRT, innerHighlightMaskHalfRT); // innerHighlightMaskRT = (Figure-4);
BlurRTGaussianPixelSize(innerHighlightMaskHalfRT, tempRT, 1, blur parameters 1);
BlurRTGaussianHorizontal(innerHighlightMaskHalfRT, tempRT, blur parameters 2);
BlurRTGaussianVertical(tempRT, outerHighlightMaskRT, blur parameters 2); // outerHighlightMaskRT = (Figure-5)
 
// Object Highlight pixel shader
SceneColor = G-Buffer Texture (Figure-2)
InnerHighlightMask = Highlight Object Mask (Figure-4)
OuterHighlightMask = Blurred Highlight Object Mask (Figure-5)
highlightMask.rgb = ShaderParam_OuterHighlightScale * OuterHighlightMask.rgb - ShaderParam_InnerHighlightScale * InnerHighlightMask.rgb; // (Figure-6)
HighlightColor = (highlightMask.r * ShaderParam_MaterialColor1 + highlightMask.g * ShaderParam_MaterialColor2 + highlightMask.b * ShaderParam_MaterialColor3);
float3 sceneColorHsv = RGBtoHSV(SceneColor.rgb);
sceneColorHsv.y *= ShaderParam_SaturationScale;
sceneColorHsv.z *= ShaderParam_BrightnessScale;
ColorCorrection = HSVtoRGB(sceneColorHsv);
FinalShading = ColorCorrection + HighlightColor; // (Figure-7)

Optimization
Reduced highlight mask texture sizes and tweaked blur parameters to make shaders faster.
Before: Render Post Process: 4 blurs + object_highlight = 2370 us (micro seconds)
After: Render Post Process: downsample Render Target + 4 blurs + object_highlight = 586 us

Misc notes
Designers would have liked to have more categories of objects (each with a different color), but this tech allowed only 3. Supporting more categories would have been worse for performance, and led to a more complicated algorithm as well. So in the end designers worked with what we could give them.

Thank you for reading. Questions, comments and feedback are welcome :)