Breaking down my approach for writing a Unity shader to replace colours using a pre-configured palette and the interesting discoveries I made along the way.
[Reposted from www.drunkenmonkeystyle.com]
Recently at work I was developing a shader for doing colour replacement. I wanted to create a palette of five colours and have a shader that would replace colours in a source image, video, model, or work as a post-processing screen effect. After four versions I got the types of results I was hoping for and I wanted to share a breakdown of the technique I was using to do the replacements and note a few interesting things I learned along the way including the mystery of these magic numbers: 0.3, 0.59, 0.11.
The first version of the shader was very simple. It looked at the rgb colour of the current pixel and then tried to match it to the closest matching colour from the pallette. The first really interesting thing I learned while building this version was to think of RGB values like a 3D point. This was significant for me because I previously didn’t have a good way to think of colour values in a way that could be compared but after stumbling on a forum post where someone said to think of the RGB values this way it made it really easy for me to visualize the data. At this point I wrote some shader code to take the current pixel colour and compare the distance to the palette colours and then replace with the closest match. This worked as a basic first approach but there was a lot of detail loss
For the second version I tried adding a parameter to the shader called Strength which was used to blend the replacement colour back into the original image. This way I could turn down the blending strength to retain some of the detail. The result was a slight but noticeable improvement but still not good enough for what I was going for.
In version three I started taking a different approach after having the idea that if I convert the image to grayscale first there would be a better match for colour replacement, better blending, and ultimately better detail retention on the overall picture. In figuring out how to convert an image to grayscale I found a line of code that had three magic numbers.
float3 greyScaledColour = dot(sampled.rgb, float3(0.3, 0.59, 0.11));
I didn’t understand what this code was doing but it clearly resulted in a grayscale image so I integrated it into my shader. It really bugged me that I was using a piece of code I didn’t understand so I showed it to my team lead at work and we began a discussion and some investigation to find the source of the numbers. After some back and forth my lead had an idea that was quickly verified and while still not fully understanding why this results in a grayscale pixel when using the dot product of the sampled colour against these three magic numbers what we did find is the source of the numbers. These numbers correlate to the number of Red, Green, and Blue rods and cones we have in our eyes. WILD!
Here’s some links to some of the info we read that led to this understanding.
So after converting the image to grayscale and then doing the colour replacement there was a further improvement as I had hoped. If you compare the v3 image to the v2 image it’s hard to notice the difference at first but if you look at the head of Pennywise in v3 you should notice that the colour gradiation resulted in better groupings of colours whereas in v2 there’s a split on the forehead between blue and pink areas that seems unnatural even in this extreme palette.
At this point I was pretty happy with the shader but had one more idea that I thought might yield an even better result, which proved to be true. The final technique worked better than the previous versions and gave you a lot of control over how the colour replacements are done. So here’s what I did…
I realized that I could treat the grayscale colour values as a single number in the range of 0 to 1. I could also take the 5 colour palette and treat those values as a range of 0 to 1. Now instead of doing distance checks on the three values I would just be matching the 0 to 1 values between the grayscale and the replacement palette. From there the final step was to add some sliders to the shader so you could control the thresholds for each colour in the palette which gave you much better control of which colours replaced dark and light areas in the image. Once that was all done then the final value was blended as in the previous versions for the end result. I am extremely happy how this shader turned out and am already using modified versions of it in a number of projects.
You May Also Like
