Today’s Question: In a recent eNewsletter you stated that in camera “With most cameras each individual ‘pixel’ on the image sensor is only recording a single color value (generally red, green, or blue)”. Can you expand on this and explain how we end up with all three color values for each pixel from such slim camera data?
Tim’s Quick Answer: There are certainly exceptions, but with most digital cameras the image sensor uses a sensor where each “pixel” is only recording a single color of light. Essentially each photosite on the sensor has a colored filter in front of it, so each photosite records only red, green, or blue light. Either in the camera (in the case of JPEG capture) or in post-processing (with RAW capture) the “other” values are calculated based on surrounding pixel values.
More Detail: It might seem a little outrageous that two-thirds of the information in a digital photograph is created after the capture, but it is true for the vast majority of digital camera sensors. And I can make the situation sound even worse.
The red, green, and blue photosites on the image sensor aren’t distributed in equal quantities in most cases. On a typical sensor half of the photosites are capturing green light, one-quarter are capturing red light, and one-quarter are capturing blue light. This is because the green light represents the most prominent light in the scene based on nature of human vision. One way to think of this is to consider that in an RGB image, the green channel will most often represent the best black and white image if you could only choose among the three channels to create a black and white image.
So, how is that “extra” information created out of thin air? To be sure, the process is complicated and there are many advanced algorithms at work behind the scenes when processing a RAW capture. But in basic concept it isn’t actually too complicated.
Instead of thinking about the final color photograph, think about the individual channels that comprise that color image. Each of the channels (red, green, and blue) are black and white images that reflect how much red, green, or blue light should be combined to create the color value for each pixel. You can get a sense of this by viewing the individual color channels for a color image using the Channels panel in Photoshop.
Imagine a black and white photo, consisting of many shades of gray. Now imagine that every other pixel is blank. That is what the unprocessed green channel looks like for a RAW capture, in terms of the image sensor only recording green light for half of the pixels on the image sensor. The software simply (I use the term loosely) needs to calculate the missing values in between each green pixel value. In theory you could just fill in the blanks with the average of the two adjoining pixel values, but of course in the real world the math is far more complicated.
Keep in mind also that besides having half of the pixel values for the green channel, the software processing a RAW capture also has red and blue values for the surrounding pixels. So there is a reasonable amount of information to provide an ability to infer pixel values.
Of course, for the red and blue channels the situation is a little more challenging, since fully three-quarters of the information is “missing” for those channels. But the same basic process applies (with complicated math in the background) and the “other” color values for each pixel can be calculated.
To be sure, in the early days of digital capture the task of “filling in the blanks” (a process that is generally referred to as demosaicing, by the way) wasn’t performed very well. Over the years the software for RAW processing has gotten much more advanced. This enables the initial color information to be calculated more accurately than was the case with earlier software, and it also means that greater detail and sharpness can be preserved.
I suppose it would be fair to say that in simple concept, the task of demosaicing is relatively straightforward. You “simply” need to fill in the blanks based on what you can infer from surrounding pixel values. But, of course, the math behind this processing is quite complicated. Thank goodness there are very smart software engineers who have tackled this task and provided us with a variety of high-quality RAW-processing software!