Guide: Demosaicing and Colour in Photos

The Basics

The vast majority of digital cameras use sensors that rely on a 'mosaic', with each individual pixel only storing data about how much red, green or blue light is falling on it.

An algorithm then 'recreates' the quantities of the other two colours, based on the values recorded by nearby pixels.

This process, known as 'demosaicing', can introduce visual artefacts, including one known as moiré, the presence of coloured lines instead of sharp areas of black and white.

Sensor Structure

Perhaps you remember this diagram from some of other photography guides:


On the left is the CMOS sensor from a digital SLR, in the middle, the CCD found on a point-and-shoot and finally, on the right, Fujifilm's Super CCD EXR.  All three sensors are laid out differently, but they all have something in common: a coloured mosaic.  Each pixel is only sensitive to a single colour of light (red, green or blue) and completely ignores the others.  For instance, in our first example, the pixel at the top left is only sensitive to green light.  If any blue or red light falls on it, it won't react.

Why make sensors like that?  Sensors can only record the intensity of light, often relying on infrared.  To recreate colour, manufacturers needed to find a way to separate out the different wavelengths, and they started by putting a filter in front of each pixel.  It's a little bit like the different cone cells in the retina of your eye, each one of which is only sensitive to orange, green or indigo light.

Various layouts have been tried over the years, but the most common is the Bayer filter, found on both CCD and CMOS sensors.  The only exception is Super CCD sensors, which use a special matrix based on the same principle.

The key feature is that there are twice as many green pixels as there are either red or blue.  That's to compensate for the human eye, which is more sensitive to yellow-green light than other parts of the spectrum.

The Mosaic

The upshot of all of this is that the sensor doesn't see a perfect representation of reality.  Compare the actual photo (left) to what the sensor actually recorded (right).



For every pixel, only one channel (red, green or blue) is recorded.  When we zoom into the picture, the Bayer filter is very clear in white areas, but replaced by gaps in areas of bright colour: in the red area, for instance, the red-sensitive pixels are red, but the green and blue pixels look black.


What happens is that the sensor records a grid of numbers, each one corresponding to a single channel at one pixel.  In this example, we've used percentages of colour to cover an area in the middle of the folding colour chart.  It's this table of numbers that is stored in a RAW file in cameras that have this feature.

Demosaicing

Of course, a digital image actually has information for three channels per pixel: red, green and blue, different combinations of which create different colours.  That means we need three different numbers for each pixel.  Calculating the other figures by looking at the values recorded in adjoining pixels is called 'demosaicing'.


To take our example, we can look at the values recorded from the neighbouring pixels, comparing either two or four numbers to produce a table like the one on the right.  Here, the figures actually recorded are in bold, with the calculated values in ordinary type.  What's important is that each cell now has information containing all three colours, and is now a bitmap image that can be recorded as a JPEG and displayed onscreen.

Moiré

Demosaicing is so important because during this processing, some problems can creep in, including the most well-known defect, moiré.

Here's an extreme example that's a perfect illustration because it only works in theory.


If we photograph some black and white lines which just happen to be spaced out on exactly the same grid as the pixels in our sensor, you can see what happens.  The blue pixels are all in front of black lines and record their light level as 0, while the red pixels see white lines and all record a light level of 100.  The green pixels alternate between the two.

When this data is demosaiced and average values are calculated, chromatic interference results and instead of black and white lines, we end up with a red, orange an yellow grid.

This can happen in real life when small details are close to matching the layout of the pixels on, the sensor.  Common problems include tiles or shutters on a faraway house, as well as hair, leaves and other fine detail.

There are two ways of reducing these artefacts.

The first is to use a low-pass filter, which deliberately blurs the image so that nothing smaller than two pixels across is visible.  As a result, our black and white lines become a block of grey.  We lose this fine detail, but the artefacts are gone too.

The second solution is to adjust the demosaicing algorithm, adding for instance the ability to detect the presence of lines.  In our example, the software would spot the vertical lines and ignore the data from either side, and rely on the values from the pixels immediately above and below.  The white stripes would appear yellow (because they're a mix of red and green, with no blue data available), but the black lines would stay black, which is already an improvement.

An alternative method is to study the information from three or four nearby pixels, instead of just one.  Like with a low-pass filter, the problem is that detail is reduced, but so are artefacts.

These extracts show the difference between four different demosaicing algorithms working on the same RAW file: AHD, four colour VNG, bilinear (in the DCraw software) and the one offered in Bibble 5.  Bibble produces the sharpest photo (look at the contours on the map) but there's still moiré in the connections for the computer chip at the bottom left.  AHD manages to capture the blue reflection at the top left, that VNG wipes out, and while the bilinear algorithm eliminates moiré, it also blurs the contours most.

In general, the trade-offs between losing details and reducing artefacts is complex.  That's why photographers like RAW files so much: not only can you use more powerful demosaicing algorithms on a computer, you can also choose the right algorithm for the job according to the photograph you're working with.

As an aside, this is one of the rare situations where a very high resolution sensor is better than one with fewer pixels.  On the one hand, more pixels reduce the possibility of moiré creeping in because details will need to be much smaller before they line up with the pixels, and on the other, a much less destructive low-pass filter is needed.