How to measure noise in digital camera raw images

In my previous article I looked at noise in digital images and used ImageJ to produce plots of noise profiles from three different cameras.

Since then I have developed a method which permits accurate measurements to be made without special equipment and using entirely free software. Raw files were processed in RawTherapee, Darktable or ImageJ (using the DCRAW plugin). ImageJ is perhaps now better known as part of the Fiji distribution (Fiji Is Just ImageJ – Fiji is a distribution of ImageJ which includes many useful plugins). ImageJ was used for analysis. It is available for Linux, Mac and Windows.

Selection_026

The present article will compare the results of noise measurements on three cameras, and will explore the factors affecting noise levels and relate measured results to the theory. I hope to compare a few other cameras in a future article. However, I have tried to include sufficient detail of the method to allow you to test your own.

The term “signal to noise ratio” (SNR) is sometimes used, but is more appropriate to audio processing. I decided that a more useful graph would be a plot of noise versus signal, that is noise versus pixel value. (In the terminology of mathematical statistics Standard Deviation versus Mean). If we are considering 8-bit files this would mean a pixel value or signal in the range 0 (black) to 255 (white). The raw files will always have a bit depth greater than eight but it is convenient to think of the range being 0 to 255. (This can be thought of as referring to the most significant 8 bits). All noise values in this article are quoted in these units, 0 to 255 covering black to white.

Photographing the target

An easy way to get a rough idea of the shape of the noise graph (and incidentally to get an insight into the inverse gamma curve applied in processing) is to photograph a uniformly illuminated white card with exposures at one stop (or smaller) intervals. In every case though, it is important that the image is slightly blurred so as not to record the surface texture of the card. The best target is, however, a set of grey patches from black to white.

I was considering using a print of my own monitor test card when I came across the article by Stephen Burch. He compares various Canon cameras after processing the images in Adobe Camera Raw and alternatively in Canon’s own DPP software. Stephen offers a downloadable tiff file of his test card which I used in my own experiments. I modified it after printing by drawing lines to facilitate locating the patch boundaries on underexposed images.

The test card should be printed on photo quality paper but there is no need for accuracy, in that any colour profile can be used as it only affects the positioning of the sample points, not the accuracy of the noise measurement. When photographing the test card even illumination is important as any variation across a sample would give a higher noise measurement. I illuminated the test card with a 6500K CFL bulb in a photoflood reflector positioned close to the camera. Window light from an overcast sky would have about the same colour temperature, though this parameter is not particularly important. It is important to use manual camera settings and ensure that the image is slightly blurred for each shot.

Image processing and noise measurement

Being a fan of Linux, I first processed some images in RawTherapee and in Darktable, in each case setting a “zero processing profile”. This meant that anything that could be switched off, noise smoothing, sharpening, vibrance etc. was switched off. The resulting image was saved as a tiff file and opened in ImageJ for analysis. It was reassuring to find that this gave identical results in the two raw processing applications.

ImageJ in use

Analysis of a typical image in ImageJ showing a sample of 100 x 100 pixels selected – Target is by Stephen Burch with my lines added

To Compare two cameras

The fastest measurement method, however, is to use the DCRaw plugin in ImageJ and selecting the demosaicing method. (Menu: Plugins > Input-Output > DCRaw Reader). From the darkest patch select a square sample of, say, 100 x 100 = 10,000 pixels. A sample large enough to satisfy any statistician! From the menu select Analyse > Measure or Click Ctrl+M and a table opens with sample number, size, mean, standard deviation, minimum and maximum. Move the sample square to the next patch (no need to redraw it), click Ctrl+M and the second result is tabulated. Repeat for all the patches and you have a table of ten samples which can be saved as an xls file or just as easily copied and pasted into LibreOffice Calc. A plot of standard deviation versus mean (noise versus signal) is then easily plotted.

I discovered a very smart graph plotting program, QtiPlot, (available for Linux, Mac and Windows) which I used for all the graphs in this article.

Results – Effect of various factors on measured noise

1. Raw file processing

Not surprisingly, different raw processing, will produce different results, because of the combined effect of the demosaicing algorithm itself and possibly a subsequent denoising process. Whilst we might never know what goes on inside the camera it is reasonable to compare the outputs of different cameras when the same external processing is applied. The graph suggests that bilinear is far superior to AHD with regard to noise, but in other respects it is not and is not included as an option in RawTherapee or Darktable.

AHDvBilin

2. Sensor technology

Earlier digital camera sensors used ccd technolgy, more recently cmos technology has become the norm. I have included measurements of my own Pentax K10D (ccd) and K5 IIs (cmos) for comparison. We might expect a greater difference in noise levels but the larger pixels of the K10D mitigate against this. Both cameras have the same size sensor but the K10D has 10 Megapixels, the K5 IIs has 16 Megapixels. Both raw files (shot at ISO 400) were processed in ImajeJ via the DCRaw plugin using the AHD demosaicing method.

K5-vs-K10-AHD

3. ISO setting

This is simply an analogue gain control before the ADC (Analogue to digital converter) in the camera. As expected images at higher ISO have higher noise levels. In fact doubling the ISO doubles the noise level. But we are working in a non-linear gamma and this means that we see only a 1.4x increase in both noise and signal. (The three graphs relate to the same target with the same illumination).

This is analogous to the situation with film: faster films are noisier.

Noise vs ISO

4. Resampling

The theory

I explained in my previous article why a sensor with larger pixels will exhibit lower photon noise.  Whilst there is also a small contribution from “read noise”, photon noise (or shot noise) dominates. The magnitude of shot noise from a single pixel on the sensor (measured as number of electrons) is equal to the square root of the electron count from that pixel. For a particular incident light level an increase in pixel area by 4x will produce 4x the electron count but only 2x the noise, an imrovement in signal to noise ratio (SNR) of 2x. SNR is a useful concept here because we are considering the signal input to the ADC to be the same in the two cases. Thus 4x the pixel area means half the noise.

So, what happens when a raw file is resampled? Suppose we wish to resample an image from 12Mp down to 3Mp. One way to do this would be to make one new pixel from each group of four (2 x 2) by averaging the outputs. We would expect the same reasoning to apply and, in this case, result in half the noise. This is, of course, a simplistic approach but it sets a limit to what might be expected in practice.

In practice

RawTherapee provides six alternative resampling methods, Lanczos being preferred. I found that when an image from the Canon G9 was resampled down by a factor of four using Lanczos the original noise level of 3.1 was reduced to 1.8 (in the case of the Pentax K5 IIs 0.81 was reduced to 0.54). Not quite as good as the theory suggests but we must recognise that there are other factors involved. In a practical application the resampling algorithm has to be able to handle any scale factor, not just “divide by four”.

The table shows the results of four resampling methods in RawTherapee – sampling down by a factor of four:

Selection_023`

Darktable does not offer a choice of method, but the results suggest it is Lanczos:

Selection_024

5. Exposure value

I established early in the investigation that for a particular camera at a particular ISO setting, the noise versus signal graph depended only on the raw processing, for example a different demosaicing method can make a big difference. However, it makes no difference whether a particular signal level is the result of photographing a white patch with a short exposure or a dark patch with a longer exposure. Provided that the exposure is not too long. This means that to compare two cameras we need to use the same post-processing method and ISO for both and clearly specify any other relevant camera settings.

Noise vs shutter

Shot Noise

The graphs included so far do not show any resemblance to the curve associated with photon noise. I believe the horizontal plateau in these graphs is the result of the inverse gamma curve in the processing largely compensating for the characteristic parabola of photon noise (alias shot noise).

To confirm this I processed one of my step images in ImageJ using the DCRaw plugin (Menu: Plugins > Input-Output > DCRaw Reader) with the settings shown below:

DCRaw settings

The plot of noise versus signal is then parabolic (noise proportional to square root of signal), and when (noise squared) is plotted against signal we get a straight line. Read noise, which would show as a non-zero intercept with the Y-axis is not apparent, presumably because it is too small or my processing is not sufficiently accurate. The shape of these graphs is characteristic of photon noise.

K5-RT-Amaze-lingamma

K5-RT-Amaze-lingamma2

Noise Histograms from Two Cameras

ImageJ can also be used to plot histograms of the noise. I photographed a white card with the camera exposure set on ‘auto’ and slightly out of focus, so as to produce a mid grey image. This is processed in ImageJ as follows.

Select a sample area of the image (several thousand pixels). From the menu select Analyse > Histogram and a histogram of those pixel values will be displayed. This will probably be a very narrow spike around the mid value of 127 – the actual level is not important.

To see this in detail click on ‘List’ and the values used to plot the histogram will be displayed. We are only interested in the non-zero values so scroll down until the group of non-zero values is found and select these (include a few zeroes either side for completeness) to copy and paste into a spreadsheet or graph plotting program such as QtiPlot. The results for two of my cameras are shown below.

Histogram

The histogram from a Canon G9 showing the classical Gaussian curve – Strictly a Poisson Distribution – approximates Gaussian for a mean >10

K5 Histogram

The much lower noise (lower spread of pixel values) from a Pentax K5 IIs

Conclusion

The method described is very quick and accurate. ImageJ has a wealth of analytical processes – I have used just two of them: Measure, which produces mean, standard deviation, minimum and maximum values of the pixels in the selected sample, and Histogram, which displays a histogram of the pixel values and a listing of the number of pixels at each pixel value from 0 to 255. By selecting a range of interest from this list a detailed plot like those above is easily produced. Whilst a spreadsheet program can be used, QtiPlot produces much smarter images for publication.