--------------------------------------------------------- By Bill Claff
In an earlier article we learned that a 2-dimensional
Fourier Transform (2D FT) can be used to visualize how uniform an image is.
But while 2D FTs are good visualization they don't give us much detail on the reasons for non-uniformities.
We can take an FT and then perform a frequency analysis that will give us more information.
This article will give us an introduction to these Energy Spectra and how they can be interpreted.
In this article I'll start by following the same roadmap as
for the 2D FTs.
I am going to present a number of synthetically generated images that represent our understanding of what we ought to observe.
In each case the image will be presented on the left, the 2D FT for that image on the right, and the Energy Spectrum for that image below.
The left-hand images are linked to PGM files so you can play with them yourself.
(You will want to use Save link as ... rather than Save image as ... to get the PGM otherwise you'll get a JPG.)
The style of PGM file provided can be opened in ImageJ as well as any text editor or even Excel.
Each Energy Spectrum is linked to an interactive chart where is can be examined more closely.
Here's a diagram of the roadmap I will follow:
For a hypothetic perfect sensor a black frame one the left
the corresponding 2D FT and Energy Spectrum look like:
Not too exciting nor realistic but an opportunity to explain what we see in an Energy Spectrum.
The x-axis is labeled f/fs and is the frequency normalized
to the sampling frequency. Because there is no useful information beyond the
Nyquist frequency the x-axis runs from 0 to 1/2.
Remember, frequency and wavelength have a reciprocal relationship. So the x-axis runs from a wavelength of 2 on the right-hand-side to infinity at f/fs of 0.
The y-axis is Normalized Energy expressed as log2. This
normalization may differ from similar charts you may have seen elsewhere.
The y-axis values of -1, 0, and 1 represent minimum possible, average, and maximum possible energy respectfully.
In this first case out featureless uniform frame is entirely comprised of infinite wavelengths.
This frame is random enough that there are equal amounts of
all frequencies from 0 to 1/2.
The squiggly lines are simply due to our relatively small sample size.
Now we add read noise:
Read noise arises principally from the electronics used to read the value of the pixel (hence the name).
This represents what we might reasonably expect to see from a well behaved sensor.
(Although non-DSNU FPN is not unusual and often presents as horizontal or vertical streaks.)
Also, on some sensors, on chip Phase Detect Auto Focus
(PDAF) may form a visible pattern.
And imbalance in multi-channel readout can also show temporal or fixed patterns.
This is an important topic that we will cover in depth in a separate article.
We are not expecting to see noise reduction in our raw
image; but it happens.
Noise reduction, hot pixel suppression, and other signal processing share the same general characteristic.
They all re-compute a pixel value based on neighboring pixels.
So to simulate this effect I added a small Gaussian Blur to
the typical black image:
I boosted the contrast on the 2D FT to make the effect more obvious. The round "sphere-like" effect on the 2D FT is an important feature to remember.
And the drop in energy from left to right is also a key
characteristic of this process which is in the class of low-pass filters.
Low-pass filters allow low frequencies to pass at the expense of higher frequencies which is why the curve drops off.
We can judge the strength of the filtering by the size of the drop; in this case about 1/6 on our normalized scale.
Black images are interesting and have the advantage that
they are typically evenly illuminated; .but I also frequently analyze evenly
So here's what Signal alone would look like:
Since this is just a signal and it's associated Photon Noise it's not too exciting. Frequency-wise it's indistinguishable from a black frame.
Here's what happens if we apply a Gaussian blur to the
Once again the tell-tail circle on the 2D FT signals that some sort of nearest neighbor signal processing has occurred.
And the Energy Spectrum tell us it's some sort of low-pass filter.
The manufacturing process can result in a small gradient on
the sensor. We can also get a gradient if our illuminated image collection is
Here's what the signal image with a gradient looks like:
Well, that's something new.
This cross pattern happens when the left/right or top/bottom edges of the image don't match.
Note the horizontal gradient is stronger than the vertical; and the cross is wider than it is high.
The Energy Spectrum is also affected. The gradient destroyed any useful information but an Energy Spectrum like this confirm it is present.
A more common problem with image collection is light falloff
Applied to the signal image it looks like:
This has the same cross effect as the gradient and also a small white center due to the falloff.
And the same adverse effect on the Energy Spectrum.
For completeness, let's combine them both:
Note that I made the falloff particularly dramatic to better illustrate the effect.
The Energy Spectrum is also not helpful except perhaps to identify that a strong gradient and/or falloff is present.
The electronics used to read out the value of the pixels can
sometimes have a vertical or horizontal pattern.
Here I have rearranged the DSNU data from above into vertical bands:
The strong vertical bands present as a horizontal line of the 2D FT.
The horizontal component of the Energy Spectrum shows large amounts of relatively even energy at many frequencies.
We will see in the next article that more organized bands or patterns will produce a more organized Energy Spectrum.
In particular this will be helpful in identifying whether and how PDAF pixels are contributing to FPN.
Note that this banding image has exactly the same
statistical characteristics as the previous DSNU image shown here for
So statistically we can't distinguish banding from DSNU; but both the 2D FT and the Energy Spectrum can.
Banding acts like spatially correlated DSNU even though the origin is the external electronics and not the pixel.
When the pattern of banding remains constant from frame to frame then it is Fixed Pattern Noise (FPN).
We have seen a number of synthetically generated images and
their corresponding 2D FTs and Energy Spectra.
Now we have a good idea of how to interpret patterns in a 2D FT and Energy Spectra.
We know that a uniform FT with a white center is ideal and
that the Energy Spectrum will be roughly uniform near zero..
We know that a cross is probably the result of a gradient, falloff, or banding. These Energy Spectra show a strong decay.
We know that a slightly enlarged white center is probably from a strong falloff. These Energy Spectra also show a strong decay.
And we know that a large circular pattern is the signature of signal processing such as noise reduction.
When signal processing is present the Energy Spectrum has help us identify low-pass (and high-pass) effects as will as to gauge their strength.
Energy Spectra are a valuable tool to utilize particularly
when we have a 2D FT that shows any non-uniformity.
In a separate article, Energy Spectra and Repeating Patterns, I'll cover how Energy Spectra can help us to understand how repeating patterns such as PDAF pixels can contribute to FPN.