CFA DEMOSAICKING BY ADAPTIVE ORDER OF
APPROXIMATION
J.S. Jimmy Li and Sharmil Randhawa
School of Informatics and Engineering, Flinders University, GPO Box 2100, Adelaide SA 5001, Australia
Keywords: CFA demosaicking, adaptive order, approximation, colour smoothness.
Abstract: Colour filter array (CFA) demosaicking refers to determining the missing colour values at each pixel when a
single-sensor digital camera is used for colour image capture. It has recently been shown that missing
colour values can be interpolated or extrapolated using Taylor series. The accuracy of approximation
depends on the number of high order derivative terms included in the Taylor series. For a smooth region of
an image, the higher the order, the higher the accuracy in the approximation of the missing colour values.
However, the estimation of high order derivative terms requires pixel values from a wider area of
neighbourhood. When an image contains features closely spaced together, extrapolation using pixels from a
smaller region of neighbourhood is preferred and a low order of approximation should be applied. In order
to achieve more accurate results, we propose an algorithm using an adaptive order of approximation
depending on the colour smoothness of the image. It has been shown that our algorithm outperforms other
techniques for various images, and in particular for images with the above mentioned characteristics.
1 INTRODUCTION
Colour filter array (CFA) demosaicking is the
determination of missing colour values at each pixel
when using a single-sensor digital camera for colour
image capture. In this paper, we introduce an
algorithm using an adaptive order of approximation
to recover missing colour information which
depends on the colour smoothness of an image. Our
method is divided into two stages. In the first stage,
we use the local colour smoothness of the image to
adaptively determine the appropriate order of
extrapolation for the green plane. This will be
applied independently to produce one estimate for
each of the four possible directions, namely up,
down, left and right.
The second stage determines the best estimate
out of the possible four directional estimates, using a
classifier instead of a linear combiner. Other
demosaicking methods (Kimmel 1999, Lu&Tan
2003) determine the missing colour values by
combining weighted estimates from corresponding
directions. Inaccurate estimation of weightings will
produce artifacts which manifest themselves in the
demosaicked output.
The remainder of the paper is organized as
follows. Section 2 details the extrapolation method
of various orders. Section 3 presents the
experimental results, and compares this method with
other existing methods, with the conclusion given in
Section 4.
2 ADAPTIVE ORDER OF
APPROXIMATION
The most common colour filter array used is the
Bayer CFA (Bayer 1976). Figure 1 shows an 8 x 8
window of a Bayer array neighbourhood, where the
index (i,j) denotes the row and column of each
colour pixel. For the Bayer pattern, the green colour
is sampled at twice the rate of the red and blue
values. This is due to the peak sensitivity of the
human visual system which lies in the green
spectrum (Bayer 1976).
To estimate the green value at position x (G
x
) at
which only the blue value (B
x
) is known, consider
the one-dimensional case with an edge boundary on
the left-hand side of B
x
, as shown in Figure 2.
5
S. Jimmy Li J. and Randhawa S. (2006).
CFA DEMOSAICKING BY ADAPTIVE ORDER OF APPROXIMATION.
In Proceedings of the First International Conference on Computer Vision Theory and Applications, pages 5-10
DOI: 10.5220/0001366800050010
Copyright
c
SciTePress
Figure 1: 8x8 Bayer pattern.
Figure 2: 1D Bayer Pattern and Edge Boundary on the
left-hand-side of B
X
.
(1), (2) and (3) are the zero, first and second
order extrapolation equations for the determination
of the missing green colour
x
G
ˆ
respectively
(Li&Randhawa 2005a). From these equations the
higher the order of approximation, the more higher
order correction terms are included to improve the
accuracy. However this implies that more samples
from an extended area of neighbourhood are
required for the approximation.
Zero Order:
1xx
GG
ˆ
+
= (1)
First Order:
)BB(GG
ˆ
2xx
2
1
1xx ++
−+=
(2)
Second Order:
)GG()BB(GG
ˆ
3x1x
4
1
2xx
4
3
1xx ++++
−−−+= (3)
Zero order approximation uses the nearest
neighbour to estimate the missing colour value in the
direction of the extrapolation. It gives satisfactory
results for regions containing features closely spaced
together. In this case, the zero order avoids using
pixels from other nearby non-related features in the
estimation.
For regions with little colour variation, the
general assumptions for demosaicking will hold
(Lu&Tan 2003), and the second order will give the
best accuracy. For regions with moderate colour
variation, the first order will be applied.
The second order will provide adequate
accuracies for most applications, and third or higher
orders of approximation are not required
(Li&Randhawa 2005a).
For the red/blue missing colour values, it has
been shown that first order extrapolation will give
satisfactory results for most images ((Li&Randhawa
2005b).
2.1 Ideal Selector
To confirm that different orders of approximation
will perform better in different regions of an image,
we apply an ideal selector to determine which order
is the best choice for each missing pixel in the green
plane.
To illustrate the above mentioned concept, we
use the pixels in the original image to select one of
the twelve estimates (four directional estimates from
each order) based on the minimum mean square
error criterion. Figures 3 and 4 show the original
and the ideal selector output of the popular
Lighthouse image. These two images are visually
indistinguishable from each other (MSE = 0.0122)
and this indicates clearly that the selected output is a
very good approximation of the original. This
confirms the presence of a highly accurate estimate
within the twelve extrapolated choices. Table 1
gives the proportion for each order as the best choice
for Figure 3. (The undecided proportion in Table 1
represents those pixels that have identical values for
different orders and hence no unique order can be
decided.)
Table 1: Best choices for each order.
Order Best Choice
Zero 30 %
First 24 %
Second 30 %
Undecided 16 %
In general, different images will have different
proportions for different orders but will normally
have some proportion for each order. Hence we
propose an adaptive algorithm to select one of the
three orders according to the colour smoothness of
the image to achieve better results.
VISAPP 2006 - IMAGE FORMATION AND PROCESSING
6
Figure 3: Original Lighthouse image.
2.2 Adaptive Algorithm
An indicator to determine the colour smoothness is
proposed as follows. Based on the idea of the hue
assumption (Cok 1987), we define that the colour is
smooth when (4) is satisfied when estimating a
missing green value at a blue pixel location. A
similar equation applies for the estimation of a
missing green value at a red pixel location. In this
case, when the difference between the variations in
blue and green is small, the hue assumption is valid,
and hence we can apply the highest (second) order
of extrapolation to obtain the best results.
Experimentally, a normalised value of 0.7 for
ε will
give satisfactory results for most images.
ε≤−−−
+++
GGBB
3X1X2XX
(4)
Otherwise, if the variation in blue (or red) is
greater than the variation in green (5), this may
indicate a colour edge in blue (or red) and so the hue
assumption does not apply. Hence the zero order is
preferred. In this case the application of the invalid
hue assumption in the estimation of the missing
colour pixel can be avoided.
3X1X2XX
GGBB
+++
−>−
(5)
Figure 4: Ideal selector output.
However, if the variation in green is greater than
the variation in blue, the first order is chosen
because this indicates that there is no colour edge in
blue, and so half of the small variation in blue will
be included to improve the accuracy in the
estimation of the green value as shown in (2).
Similarly, one estimate for each of the four
directions, namely up, down, left and right, can be
found by (1) to (5). The next step is to use a
classifier to select the best choice out of the four
candidates. Since a linear combiner, as used in other
methods (Kimmel 1999, Lu&Tan 2003) will blur an
edge, a median-based classifier (Li&Randhawa
2005a) which will preserve sharp edges is used
instead.
In the classifier stage, an orientation matrix for
every pixel is produced using the CFA image input.
This is used to indicate the possible orientation of an
edge for that pixel. The underlying assumption
made is that the neighbourhood orientation must be
aligned in a direction along an edge (Li&Randhawa
2005b).
The orientation matrix (Li&Randhawa 2005a) is
used to reject one sample before we apply median
filtering. This is because an odd window width for
the median filtering is preferred in order to avoid
blurring an edge. The classifier algorithm is
depicted as a flowchart in Figure 5. If it is a ‘1’ in
the orientation matrix, a possible vertical edge exists
and hence one of the two estimates in the horizontal
direction will be rejected. Similarly for a ‘0’ in the
orientation matrix, we reject an estimate in the
CFA DEMOSAICKING BY ADAPTIVE ORDER OF APPROXIMATION
7
vertical direction. The one out of the two to be
rejected has the greatest difference in magnitude
from the median of the four extrapolated estimates.
3 RESULTS
To evaluate the performance of our method against
other demosaicking methods, the picket-fence region
of the Lighthouse image in Figure 3 was used. This
area represents a challenge for many demosaicking
methods due to the presence of many edges close
together. The image quality performance measures,
using normalized color difference (NCD)
(Plataniotis 2000) and mean squared difference
(MSE), of the various demosaicking methods:
Freeman (1988), Kimmel (1999), Hamilton (1997),
Plataniotis (2004), Lu&Tan (2003), and Gunturk
(2002), are tabulated as shown in Table 2. Our
proposed method, with
ε = 0.7, has the smallest error
value among all the methods. Figures 6 to 14 show
the sample demosaicked results from our proposed
method and other methods under comparison. This
supports our quantitative measures and illustrates
that our method is also visually superior to other
demosaicking methods as it has the least false
colours in the high-frequency picket-fence region.
Table 2: Image Quality Performance Measure.
Method NCD MSE
Bilinear 0.1036 24.65
Freeman 0.0587 14.75
Kimmel 0.0687 17.35
Hamilton 0.0268 8.85
Plataniotis 0.0637 16.25
Lu&Tan 0.0163 5.05
Gunturk 0.0153 4.01
Our Proposed
Method
0.0115 3.77
We also applied other types of images for the
evaluation of our proposed method as shown in
Figures 15 and 16. The results are tabulated in
Table 3, and they confirm that our method is
superior to other techniques.
4 CONCLUSION
An adaptive order of approximation algorithm has
been proposed for colour filter array demosaicking.
This method uses the colour smoothness of an image
to determine a suitable order of approximation. It
has been shown that our method outperforms other
techniques visually and quantitatively. Research on
its implementation for real-time processing is
underway.
Table 3: NCD results for the demosaicking methods.
NCD
Method Statue Image
Red Door
Image
Bilinear 9.9222E-03 5.4689E-03
Freeman 5.9773E-03 4.1314E-03
Kimmel 7.2663E-03 5.8250E-03
Hamilton 6.9636E-03 4.0047E-03
Plataniotis 371.35E-03 530.98E-03
Lu&Tan 5.5855E-03 4.6519E-03
Gunturk 5.3888E-03 4.7444E-03
Our Proposed
Method
5.1359E-03 3.9169E-03
REFERENCES
Bayer, B.E. (1976): Color Imaging Array. US Patent 3
971 065.
Cok, D.R. (1987): Signal Processing Method and
Apparatus for Producing Interpolated Chrominance
Values in a Sampled Color Image Signal. US Patent 4
642 678.
Freeman, W. T. (1988): Median Filter for Reconstructing
Missing Colour Samples. US Patent 4 724 395.
Gunturk, B. K., Altunbasak, Y. and Mersereau, R. M.
(2002): Colour Plane Interpolation Using Alternation
Projections. IEEE Transactions on Image Processing,
11:997-1013.
Hamilton Jr., J. F. and Adams Jr., J. E. (1997): Adaptive
Colour Plan Interpolation in Single Sensor Colour
Electronic Camera. US Patent 5 629 734.
Kimmel, R. (1999): Demosaicing: Image Reconstruction
from Colour CCD Samples. IEEE Transactions on
Image Processing, 8:1221-1228.
Li, J.S.J. and Randhawa, S. (2005a): High Order
Extrapolation using Taylor Series for Color Filter
Array Demosaicing. Lecture Notes in Computer
Science, M. Kamel and A. Campilho, Eds., Springer-
Verlag, 2005, 3656:703-711.
Li, J.S.J. and Randhawa, S. (2005b): Improved Accuracy
for Colour Filter Array Demosaicking using High
Order Extrapolation. Proceedings of ISSPA 2005,
Sydney, 331-334.
Lu, W. and Tan, Y.-P. (2003): Colour Filter Array
Demosaicking: New Method and Performance
Measures. IEEE Transactions on Image Processing,
12:1194-1210.
Plataniotis, K. N. and Venetsanopoulos, A. N. (2000):
Colour Image Processing and Applications, Springer
Verlag.
Plataniotis, K. N. and Lukac, R. (2004): An Efficient
Demosaicing Approach with a Global Control of
Correction Steps. IEEE International Conference on
VISAPP 2006 - IMAGE FORMATION AND PROCESSING
8
Acoustics, Speech, and Signal Processing
Proceedings, pp. III 469-472.
Randhawa, S. and Li, J.S.J. (2005): CFA Demosaicking
with Improved Colour Edge Preservation. Proceedings
of IEEE Tencon’05, 21-24 November 2005,
Melbourne Australia.
Figure 5: Flowchart of Classifier.
Figure 6: Original picket-fence from Lighthouse image.
Figure 7: Bilinear Output.
Figure 8: Freeman Output.
Figure 9: Kimmel Output.
Figure 10: Hamilton Output.
no
Orientation
= 1?
y
es
Classifer output =
Median{remaining 3
Extrapolated Estimates
}
Orientation
matrix of
CFA image
Proposed
Adaptive
Al
g
orith
m
4 Extrapolated
Estimates
Reject one
vertical
estimate
which is
furthest
from the
median
Reject one
horizontal
estimate
which is
furthest
from the
median
CFA DEMOSAICKING BY ADAPTIVE ORDER OF APPROXIMATION
9
Figure 11: Plataniotis Output.
Figure 12: Lu&Tan output.
Figure 13: Gunturk output.
Figure 14: Our proposed method.
Figure 15: Statue Image.
Figure 16: Red Door Image.
VISAPP 2006 - IMAGE FORMATION AND PROCESSING
10
