Demosaicing of True Color Images Using Adaptive Interpolation Algorithms

In an electronic color-imaging device, such as a digital camera, using a single CCD (Charge Coupled Device) sensor, the color information is usually acquired in sub-sampled patterns of Red (R), Green (G) and Blue (B) pixels. The methodology to recover or interpolate the missing color components is known as color demosaicing, a type of image interpolation. Demosaicing of True Color Images Using Adaptive Interpolation Algorithms. ٦٥ This paper introduces and compares some of the most popularly demosaicing algorithms of adaptive type applied on true color images. Another comparison is done between these methods and the non-adaptive traditional methods. Also, a new method is invented, which depends on both types. The used images are of Bitmap type. Signal to noise ratio measure is used as an objective quality metric. The results of comparison are presented with some sampled images. Programming has been applied using Visual C++ 6.0 programming language.


ABSTRACT
In an electronic color-imaging device, such as a digital camera, using a single CCD (Charge Coupled Device) sensor, the color information is usually acquired in sub-sampled patterns of Red (R), Green (G) and Blue (B) pixels.The methodology to recover or interpolate the missing color components is known as color demosaicing, a type of image interpolation.

Introduction
Due to the cost and packaging consideration, in digital imaging devices, such as the commercially available Digital Still Cameras (DSC), the image color is captured in a sub-sampled pattern.Typically each pixel in the captured raw image contains only one of the three primary color components, R (Red), G (Green), or B (Blue).This sub-sampled color image is generated using certain pattern of a Color Filter Array (CFA).
This CFA is realized by coating the surface of the electronic sensor array or a single CCD array using some optical material that acts as a band-pass filter.This coating allows the photons corresponding to only one color component (frequency range) to be transmitted to the sensor and the other two color components are blocked.A typical and widely used CFA pattern is called Bayer Pattern.As shown in Figure 1  Many other implementations of a color-sampling grid have been incorporated in commercial cameras, most using the principle that the luminance channel (green) needs to be sampled at a higher rate than the chrominance channels (red and blue).The choice for green as representative of the luminance is due to the fact that the luminance response curve of the eye peaks at around the frequency of green light [3].

٦٦
Each cell in Bayer pattern represents a pixel with only one color component as indicated by either R or G or B.
A full-color image needs the information of all the three colors in each pixel location.As a result, each pixel needs to be represented by 24bit colors, assuming 8 bits for each of R, G, and B, as in true color images.So that, it is essential to interpolate the missing two colors in each pixel location using the information of the neighboring pixels.
The methodology to recover or interpolate these missing colors is known as Color Interpolation or Color Demosaicing [1] [2][4].
The aim of this paper is to apply commonly adaptive and nonadaptive demosaicing algorithms on different Bayer images to recover the original true color images.
A combination of two methods of adaptive and non-adaptive type is performed to get a new method, which gives better results.
The applied methods are compared together in order to observe the differences.

Principles of Capturing Colors in Digital Camera
The sensors, which are used in most cameras, are either Charge Coupled Device (CCD) or CMOS (Complimentary Metal Oxide Semiconductor) sensors.The CCD camera comprises a very large number of very small photo diodes, called photo-sites.The electric charges, which are accumulated at each cell in the image are transported and are recorded after appropriate analog to digital conversion.
In CMOS sensors, on the other hand, a number of transistors are used for amplification of the signal at each pixel location.The resultant signal at each pixel location is read individually [1][5] [6].
There are several ways in which a digital camera can capture colors.One approach uses red, green, and blue filters and spins them in front of each single sensor sequentially one after another and records three separate images in three colors at a very fast rate.Thus the camera captures all the three colors components at each pixel location.While using this strategy an automatic assumption is that during the process of spinning the three filters, the colors in the image must not change, i.e., they must remain stationary.This may not be a very practical solution.
A practical solution is based on the concept of color interpolation or demosaicing, which is a more economical way to record the three primary colors of an image.In this method, only one type of filter would be permanently placed over each individual photo-site.Usually the sensor placements are carried out in accordance to a specified pattern.The most popular pattern used is Bayer pattern.It is possible to make very accurate guesses about the missing color component in each pixel location by a color interpolation or demosaicing algorithm [1] [6].

Color Interpolation Algorithms
Color demosaicing algorithms can be broadly classified into two categories: non-adaptive algorithms and adaptive algorithms, as described below [1][2]: • Non-adaptive Algorithms: In non-adaptive color interpolation algorithms, a fixed pattern of computation is applied in every pixel location in the sub-sampled color image to recover the missing two color components.Usually, this type is easy to implement with low cost in terms of computational requirements.
• Adaptive Algorithms: In adaptive color interpolation algorithms, intelligent processing is applied in every pixel location based on the characteristics of the image in order to recover the missing color components.This type of algorithms yield better results in terms of quality as compared with the non-adaptive algorithms.However, effective algorithms in this category are usually more computationally intensive In this paper four algorithms of non-adaptive type with the other four of adaptive types have been applied at Bayer pattern of images of different specifications.Also, a new method is invented, which is a combination of both adaptive and non-adaptive methods.

Non-adaptive Interpolation Algorithms
Nearest neighbor, bilinear, median, and smooth hue transitions are the non-adaptive methods used in this paper.

Nearest Neighbor Algorithm
Probably, nearest neighbor is the most basic form of image interpolation.In this method, each missing color is approximated by nearest pixel representing that color in the input image.The nearest neighbor can be any one of the upper, lower, left, and right pixels.
The advantage of this approach is that the computational requirement is very small and suitable for applications where speed is very crucial.However, the significant color errors make it unacceptable for a still imaging system, such as high-resolution digital cameras.
With the most basic nearest neighbor interpolation, just copy the exact same pixel values over to the filler pixel closest to the pixel

Bilinear Interpolation
A slightly more sophisticated way of accomplishing gray-level assignments, using the four nearest neighbors of a point.In this algorithm, a missing color component is interpolated by linear average of the adjacent pixels representing the missing color.
Consider pixel location (m,n), to estimate G m,n , given G m-1,n , G m,n-1 , G m,n+1 , G m+1,n , the four horizontal and vertical neighbors.The estimate for G m,n is given by: G , the four diagonal neighbors.The estimate for R m,n is given by: R m Blue component is estimated accordingly.This algorithm is better than nearest neighbor that takes into account the gradual transition of pixel color values.Some times the results suffer from pixel artifacts.This may be acceptable in a moderate quality video application because the artifact may not be immediately visible by the human eye.
It is possible to use more neighbors for interpolation.Using more neighbors implies fitting the points with a more complex surface, which generally gives smoother results

Median Interpolation
Median interpolation allocates the missing color component with the "median" value of the color components in the adjacent pixels, as opposed to the linear average used in bilinear interpolation.
The median value is computed for the pixel p at location (m,n) from the four horizontal and vertical neighbors, for the case of missing Green component.And from the four diagonal neighbors, for Red and Blue missing components [1] [2].

Smooth Hue Transition Interpolation
The key problem in both bilinear and median interpolation is that the hue values of adjacent pixels changed suddenly because of the nature of these algorithms.On the other hand, the Bayer pattern can be considered as a combination of a luminance channel (green pixels, because green contains mostly luminous information) and two chrominance channels (red and blue pixels).The smooth hue transition interpolation algorithm treats these channels differently.The missing Green component in every Red and Blue pixel in the Bayer pattern can first be interpolated using bilinear interpolation.The idea of chrominance channel interpolation is to impose a smooth transition in hue value from pixel to pixel.In order to do so, it defines "blue hue" as B/G, and "red hue" as R/G.To interpolate the missing Blue component B m,n in pixel location (m,n) in the Bayer pattern, the following three cases may arise: 1.The pixel at location (m,n) is Green and the adjacent left and right pixels are Blue color.The missing Blue component in location (m,n) can be estimated as: 2. The pixel at (m,n) is Green and the adjacent top and bottom pixels are Blue.The missing Blue component can be estimated as: 3. The pixel at (m,n) is Red and four diagonally neighboring corner pixels are Blue.The missing Blue component can be estimated as: ).The missing Red component can be interpolated in each location in a similar fashion [1][2][3].

Adaptive Interpolation Algorithms
Pattern matching-based interpolation, block matching, edge sensing interpolation, and linear interpolation with second order correction, are the adaptive methods used in this paper.Besides, an enhanced linear interpolation method is proposed here.

Pattern Matching-Based Interpolation Algorithm
In the Bayer pattern, a Blue or Red pixel has four neighboring Green pixels.A simple pattern matching technique for reconstructing the missing color components based on the pixel contexts.It defines a green pattern for the pixel at location (m,n) containing a non-Green color component as a four-dimensional integer-valued vector: The similarity (or difference) between two green patterns g1 and g2 is defined as the vector norm, in other word, the Euclidean distance between two vectors.
When the difference between two green patterns is small, it is likely that the two pixel's locations, where the two green patterns are defined, will have similar Red and Blue color components.
A weighted average proportional to degree of similarity of the green patterns is used to calculate the missing color component.For example if the pixel at location (m,n) is Red, the missing Blue component ٧٠ B m,n is estimated by comparing the green pattern g m,n with the four neighboring green patterns g m-1,n-1 , g m+l,n-l , g m-1,n+1 , and g m+l,n+l .If the difference between gm,n and each of the four neighboring green patterns is uniformly small (below a certain threshold), then a simple average is used to estimated the missing Blue color component: Otherwise, only the top two best-matched green pattern information items are used.For example, if ||g m,n -g m-1,n-1 || and ||g m,ng m+1,n-1 || are the two smallest differences, then the missing Blue color is estimated as: The missing Red components can be interpolated in as similar fashion.While the missing Green component in every Red or Blue pixel is interpolated using bilinear interpolation

Block Matching Algorithm
A block matching algorithm based on a concept of Color Block.The color block of a non-Green pixel is defined as a set x = (x 1 ,x 2 ,x 3 ,x 4 ) formed by the four neighboring Green pixels, say, x 1 , x 2 ,x 3 , and x 4 .A new Color Gravity is defined as the mean x = (x 1 +x 2 +x 3 +x 4 )/4.
The similarity between two color blocks is defined as the absolute difference of their color gravities.The block matching algorithm is developed based on the selection of a neighboring color block whose color gravity is closest to the color gravity of the color block under consideration.
For any non-Green pixel in the Bayer pattern image, as shown in Figure 1, there are four neighboring Green pixels G N (the North neighbor), G S (South), G E (East), and G W (the West neighbor), which form the color block g = (G N , G S , G E , G W ). Its color gravity is g = (G N + G S + G E + G W )/4. The missing Green value is simply computed by the median of G N , G S , G E , and G W .If the pixel at (m, n) is Blue, it will have four diagonally Red pixels R NE (the Northeast neighbor), R SE (the Southeast neighbor), R SW (the Southwest neighbor), and R NW (the Northwest neighbor), whose color blocks are g NE , g SE , g SW , and g NW and the corresponding color gravities are NE g , SE g , SW g and NW g respectively.The missing Red component is assumed to be one of these four diagonal Red pixels based on best match of their color gravities.The best match (or minimal difference ∆ min ) is the minimum of The second-order correction term based on the neighboring blue or red components.

2-Estimation of
c) Estimate Red (Blue) component at a Blue (Red) pixel.In this case two diagonal gradients must be defined as: Using these diagonal gradients, the algorithm for estimating the missing color components is described as: This algorithm provides much better visual quality of the reconstructed image containing a lot of sharp edges.In other case, the second-order calculations of the gradients give some unacceptable results

Modified Linear Interpolation with Second-Order Corrections
This new invented method is based on linear interpolation method discussed above in addition to bilinear algorithm.The idea is to compare the second-order correction term with a certain threshold value, if this term is greater than the threshold value then the correction term is eliminated and only a linear average is computed.Otherwise, the correction term is added.For example, the missing Green component G m,n at location (m,n) is estimated as follows:

Program Interface
The program is created using Visual C++ programming language, as MFC AppWizard (exe) of type Single Document Interface (SDI) [8] [9].
Besides its basic function i.e., image demosaicing, the project has other properties such as image scrolling, file serialization, and providing (Signal to Noise Ratio) SNR results.
The base class that is used for viewing is derived from CScrollView rather than CView class, because the latter doesn't provide the utilities for scrolling, which is very important in viewing highresolution images [8] [9], see appendix A.
File Serialization is the method used by MFC programs to read and write application data to files, with the aid of CArchive class, which is used to model a generic storage object.In most cases, a CArchive object ٧٥ is attached to a disk file [8] [9], see appendix A. Figure 2 shows the main program interface.

Creating Bayer Pattern
To show the effectiveness of the demosaicing algorithms, the Bayer pattern is synthetically generate from a full-color RGB image by simply dropping two color components in each pixel coordinate (if the original Bayer pattern image cannot be grabbed from a digital camera directly) [1][2][4].

Experimental Results
More than twenty images have been chosen to work with depending on their high frequency nature and other interesting image characteristics to demonstrate performance of different color interpolation algorithms.
In order to compare between the color intensity of the original image the resultant image, signal to noise ratio measure is used.The following formula clarifies the relation between two images [10]: Where X refers to the original image and W refers to the image produced from the selected demosaicing process.The formula is applied three times for Red, Green, and Blue intensities of each pixel in the true color bitmap.Table 1 shows the comparison results of interpolation algorithms depending on Green component, as Green is sampled in Bayer image at a higher rate than Red and Blue.Table 2 and 3 give the SNR results.
Figure 3(a) is a full color image of a parrot.Bayer pattern image is generated from this image, as shown in Figure 3(b).The Figures 3(c), (d), (e), (f), (g), (h), (i), (j), and (k) show the resultant images produced by applying the methods: nearest neighbor, bilinear, median, smooth hue transitions, pattern matching, block matching, edge sensing, linear interpolation with second order correction, and the new enhanced linear interpolation, respectively, on Bayer pattern image.

Conclusion
The comparison between the different applied demosaicing algorithms presents the following conclusions: • Applying nearest neighbor method gives some significant color errors, which make it unacceptable for all types of images.• Bilinear algorithm gives better results than other non-adaptive methods and in some cases adaptive methods, as bilinear takes into account the gradual transition of pixel color values.
• Median method provides a slightly better result as compared with nearest neighbor.• Applying smooth hue transition on images having the characteristic of hue values changing suddenly of adjacent pixels, gives results much better than other non-adaptive or adaptive methods.• Best results are achieved when using pattern matching, block matching, and the new enhanced linear algorithms.This is because of the accurate approach to find the best color component match.Also, they rely on bilinear method, as the latter gives good results.• Edge sensing gives better results and high quality images depending on the threshold value and when applying on sharp edges images.• Linear interpolation with second order correction method gives good image quality for sharp edges image.However, it has some unacceptable results and disparate pixels can be obviously noticed.• The new enhanced linear interpolation method, proposed in this paper, provides better image quality as compared with the both types.Also, it gives the best image quality than the original linear interpolation method for all the used images.
Red (or Blue) component: The missing Red (Blue) color components are estimated in every pixel location after estimation of the missing Green components is complete.Depending on the position, we have three cases: a) Estimate the missing Red (Blue) component at a Green pixel G m,n , where nearest neighbors of Red (Blue) pixels are in the same column, as: R m,n = (R m-1,n + R m+1,n )/2 + (2*G m,n -G m-1,n -G m+1,n )/4.b) Estimate the missing Red (Blue) component at a Green pixel G m,n , where nearest neighbors of Red (Blue) pixels are in the same row, as: R m,n