Image Fusion by Shift Invariant Discrete Wavelet Transform for Remote Sensing Applications

The fusion technique of the spectral bands captured by the sensors carried onboard satellites is one digital processing method for extracting information and detecting ground targets. Image fusion also known as pan-sharpening-provides the necessary means to combine many images into a single composite image that is suitable in visual interpretation processes or in digital interpretation. The principal objective of this study is to find the best suitable algorithms for obtaining integrative information from several separate images in one combined image. Based on the above, a special software system was designed to implement and test the fusion methods used in remote sensing applications by selecting and applying a Shift Invariant Wavelet Transform (SIWT) method to the remote sensing images and then comparing with four other different image fusion algorithms. Two objective mathematical methods were also used to measure the amount of shared information obtained in the images resulting from the fusion, as well as using the visual and Near-Infrared images of the new Sentinel-2 European satellite for a part of Nineveh province as experimental images. The results showed a preference of the wavelet transform method over the other fusion methods for the remote sensing images.


1
Introduction Image fusion software offers a mechanism to connect several pictures forming a composite image better suitable to perceptual and analytical tasks for people and computer processing. There are several image fusion programs but none are designed for remote sensing applications and they obtain other purposes though they are considered complex and expensive packages such as GIS packaging software [1].
A general concept of image fusion is: "the combining of two or more images to create a new picture using a certain method" (Figure 1) [2]. Sensors 1 and 2 in Figure 1 mean any remote sensing acquisition source, could be RGB images or Multispectral images. The motivation of image fusion research remains the result of recent technological improvements in the fields of sensing procedures and sensor design. For many imaging applications, the use of multiple sensors has enhanced and increased the amount of information collected compared with the use of image sensors that operate alone [3].
Due to the fact that different sensors capture the image in a different electromagnetic wave, so the images give different information depending on the spectrum use as showed in Figure 2. However, the visible sensor gives the exact localization of the object, subsequently visible and infrared image fusion will indicate more reliable information for the exact area or object. This research will avoid the limitation of the visible and infrared remote sensing images by using two different electromagnetic satellite bands with different image fusion and processing techniques. This research uses band 4 and band 8 Sentinel satellite images.
The aim of this research is to arrive at a fused image with more information than before a fusion operation, to highlight new details. In other words, using fusion techniques and reliable image processing procedures to integrate the data gained from a variety of disparate remote sensing devices into a single fused image. To achieve this, a toolbox was designed and implemented using MATLAB program to apply all stages of image fusion techniques without any complexity and in one simple program interface. Five different image fusion methods were applied, including with Shift Invariant Discrete Wavelet Transform (SIDWT) method, then these methods were compared by two objective measures.

Image Fusion Algorithms
In general, according to the image processing approaches, there are two main kinds of image fusion: single resolution approaches, and multiresolution processing approaches. Single resolution approaches have different modalities such as arithmetic algorithms; they produce the fused image pixel by pixel [4].
On the other hand, multiresolution approaches have the most successful image fusion techniques. Gaussian pyramids remain as a foundation of a different type of multiresolution approaches such as Wavelet Transform, Laplacian, Morphological, Contrast and Filter-Subtract-decimate (FSD) pyramid [5,6].
Wavelet Transform is one of the most important structures in the image fusion area. Waveletbased pixel-image fusion procedures raise the data content of fused images through choosing the greatest important geographies from contribution images and transmitting them to the compound image. Input images -for simplicity and as an example only-two images A and B are first decayed into multiresolution pyramids employing a series of multiresolution analysis filters. Then a new pyramid array is modified to contain new information. The pyramid fusion procedure then considers, in a regular way, individual or sets of pixels from the multiresolution pyramid representations of the contribution images, and forms values of the equivalent pixels of the new pyramid. The coefficients of the new pyramid are formed either by transmitting the input coefficient values straight or as arithmetic groupings of the equivalent coefficients from the contribution pyramids. When the pyramid fusion procedure is completed, the fused pyramid is inputted into the wavelet reconstruction process to obtain the final fused image (see Figure 3) [7].
The elementary plan of the image fusion structure is to use exact fusion instructions to obtain the information of the different contribution images. The DWT multiresolution representation has many advantages over the other mostly Gaussian pyramid based multiresolution methods. One of the greatest fundamental matters is that wavelet function used image analysis from an orthonormal basis, which results in a non-redundant signal demonstration [8,9].

Infrared and Visible Image Fusion
Over the past years, many researchers have used image fusion techniques to combine thermal and visual images and gain from them in the extraction of additional information in a variety of applications. Infrared images can discriminate goals from their surroundings based on a gradient of radiation that works well all day and evening. The thermal reflections enable the spectral information to be distinguished, and this is no less important than the spatial information in the visible spectrum. Visible images, on the other hand, can offer high spatial resolution and interpretation of material information in a way compatible with the human vision scheme. Thus, it is beneficial to merge the two kinds of pictures that combine the benefits of infrared radiation data with detailed textures in visible images. In a comparative analysis, Jiayi Ma etal. [10] cited the evaluation of wavelet transformation and other traditional image fusion algorithms. The analysis included nine methods for transformed wavelets, such as the dual-tree discreet transformation of wavelet; quaternion-wavelet transformation; lifting transform wavelet; lifting of a stationary transformation of wavelet; movement-compensated wavelet transformation; redundant multidisciplinary lifting of wavelet; spectral graph wavelet transformation and multiwavelet. Yaochen Liu etal. [11] were already designing a new infrared and visible picture fusion algorithm based on technologies for visual focus. While the superlative setup for discrete wavelet transformation (DWT) based image fusion is defined by Angel Sappa etal. [12], especially in a visible and infrared case and different wavelet dependent cross-spectral image fusion strategies are analyzed. In the end, it can be confirmed that no one in the literature has used Shift Invariant DWT (SIDWT) techniques for fusion of sentinel satellite infrared and visible images.

Image Fusion Software
There are many image fusion toolboxes integrated in frequently used commercial remote sensing software packages such as ENVI, ERDAS and they are considered as complex and expensive packaged software. as Additionally, not all the fusion approaches are presented in this marketable software [13] such as SIDWT. As an example, in the famous remote sensing software ENVI, image fusion operates under a toolbox entitled "Image Sharpening" and includes some limited image fusion methods. Users need to go through these software to do all the image fusion stages. Different sensors may cause shifting, translating between the image pixels, and this needs a registration stage whichif it is available -will be in another toolbox or other software. To measure the quality of image fusion methods you not only need visual consideration but also objective measures, which cannot be found in the commonly used software.
In this research, software with a special interface toolbox was developed in Matlab, which can be used for picture fusion of any satellite image. This software is designed for use in the philosophy of programming and image processing without any difficulty or technical expertise. All the image fusion requirements with the different stages and with different options can be achieved with help of the program's interface in the same place without writing any code and without any complexity to display original inputs, fused images, and qualitative outcomes, after treating all in one interface toolbox. as Additionally, the proposed software covers the SIDWT algorithm with four other multiresolution fusion algorithms which cannot be found in the traditional software. All details of proposed software will be provided in the next section.

A Fusion Method Based on a Shift Invariant DWT
It is well known that DWT generates a show of shift variant signal which induces a changerelated fusion system. The images input must be decomposed into a Shift Invariant DWT (SIDWT) representation to resolve the shift dependence of the technique of wavelet fusion. Rockinger Oliver [14] proposes a new approach of fusion, which has been used to explore remote sensing data in this research. For the DWT, the input sequence of each point of the SIDWT is split into the wavelet sequence Wi (n) that is saved and into the scale sequence Si (n), which for the next decomposition level serves as the input, equations 1 and 2.
In order to detect the full decomposition process, the zero level scale sequence is set to match the s0(n)=f(n) input sequence. Unlike the standard decomposition scheme of DWT, a sub-sample is produced, which results in a redundant wavelet view. The g (2i.k) and h (2i.k) filter at I level is accomplished by adding the necessary number of (zeros) between the filter taps of g(k) and h(k) prototype filters. The input sequence is reshaped by the reverse SIDWT as a convolution of both SIDWT sequence and scale sequence with a suitable re-filter g~(2i.k) and h~(2i.k) as in equation (3). si(n)= ∑h~(2 i .n-k).si+1(n) + ∑g~(2 i .n-k)wi+1(n) ……………. (3) The normal product formulation for tensor products follows an expansion in the decomposition technique for 2D images.
The real process of fusion in the SIDWT situation is the same as in the generic fusion situation of wavelets: the input images are decomposed into the representation of their SIDWT and the composite SIDWT image is created by an appropriate selection structure. At all stages, analytical filters are obtained by adding the required number (zero) between the sample filter. Recovery of the input sequence is performed by the inverse SIDWT as a convolution with the related reconstruction filters of the invariant wavelet sequence and the scale sequence.

Image Fusion Performance Evaluation
As the interest in the subject increased, more fusion approaches have found their way into the literature and consequently the need arose for a way to measure the performance of these systems. Image fusion performance evaluation could be divided into two methods: subjective and objective image fusion performance measures.
Subjective tests are still not widely used. They involve the use of human subjects inspecting fused images under controlled conditions to determine the advantage of fused resulted images [15].
Objective performance (quantitative measure) means to implement mathematical conditions to evaluate the quality of fused images. There are two main kinds of objective measures [16]: A. Quality measures based on references: construction and use of an ideal fused image as a guide for experimental evaluations [17]. For these comparisons Root Mean Squared Error (RMSE) methods were commonly used. For the same reasons, many researchers use (PSNR) and correlation [14]. B. Non-Reference-based quality measures: The current metric to evaluate image fusion procedures is usually based on a measure of the fidelity of the transfer of a feature (such as edges) from the input images to the fused output. Mutual Information is a well-known method for this purpose [18] and it is adopted in this research. The Mutual Information (MI) represents how much evidence is gained from the input images. That is, the quantity of information one image holds on another defines the joint histogram of source image A(B) and the fused image F as PFA (f, a) (PFB (f, b)). Then the mutual information between the picture in the source and the fused image is deduced from: [19] IFA (f, a) = ∑ P FA (f, a) log2 Thus the Image fusion performance is measured via performing the summation between (4)  Equation (6) shows that the recommended measure returns the total amount of evidence that result fused image F comprises about A and B. Better image quality is represented by a greater value.

Sentinel Satellite Images
Sentinel satellite is a European satellite devoted to earth imaging, providing a great opportunity to observe global vegetation and terrestrial cover due to its improved spatial, spectral, and temporal characteristics compared to the US Landsat satellite and the French Spot satellite. Sentinel-2 generation launched by the European Space Agency in the year 2017 was a part of the European Copernicus Space Program. This satellite contained 13 spectral beams as shown in Table 1. The goal is to scan the earth once every five days. The high temporal accuracy of the satellite, which is up to 5 days, coupled with the speed of delivery of the results, 24 hours for normal use and up to 10 minutes in emergencies, has given the importance of quality characteristic of this satellite [20]. Band 4 and band 8 were chosen in this research to fuse the visible and infrared information.

Image Fusion System Framework
In order to achieve the research aims and implement the image fusion methods, an integrated software application, including an interface toolbox, as well as schemes of image pre-processing, image registration, image fusion, and objective image quality were designed and implemented. This toolbox has been developed using MATLAB software to investigate different image processing approaches and to demonstrate the results of the image fusion methods results.
A framework for the image fusion system is proposed and built. It includes implementation, testing and evaluation of fusion algorithms as shown in Figure 4. After input images in the first stage, image enhancement can be made in the second stage to enhance the results by using different image pre-processing techniques. Image registration is taken into account in the third stage to transforming the different images into one coordinate system to preserve accurate spectral information of the multispectral images. Then comes the stage of choosing an appropriate type of merging method. Measuring the image fusion quality comes in stage five, if the measurement result is not good then decision making in stage six will be taken to return back to the pre-processing stage to try enhancing the input images. A diagram of the imaging system and associated algorithms is showed in Figure 5.
The developed toolbox has six menus, which are 'File', 'Image Pre-processing', 'Image Registration', 'Image Fusion', 'Feature Extraction', and 'Help'. Each menu has several submenus. The 'File' menu allows the user to open a new image, save the processed image, reset the image and exit the program. The main interface of the toolbox is shown in Error! Reference source not found. 6.

Image Pre-Processing (Image enhancement)
Image enhancement before fusion will improve the fused image. All image data corrections should be applied before using images because some indicate specific sensor effects [21]. Image preprocessing techniques have been included as an optional part in the toolbox. 'Pre-processing' menu offers to the user an opportunity to process the image based on prior experience and compares the results. In menu 'Pre-processing', there are several embedded functions: • RGB Image to Grey Image: Invert the RGB image to a grey level image.
• ROI selection: Manual selection of the Region of Interest (ROI). The user can apply a window to the input image and save the specific region for further investigation. • Noise Removal: Remove noise from the input image via the application two typical noise removal methods: mean filter and median filter. • Edge Detection: Apply edge detection processing via two commonly used edge operators: Canny Filter and Soble Filter. This is an optional step to demonstrate the edges before doing fusion techniques.

Image Registration
One of the issues a fusion system has to deal with is a registration of the source images. Image fusion allows the pixels in the input images to be matched. This problem is very necessary to achieve the right fusion results [22,23]. Images from different sensors and different spatial positions of the sensors are mostly acquired from the same scene. These images can be interpreted, rotated, scalable, and transformed in relative terms [24]. The purpose of image registration is the implementation of a spatial transformation of the input image by the reference image [25].
Manual image registration is embedded in the toolbox package, non-reflective similarity transforms are used to register a thermal image to a visible image. When the menu 'Image Registration' is clicked, a new window called (Control Point Selection Tool) appears with the flexibility to choose control points. The approach selects at minimum three control points to do the registration between the images. The experiments in this research used four control points for better certainty as shown in Figure 7.

Image Fusion Methods
After the images were registered, five multiscale image fusion methods were investigated and tested; (a) Wavelet Transforms (b) SIDWT (c) Lablacain, (d) Contrast, and (e) morphological pyramid. SIDWT has been adopted in this research to investigate remote sensing data as a recent trend as described in the 3.1 subsections. The source images for all algorithms were grayscale. In this menu, you can apply different image fusion methods. The wavelet submenu contains different wavelet filters (Haar, Daubechies, Coiflets, Symlets). Other image fusion algorithms are proposed for future work viewed in this menu (See Figure 6). All Wavelet Transform kernel filter types are embedded in the toolbox. Also, the wavelet pyramid can perform up to seven depth multiscale levels without any complexity and in a very fast and easy way.

Image Fusion Performs Evaluation
The established criteria for measuring image fusion procedures are generally based on a calculation of the attribute transition fidelity (e.g., edges) from input images to fused material. The resultant fused images were evaluated and compared using two different objective performance evaluations, which are MI (as in Eq. 6) and RMSE. The less value of RMSE considers the fusion technique as better and verse versa for the MI matric, a large value of MI considers the fusion technique as better.

Experiments and Results
The study area was chosen at the beginning of the entry of the Tigris River from the northern side of the city of Mosul (Figure 8). Experiments were undertaken using two Sentinel satellite images (band4 and band8) for the part of Mosul city as the test images (Figure 9) to compare the efficiency of different image fusion procedures. The registration operation for these images was carried out. Image fusion algorithms like wavelet transform (Haar filter), Laplacian, Contrast, and morphological pyramid [8], as well as SIDWT were applied. Figure 10 represents the results of these different fusion methods.
The detail in the visual image in Band 4 increases the visual information and awareness of the situation in the combined image. While the near-infrared image in Band 8 adds other different information that depends on the part of the electromagnetic spectrum that represents the nearinfrared section. Evaluation of the performance of the fusion techniques using the image fusion metrics is shown in Table 2 and Figure 11. It shows five different image fusion algorithms and two objective evaluation metrics for the resultant fused image.

Discussion of Results
In this research, the image resulting from the merging was examined using objective performance (quantitative measurement), which means the implementation of mathematical conditions to assess the quality of the combined images. The human eye was not used in examining the resulting image due to its limitations in detecting the fine details in the remote sensing images. From Table 2 and Figure 11, the objective measures used to measure the efficiency of the resulting merging image (RMSE and MI) ranked the WT method and the derived method SIWT-as the best performing of the five methods. This means that the merging has enhanced and increased the amount of information in the resulting image. SIWT used zero as the parameter to compensate for the deficiency in the WT process; hence, there were some changes in the pixel information, which made the WT method superior. As for the rest of the methods used for comparison, their results varied, but all of them showed fewer results than the WT methods. Since the MI metric gives an indication of how much information is being shared between the input images and the resulting merged image, therefore, it identified the WT and SIWT algorithms as being better than other algorithms because they preserved the original information, and converted this information into the resultant merged images. As for the RMSE measurements, it also showed promising results as it showed that the WT method is the least method that generated an error or difference in the data between the input images and the resultant image. It was also noted that the difference in the numerical results of this scale is greater than the difference in the numerical results of the MI scale of the fusion methods used, which indicates the possibility of its use in detecting the efficiency of the results of the merging methods. Finally, it can be concluded that the WT algorithms are preferable image fusion algorithms for remote sensing applications.
Conclusion In this paper, a framework for an image fusion system that is particularly suitable for remote sensing is proposed and implemented. Multilevel analyses have been applied to integrate the visual and infrared images of the Sentinel satellite. An image fusion toolbox is designed and implemented with an easy-to-use software interface, which can be used to fusion any satellite images. All requirements and phases of implementing image fusion algorithms have been grouped into one proposed toolbox. Five different image fusion approaches were implemented and examined. Then an objective evaluation was made of the performance of these approaches. A recent approach has been applied to image fusion which is the use of Shift Invariant Discrete Wavelet Transform SIDWT and compares it with four other algorithms previously used for image merging. Image pre-processing algorithms and image registration algorithms have also been implemented.

RMSE MI
In terms of the efficiency of image fusion, two different objective methods were used in this research to measure and evaluate performance, namely (RMSE, MI). The results of the objective measurement showed that the WT is the best method and SIDWT comes in second place for remote sensing applications. This research represents the basis for future research that will test various other types and approaches of image fusion on satellite images.