Multi-resolution Image Fusion Using MultistageGuided Filter

Sharad Joshi and Kishor P. UplaS. V. National Institute of Technology

Surat, Gujarat-395007, IndiaEmail: {sharadjoshi,kishorupla}@gmail.com

Manjunath V. JoshiDhirubhai Ambani Inst. of Information and Comm. Technology

Gahdhinagar, Gujarat-382 007, India.Email: mv joshi@daiict.ac.in

Abstract—In this paper, we propose a multi-resolution imagefusion approach based on multistage guided filter (MGF). Giventhe high spatial resolution panchromatic (Pan) and high spectralresolution multi-spectral (MS) images, the multi-resolution imagefusion algorithm obtains a single fused image having boththe high spectral and the high spatial resolutions. Here, weextract the missing high frequency details of MS image by usingmultistage guided filter. The detail extraction process exploits therelationship between the Pan and MS images by utilizing oneof them as a guidance image and extracting details from theother. This way the spatial distortion of MS image is reducedby consistently combining the details obtained using both typesof images. The final fused image is obtained by adding theextracted high frequency details to corresponding MS image.The results of the proposed algorithm are compared with thecommonly used traditional methods as well as with a recentlyproposed method using Quickbird, Ikonos-2 and Worldview-2satellite images. The quantitative assessment is evaluated usingthe conventional measures as well as using a relatively new indexi.e., quality with no reference (QNR) which does not require areference image. The results and measures clearly show that thereis significant improvement in the quality of the fused image usingthe proposed approach.

I. INTRODUCTION

The commercial remote sensing satellites such as Quick-bird, Ikonos-2 and Worldview-2 are used to collect hugevolume of data for various applications. Due to the hardwarelimitations of the satellite equipment, especially related to theradiometric resolution of detectors, the satellite sensor capturesone Pan image and a number of MS images for a particularscene. Since the former has higher spatial information andthe MS image has the higher spectral information, a fusionof the Pan and MS images is quite suitable for use in manyapplications.

A large number of techniques have been proposed for thefusion of Pan and MS images. They have been broadly classi-fied into three different categories as projection-substitutionmethods such as principal component analysis (PCA) andintensity hue saturation (IHS) [1]–[3], the relative spectralcontribution methods based on a linear combination of bands,and finally, the multi-scale or multi-resolution based methodsbased on obtaining a scale-by-scale description of the infor-mation content of both images. Among these methods, themulti-resolution based methods have proven to be successful[1]. Most of the multi-resolution techniques have been basedon wavelet decomposition [4], [5] in which the MS and Panimages are decomposed into approximation and detail sub-

bands, and the detail sub-band coefficients of the Pan imageare injected into the corresponding sub-band of MS image by apredefined rule. This is followed by inverse wavelet transformto obtain the fused image. This concept was taken forward in “atrous” Wavelet Transform (ATWT) based fusion [5] in whichthe image is convolved with cubic spline filter and decomposedinto wavelet planes. Context Based Decision (CBD) fusionmethod [6] is an improved and successful version of ATWT. Itproposes to add the wavelet coefficients not directly but afterweighting them with a constant which is computed based onlocal correlation of MS and Pan images. Another techniqueknown as additive wavelet luminance proportional (AWLP)[7] is based on ATWT method. It intends to preserve theradiometric signature between the bands of the MS image byinjecting high frequency values proportional to their originalvalues. The spectral response of Ikonos in [3] shows that thereexist considerable difference between the bands of the MSimage and the Pan image. So, the methods involving extractionof details from the Pan image and its injection into the MSimage lead to inconsistency while injecting the Pan details.So, a new multi-resolution based technique, which takes intoaccount the characteristics of multiple information sourcessimultaneously was introduced. This method was based on dualbilateral filter [8]. Though this approach takes into accountboth images while extracting detail bands, there is no clearrelationship established between the Pan and the MS images.Since the Pan and MS images correspond to the same scenecaptured with the same sensor, there exists a definite relationbetween their detail bands. We make use of this relationshipin our proposed approach. Assuming that this relationship islinear (this assumption is justified by the experimental results),we derive a guided multistage filter in which the Pan imageis used as a guidance image for the MS image while fusing.The proposed method of fusion is accomplished by using amultistage form of guided filter. A recently proposed methodmakes use of the guided filter for general image fusion [9].

The outline of the paper is as follows. In Section II, ageneral description of the multistage guided filter is presented.Enhancing the spatial resolution of MS images is discussed inSection III. Section IV contains the experimental results andtheir comparison with other methods. Finally, Section V drawsthe conclusions of the proposed method.

II. MULTISTAGE GUIDED FILTER (MGF)

The guided filter [10], is an edge preserving smoothingfilter. It is based on the assumption that the filtered image

has a linear relationship with a guidance image. It has beenexperimented successfully for a variety of applications includ-ing detail enhancement and compression, flash/no-flash de-noising etc [10]. For an input image p, the guided filter makesthe assumption that the filtered output q is composed as alinear transformation (in a strict sense: incrementally lineartransform) of the guidance image I in any local window wkcentered at a pixel k [10]

qi = akIi + bk,∀i ∈ wk, (1)

where wk is a window of square shape with size r×r, r beingan integer. Here, Ii represents the ith pixel intensity. The linearcoefficients ak and bk are constant in wk and are computedby minimizing a cost function modeled as a squared differencebetween the input image p and the output image q. The costfunction which is to be minimized can be given by [10]

E(ak, bk) = Σi∈wk((akIi + bk − pi)2 + εa2k), (2)

where ε is a regularization parameter which needs to be setby the user. The parameters ak and bk are given by [10],

ak =

1|w|Σi∈wkIipi − µkpk

σ2k + ε

, and bk = pk − akµk, (3)

where σ2k and µk are the variance and mean of I in wk, |w|

is the number of pixels in wk, and pk = 1|w|Σi∈pipi is the

mean of p in wk. All the overlapping windows wk which covera pixel i, have different values of ak and bk. This providesdifferent values of qi in equation (1) as it is calculated onvarious windows. As a remedy, an average of all the obtainedvalues of ak and bk for the overlapping windows is computed.Then, the filtering output can be given as follows [10]:

qi = aiIi + bi, (4)

where ai = 1|w|

∑k∈wi ak and bi = 1

|w|∑k∈wi bk. Equations

(3) and (4) combinedly represent the guided filter.

For the purpose of fusing Pan and MS images, guided filteris extended to a multistage guided filter. In order to get theconsistency in extracted details, either of the Pan or MS imageis used as a guidance image for the other as depicted below.Since the Pan and MS images have different sizes, resamplingis done on the MS image to obtain the same size for both.At the first stage (j = 1), to extract the details from the Panimage the MS image is used as guidance image (I) and Panimage is used as an input image (p). This can be representedas,

GF (Pan) = F (Pan,MS), (5)

where GF represents the guided filtered output and F representsthe guided filter function. Similarly, the details are extractedfrom the MS image by considering the Pan image as a guidanceimage (I) and MS image as an input image (p) which can begiven as,

GF (MS) = F (MS,Pan). (6)

For the jth stage (j > 1), guided filter equations can bedepicted as follows:

GF j(Pan) = F (Panj−1A ,MSj−1D ) and (7)

GF j(MS) = F (MSj−1A , Panj−1A ), (8)

Fig. 1. Block Diagram of the proposed approach

where Panj−1A and MSj−1A are the approximation layers(guided filtered outputs) of Pan and MS images, respectivelyfor (j− 1)th level. The detail layers PanjD and MSjD for thejth decomposition stage for the Pan and the MS images canbe expressed as,

PanjD = GF j−1(Pan)−GF j(Pan) and (9)

MSjD = GF j−1(MS)−GF j(MS), (10)

where GF j−1(Pan) and GF j−1(MS) correspond to originalPan and MS images for j = 1 i.e. first stage.

III. THE PROPOSED APPROACH

The proposed fusion technique is outlined in Fig. 1. Themultistage guided filter is used to combine the details of thePan and MS images in a consistent manner. Note that unlikethe conventional methods, information is not extracted onlyfrom the Pan and injected into the MS image. Rather the detailshere are extracted from both the images in which each one isfunctioning as a guidance image. The detail is then adjustedusing a weighted average to achieve a fused image with higherspatial as well as higher spectral content. Given an MS image(MS) set with four bands (R, G, B, NIR) and the Pan image(Pan), the proposed fusion technique obtains the fused image(F ) for each band. The steps of the proposed fusion methodcan be detailed as follows:

1) The low resolution (LR) MS image is resampled tothe size of Pan image by using a suitable interpo-lation technique (bicubic interpolation is normallypreferred).

2) In our approach, the four band MS image needsto be converted into a single image. For this, theintensity component I which can be obtained usingthe advanced form of generalized IHS (GIHS) [3] isused. Based on the spectral response of the satellitesensors, I represents weighted average of the four MSbands [3] i.e.,

I = (a1 ∗B + a2 ∗G+R+NIR)/4, (11)

where B, G, R and NIR denote the blue, green, redand near-infrared bands of the MS image data set. Inthe proposed method, the constant a1 is set to 0.25and a2 as 0.75.

3) The intensity component I and Pan images are usedfor multistage guided filter decomposition given inequations (5) to (10) and the detail and approximation

layers are obtained. We use two stage (N = 2) guidedfilter decomposition.

4) In order to combine the extracted details in a con-sistent manner, they are merged band wise using aweighted gain. The gain factor gk, calculated sepa-rately pixel wise for each band, is given by

gk(i) =MSk(i)

I(i)k ∈ {R,G,B,NIR} , (12)

where (i) denotes the pixel location in an image.5) Finally, the sum of extracted detail bands is merged

band wise with the MS image after weighting withgk,

Fk = MSk + gk

N∑j=1

PanjD, (13)

where N represents the number of stages for guidedfilter. Here, the first term i.e. the MS image con-tains spectral information while spatial informationis carried by the second term i.e. the weighted sumof extracted details.

IV. RESULTS AND DISCUSSION

The performance of the proposed method is evaluated byconducting the experiments on various satellite sensors. Weuse the dataset of images captured using Ikonos-2, Quickbirdand Worldview-2 satellites. These images were downloadedfrom internet [11]–[13], respectively. The images of Ikonos-2 satellite has MS image of 4-m resolution and Pan imageof 1-m resolution. They correspond to an area of MountWellington near Hobart Tasmania and the results using theseare displayed with the color composition of bands 4, 3 and2 in the first row of Fig. 2. QuickBird satellite providesPan and Ms images with radiometric resolution of 0.6-m and2.4-m, respectively. The results of Quickbird satellite imagewhich covers an area around Boulder city, USA are displayedwith color composition of bands 3, 2 and 1 in second rowof Fig. 2. Similarly, Worldview-2 satellite has a resolution of0.5-m for Pan image and with 2-m resolution for MS image.The third row in Fig. 2 corresponds to the results using 4, 3and 2 bands of Worldview-2 satellite which covers an areaof San Francisco, USA. We mention that different windowsize (r) is used for all three datasets for the proposed methodwhile the regularization parameter ε is chosen as 10−6 forall experiments. The values of r are 2, 7 and 9 for Ikonos-2, Quickbird and Worldview-2 satellite images, respectively.These parameters are selected by trial and error approach. In allour experiments, the size of MS and Pan images are 256×256and 1024×1024, respectively. These datasets are co-registeredfirst. The experimental results for the proposed method arecompared with popular methods such as GIHS [3], AWLP[7], CBD [6] and multi-scale dual bilateral filter (MDBF)[8]. Based on Wald’s protocol [14], the MS and Pan imagesare spatially degraded and the experiments are carried out onthem to quantitatively test the output of the proposed approachby analyzing the fused MS and the true MS images. Thevalues of correlation coefficient (CC) [7], erreur relative globaladimensionnelle de synthse (ERGAS) [7], average quality(QAVG) [7], root mean square error (RMSE) [7] have beenevaluated by conducting the experiment on degraded versionsof MS and Pan images. These measures are displayed in Table I

for three different satellite images. The experiments are alsoconducted by analyzing the MS images without degradationand the results are displayed in Fig. 2. Due to space constraintwe have not displayed the results with degraded versions ofMS and Pan images. The QNR measure [15] that does not usea reference (true) image is also listed in the Table shown. TheQNR [15] measure is a combination of the spectral and spatialdistortions which are represented by Dλ and Ds respectivelyin the table.

Following observations can be made from the fused imageresults displayed in Fig. 2. In the results of the GIHS methoddisplayed in Fig. 2(a) we see that the distortion is present dueto the lack of spectral information. In Fig. 2(b) we display theresults of CBD method. This method has poor spatial fidelity.This may be due to inconsistency in combining the images.The results of AWLP and MDBF methods are displayed inFig. 2(c) and Fig. 2(d), respectively. In Fig. 2(e) we displaythe results of the proposed method. For better visualization wealso display the magnified version of small area at top rightcorner of each image. Comparing the results of the proposedmethod with the results of AWLP and MDBF approaches, onecan clearly see that the proposed method performs better inenhancing the features as well as in preservation of spectralinformation.

Table I lists the quantitative measures. We display the idealvalue of the quantitative measure in bracket as a reference. Theboldface value in the Table indicates that the value is closer tothe ideal value for that method. In terms of parameters amongthe five methods, on an average, GIHS technique stands last forall three image sets. It is bettered by CBD method followed bythe AWLP method. For Quickbird and Ikonos-2 satellite imagesets, MDBF gives better results than GIHS, CBD and AWLPapproaches while for Worldview-2 satellite image set, CBDmethod outperforms MDBF method and stands second afterthe proposed method. In contrast, the average performance ofthe proposed method is better for images from all the threesatellites.

All the experiments are conducted on intel core i5 (3rdGen.) processor with 2 GB RAM using Matlab 7.14. Theaverage time to run the algorithm of the proposed method isfew seconds only.

V. CONCLUSION

The proposed fusion method uses a new technique to bal-ance both spatial as well as spectral distortion. The multistageform of guided filter is introduced which extract details fromPan image. The extracted details are transferred to MS imagevia a weighted average gain factor. The results have beencompared with state of the art methods by conducting theexperiments on images of Quickbird, Ikonos-2 and Worldview-2 satellites. The experimental results and quantitative measuresclearly outlines that there is significant improvement in thequality of fused image with this method.

REFERENCES

[1] C. Thomas, T. Ranchin, L. Wald, and J. Chanussot, “Synthesis ofmultispectral images to high spatial resolution: A critical review offusion methods based on remote sensing physics,” IEEE Trans. Geosci.Remote Sens., vol. 46, no. 5, pp. 1301–1312, 2008.

Ikonos-2 Imagery

Quickbird Imagery

Worldview-2 Imagery(a) (b) (c) (d) (e)

Fig. 2. Multi-resolution image fusion results. Fused images obtained with (a) GIHS method [3], (b) CBD method [6], (c) AWLP approach [7] , (d) MDBFmethod [8] and (e) proposed approach. The Magnified region of a small square are shown in (e) is displayed at top right corner.

TABLE I. QUANTITATIVE MEASURES FOR DIFFERENT SATELLITE IMAGES SHOWN IN FIG. 2

Method GIHS CBD AWLP MDBF Pro- GIHS CBD AWLP MDBF Pro- GIHS CBD AWLP MDBF Pro-[3] [6] [7] [8] posed [3] [6] [7] [8] posed [3] [6] [7] [8] posed

Ikonos-2 Imagery Quickbird Imagery Worldview-2 ImageryCC(1) 0.980 0.960 0.964 0.946 0.973 0.764 0.830 0.850 0.861 0.859 0.908 0.882 0.916 0.901 0.925

ERGAS(0) 10.929 5.272 5.511 8.337 4.160 47.948 5.104 4.801 4.682 4.661 9.031 9.351 7.612 10.735 7.334QAVG(1) 0.860 0.920 0.896 0.871 0.906 0.506 0.623 0.645 0.871 0.705 0.665 0.645 0.675 0.629 0.695RMSE(0) 21.816 6.923 7.636 11.493 5.645 113.368 35.265 32.745 32.048 31.959 30.448 24.309 20.915 30.910 19.807QNR(1) 0.429 0.268 0.576 0.609 0.640 0.283 0.764 0.765 0.728 0.760 0.476 0.712 0.614 0.634 0.615Ds(0) 0.429 0.581 0.299 0.253 0.228 0.452 0.133 0.163 0.177 0.173 0.360 0.214 0.239 0.178 0.195Dλ(0) 0.248 0.359 0.177 0.183 0.171 0.482 0.117 0.085 0.113 0.079 0.255 0.092 0.196 0.228 0.236

[2] P. Chavez and A. Kwarteng, “Extracting spectral contrast in landsat the-matic mapper image data using selective principal component analysis,”Photogramm. Eng. Remote Sens., vol. 55, pp. 339–348, 1989.

[3] T. Tu, P. Huang, C. Hung, and C. Chang, “A fast intensity-hue-saturationfusion technique with spectral adjustment for ikonos imagery,” IEEEGeosci. Remote Sens. Lett, vol. 1, no. 4, pp. 309–312, 2004.

[4] S. Li, J. Kwok, and Y. Wang, “Using the discrete wavelet frametransform to merge landsat tm and spot panchromatic images,” Inf.Fusion, vol. 3, no. 1, pp. 17–23, 2002.

[5] J. Nunez, X. Otazu, O. Forsa, A. Prades, V. Pala, and R. Arbiog, “Mul-tiresolution based image fusion with additive wavelet decomposition,”IEEE Trans. Geosci. Remote Sens., vol. 37, no. 3, pp. 1204–1211, 1999.

[6] B. Aiazzi, L. Alparone, S. Baronti, and A. Garzelli, “Context-drivenfusion of high spatial and spectral resolution images based on over-sampled multiresolution analysis,” IEEE Trans. Geosci. Remote Sens.,vol. 40, no. 10, pp. 2300–2312, 2002.

[7] X. Otazu, M. Gonzales-Audicana, O. Fors, and J. Nunezi, “Introductionof sensor spectral response into image fusion methods, application towavelet-based methods,” IEEE Trans. Geosci. Remote Sens., vol. 43,pp. 2376–2385, 2005.

[8] J. Hu and S. Li, “Fusion of panchromatic and multispectral imagesusing multiscale dual bilateral filter,” in IEEE Inf. Conf. Image Proc.,2011.

[9] S. Li, X. Kang, and J. Hu, “Image fusion with guided filtering,” IEEETrans. Image Process., vol. 22, pp. 2864–2875, 2013.

[10] K. He, J. Sun, and X. Tang, “Guided image filtering,” IEEE Trans.Pattern Anal. Mach. Intell., vol. 35, no. 6, pp. 1397–1409, 2013.

[11] [Online]. Available: http://www.isprs.org/data/ikonos hobart/default.aspx

[12] [Online]. Available: http://www.digitalglobe.com/product-samples[13] [Online]. Available: http://www.digitalglobe.com/downloads/microsites/

dfc/optical/WV2 9Oct2011.zip[14] L. Wald, T. Ranchin, and M. Mangolini, “Fusion of satellite images of

different spectral resolutions: Assessing the quality of resulting images,”Photogramm. Eng. Remote Sens., vol. 63, no. 6, pp. 691–699, 1997.

[15] L. Alparone, B. Aiazzi, S. Baronti, A. Garzelli, F. Nencini, andM. Selva, “Multispectral and panchromatic data fusion assessmentwithout reference,” Photogramm. Eng. Remote Sens., vol. 74, no. 2,pp. 193–200, 2008.