MULTIRESOLUTION FUSION USING CONTOURLET TRANSFORM BASED EDGELEARNING

Kishor P. Upla, Prakash P. Gajjar, Manjunath V. Joshi

Dhirubhai Ambani - Institute of Information and Communication Technology, Gandhinagar, India.{kishor upla, prakash gajjar, mv joshi}@daiict.ac.in

ABSTRACT

In this paper, we propose a new approach for multi-resolution fusionof remotely sensed images based on the contourlet transform basedlearning of high frequency edges. We obtain a high spatial resolution(HR) and high spectral resolution multi-spectral (MS) image usingthe available high spectral but low spatial resolution MS image andthe Panchromatic (Pan) image. Since we need to predict the missinghigh resolution pixels in each of the MS images the problem is posedin a restoration framework and is solved using maximum a posteriori(MAP) approach. Towards this end, we first obtain an initial approx-imation to the HR fused image by learning the edges from the Panimage using the contourlet transform. A low resolution model isused for the MS image formation and the texture of the fused imageis modeled as a homogeneous Markov random field (MRF) prior. Wethen optimize the cost function which is formed using the data fit-ting term and the prior term and obtain the fused image, in which theedges correspond to those in the initial HR approximation. The pro-cedure is repeated for each of the MS images. The advantage of theproposed method lies in the use of simple gradient based optimiza-tion for regularization purposes while preserving the discontinuities.This in turn reduces the computational complexity since it avoids theuse of computationally taxing optimization methods for discontinu-ity preservation. Also, the proposed method has minimum spectraldistortion as we are not using the actual Pan digital numbers, in-stead learn the texture using contourlet coefficients. We demonstratethe effectiveness of our approach by conducting experiments on realsatellite data captured by Quickbird satellite.

Index Terms— Contourlet transform, MRF prior, Fusion

1. INTRODUCTION

In remote sensing, obtaining images with high spatial and spectralresolutions from a satellite is a difficult task due to hardware lim-itations. Degradation occurs in the captured MS images when onetries to capture them with high spectral and spatial resolution. Multi-resolution fusion is a method of combining a high spatial resolutionPan image and a low spatial but high spectral resolution MS imagein order to obtain a high spatial and spectral resolution MS image[1]. Due to the physical constraint involving a trade-off betweenspatial and spectral resolutions, a great amount of research is beingcarried out in this field. Several approaches have been proposed toaddress the problem of multi-resolution fusion for remote-sensingapplications. Some of the earlier approaches include Intensity-hue-saturation (IHS) transform technique [2, 3, 4], High-pass filtering(HPF) [5, 6] approach. Several approaches are also proposed basedon the principal component analysis and the wavelet transform (WT)technique [7, 8]. The WT technique is somewhat similar to the HPF

where in the high frequency components of the MS images are re-placed by that of the Pan image in the wavelet domain. More re-cent works on the multiresolution fusion can be found in [9, 10, 11].Recently the authors [12] proposed a model based approach for mul-tiresolution fusion. Here the authors used a decimation model for theimage formation and the problem was solved using MAP estimation.A Inhomogeneous Gaussian Markov Random Field (IGMRF) wasused as the prior. The prior model parameters were estimated usingthe available HR Pan image. Howerver, their method seems to sufferfrom the following drawbacks. Since the Pan image has low spectralresolution, it does not represent a better approximation to the fusedimage. Hence the estimated IGMRF parameters tend to approximatetheir true values [13]. Also, these parameters get saturated when theadjacent pixels in the fused image have equal intensities leading tonumerical instability while optimizing. Another factor that affectsthe performance in [12] is the spectral distortion in the fused imagedue to learning of the spatial relationship entirely based on the Pandata. Our approach in this paper differs from their approach in thefollowing way. We obtain the initial HR estimate (approximation tofused MS image) using the contourlet based learning in which thePan image as well as the corresponding MS image data are consid-ered (discussed in the next section) This captures the smooth contouredges (i.e., spatial dependencies) more effectively while preservingthe spectral content. We model each of the high resolution fusedMS images as separate non-discontinuity preserving homogeneousMRFs and a single model parameter for each is estimated from theirinitial HR approximations. A canny edge detector is used to locatethe edge pixels in the learned image which are retained as the edgepixels in the final fused image. We then use an MAP-MRF approachto obtain the final solution. The optimization is carried out onlyon those pixels which do not belong to the edge pixels as the edgepixels correspond to those obtained using the contourlet based learn-ing. This ensures edge preservation as well as reduction in com-putational complexity since a simple gradient based method can beemployed for optimization. This in turn avoids the use of compu-tationally taxing optimization methods as used in super-resolutioncommunity [14, 15] for edge preservation. In addition the use ofMRF parameter estimated using the initial HR estimate avoids theneed for tuning this unknown parameter during iterative optimiza-tion process and further reduces the time complexity.

2. BLOCK DIAGRAM DESCRIPTION AND CONTOURLETBASED EDGE LEARNING

The block diagram shown in Fig.1 illustrates the fusion process fornth low resolution MS image and Pan image giving the fused MSimage as the result. The initial HR approximation is obtained usingthe Pan image and the MS image. We consider contourlet trans-

523978-1-4577-1005-6/11/$26.00 ©2011 IEEE IGARSS 2011

Fig. 1. Block diagram of multiresolution fusion process for a MSand the Pan image. Here LR and HR correspond to low resolutionand high resolution, respectively.

(a) (b)

Fig. 2. (a) Two level contourlet transform of an MS image, and (b)four level contourlet transform of PAN image.

form to recover the high frequency details for the fused MS image.The Pan image is decomposed to four levels of contourlet transform(Fig.2(b)) and the MS image into to two levels (Fig.2(a)). This cor-responds to a decimation factor of q = 4 i.e., it leads to the fused MSimage that has 4 times the size of observed LR MS image. Learningis done by copying the contourlet transform coefficients of the Panimage that correspond to the high frequency details (third and fourthlevels) to third and fourth levels of the unknown fused MS image.The initial approximation to the fused MS image is then obtainedby taking the inverse contourlet transform. We use this approxi-mation to extract the edges in the fused image and to estimate thealiasing/decimation as well as the MRF prior model parameter. Thedecimation estimation block has the inputs as the MS and the initialHR approximation and gives decimation matrix coefficients as theoutput. Finally, the cost function consisting of data fitting term andthe prior term is minimized using the MAP-MRF framework.

3. FORWARD MODEL AND DECIMATION ESTIMATION

Since we cast the problem in a restoration framework, solving such aproblem needs a forward model that represents the image formationprocess. Let Y be the observed MS image of the size M ×M pixels

and Z be the fused HR image, then the forward model for the imageformation can be written as,

y = Dz + n, (1)

where y and z represent the lexicographically ordered vectors of sizeM2 × 1 and q2M2 × 1, respectively. D is the decimation matrixwhich takes care of aliasing. For an integer decimation factor ofq, the decimation matrix D consists of q2 non-zero elements alongeach row at appropriate locations. Here n is the independent andidentically distributed (i.i.d.) noise vector with zero mean and vari-ance σ2

n and has same size as y. Now the problem can be specifiedas: estimating fused z given y, which is an ill-posed inverse prob-lem. It may be mentioned here that the observation (test image) isnot blurred. In other words, we assume identity matrix for blur. Thisdecimation model is used to obtain the aliased pixel intensities fromthe high resolution MS pixels. In general the decimation matrix canbe in written as in [16], in which the non zero entries are all of equalvalue. In our work since the close approximation to the fused image(i.e., initial HR estimate) is already available, we estimate the deci-mation entries using the LR test image and the initial HR estimate.A simple least squares method is used for the same.

4. MRF PRIOR MODEL AND FUSION USING MAP-MRFAPPROACH

The use of MAP estimation for fusion requires a suitable prior forthe same. MRF has emerged as a popular stochastic model for im-ages due to its ability to capture local dependencies. A method ofspecifying MRF prior involves considering the pair wise cliques con a neighborhood and imposing a quadratic cost which is a func-tion of finite difference approximations of the first order derivativeat each pixel location. This constitutes a homogeneous and non edgepreserving smoothness prior. By using first order neighborhood, theenergy function corresponding to the MRF prior can be written as,

∑c∈C

Vc(z) = γ

N1∑k=1

N2∑l=1

[(zk,l − zk,l−1)2 + (zk,l − zk−1,l)

2],

where z is the lexicographically ordered high resolution fused imageand γ represents the penalty for departure from smoothness in z. Cis the set of all cliques. The MRF parameter γ is estimated usingthe initial HR estimate. We use a simple approximate scheme calledmaximum pseudo liklihood for estimating the γ [17].

The MRF model on the fused image serves as the prior for theMAP estimation in which the prior parameter is already known. Thedata fitting term contains the decimation matrix estimated again us-ing the initial HR approximation. For MAP-MRF approach, it can beshown that the final cost function to be minimized can be expressedas,

z = argminz

[‖ y − Dz ‖2

2σ2n

+∑c∈C

Vc(z)

]. (2)

In (2), the first term ensures the fidelity of the final solution to the ob-served data through the image formation process. The second termis the smoothness prior. Since this cost function is convex, it can beminimized using the gradient descent optimization technique, whichquickly leads to the minima. We mention here again that in order topreserve the edges in the final solution we detect the edges from theinitial HR estimate with the help of canny edge detector and we donot perform the optimization on those edge pixels. In other wordsthe edges in the final fused image correspond to the already learned

524

Image Correlation coefficientHPF Method Proposed

Method in approach[5] [12]

Band 1 0.8611 0.8890 0.9304Band 2 0.8673 0.9161 0.9337Band 3 0.8585 0.9138 0.9100Band 4 0.8497 0.8438 0.8754

SSIMBand 1 0.8241 0.7125 0.7751Band 2 0.7805 0.7609 0.8400Band 3 0.8039 0.7565 0.8106Band 4 0.7617 0.7238 0.8105

Table 1. Performance comparison in terms of Correlation coefficientand SSIM.

edges in the initial HR estimate. Since the optimization process isiterative the choice of initial solution fed to the optimization processdetermines the speed of convergence. Use of the available HR ap-proximation as an initial solution speed-up the convergence. It maybe mentioned here that we obtain initial HR approximation sepa-rately for each of the MS observations and the optimization is carriedout independently for every LR MS observation.

5. EXPERIMENTAL RESULTS

In this section, we present the results of the proposed method for fu-sion. The experiments are conducted on real images captured usingQuickbird satellite. The original Pan image and the MS images areof size 512× 512 and 128× 128, respectively. In order to make thequantitative comparison possible, we downsapmled the images by afactor of 4 and conducted the experiments on Pan and MS imagesof size 128 × 128 and 32 × 32, respectively. The size of the fusedMS images is 128 × 128. We compare the performance of the pro-posed method with other methods on the basis of quality of images interms of qualitative as well as quantitative measures. Fig.3 show theresults of fusion for Band-1 to Band-4 using different approaches.In Table 1 we show the comparison using correlation coefficient andstructural similarity (SSIM) [18, 19] as the quantitative performancemeasures. We can see that proposed approach gives better quantita-tive values as compared to other approaches except for the correla-tion coefficient value for Band -3. From these results we concludethat the proposed method is better when compared to the other MSfusion approaches.

6. CONCLUSION

We have presented a new technique to recover the high spatial andhigh spectral resolution MS image using contourlet based edgelearning and MAP-MRF approach. Since the final solution is ob-tained using canny edge detector and MRF prior, the suggestedmethod gives finer details present in different directions with mini-mum spectral distortion. The results demonstrate that the proposedtechnique yields better solution as compared to those obtained usingthe recent approaches.

7. REFERENCES

[1] L. Wald, “Some terms of reference in data fusion,” IEEE Trans.Geosci. Remote Sensing, vol. 37, no. 3, pp. 1190–1193, 1999.

[2] R. Hayden, G. W. J. Henkel, and J. E. Bare, “Application ofthe IHS color transform to the processing of multisensor dataand image enhancement,” in Proc. International Symposiumon Remote sensing of arid and semi-arid lands, Cairo, Egypt,1982, pp. 599–616.

[3] M. Gonzalez, A. Audicana, J. L Saleta, R. Garcia Catalan, andR. Garcia, “Fusion of multispectral and panchromatic imagesusing improved IHS and PCA mergers based on wavelet de-composition,” IEEE Trans. Geosci. Remote Sensing, vol. 42,no. 6, pp. 1291–1299, June 2004.

[4] M. Choi, “A new intensity-hue-saturation fusion approach toimage fusion with a tradeoff parameter,” IEEE Trans. Geosci.Remote Sensing, vol. 44, no. 6, pp. 1672–1682, June 2006.

[5] T. Ranchin, B. Aiazzi, L. Alparone, S. Baronti, and L. Wald,“Image fusion - the ARSIS concept and some successful imple-mentation schemes,” ISPRS Journal of Photogrammetry andRemote Sensing, vol. 58, no. 1-2, pp. 4–18, June 2003.

[6] P. S. Jr. Chavez, S. C. Sides, and J. A. Anderson, “Comparisonof three different methods to merge multiresolution and mul-tispectral data: Landsat TM and SPOT panchromatic,” Pho-togrammetric Engineering and Remote Sensing, vol. 57, no. 3,pp. 295–303, 1991.

[7] H. Li, B. S. Manjunath, and S. K. Mitra, “Multisensor imagefusion using the wavelet transform,” Graphical Models andImage Processing, vol. 27, no. 3, pp. 235–244, 1995.

[8] E. Yu and R. Wang, “Fusion and enhancement of the multi-spectral image with wavelet transform,” Computer Engineer-ing and Science, vol. 23, no. 1, pp. 47–50, 2001.

[9] D. Fasbender, J. Radoux, and P. Bogaert, “Bayesian data fu-sion for adaptable image pansharpening,” IEEE Trans. Geosci.Remote Sensing, vol. 46, no. 6, pp. 1847–1857, June 2008.

[10] P. S. Pradhan, R. L. King, N. H. Younan, and D. W. Hol-comb, “Estimation of the number of decomposition levelsfor a wavelet-based multiresolution multisensor image fusion,”IEEE Trans. Geosci. Remote Sensing, vol. 44, no. 12, pp. 3674– 3686, 2006.

[11] M. Choi, R. Y. Kim, M. R. Nam, and H.O. Kim, “Fusionof multispectral and panchromatic satellite images using thecurvelet transform,” IEEE Trans. Geosci. Remote Sensing, vol.2, no. 2, pp. 136–140, April 2005.

[12] M. V. Joshi and A. Jalobeanu, “MAP estimation for multireso-lution fusion in remotely sensed images using an IGMRF priormodel,” IEEE Trans. Geosci. Remote Sensing, vol. 48, no. 3,pp. 1245–1255, March 2010.

[13] J. Zerubia A. Jalobeanu, L. Blanc-Feruad, “An adaptive gaus-sian model for satellite image deblurring,” IEEE Trans. ImageProcessing., vol. 14, no. 4, pp. 613–621, 2004.

[14] D. Rajan and S. Chaudhuri, “Simultaneous estimation ofsuper-resolved scene and depth map from low resolution de-focused observations,” IEEE Trans. on Pattern Analysis andMachine Intelligence, vol. 25, no. 9, pp. 1102–1117, 2003.

[15] A. N. Rajagopalan and S. Chaudhuri, “An MRF based ap-proach to simultaneous recovery of depth and restoration fromdefocussed images,” IEEE Trans. on Pattern Analysis and Ma-chine Intelligence, vol. 21, no. 7, pp. 577–589, July 1999.

525

Band 1

Band 2

Band 3

Band 4(a) (b) (c) (d)

Fig. 3. Results for q = 4. (a) MS images, fused images obtained using (b) HPF method, (c) method proposed in [12], and (d) proposedapproach.

[16] R. R. Schultz and R. L. Stevenson, “A Bayesian approach toimage expansion for improved definition,” IEEE Trans. on Im-age Processing, vol. 3, no. 3, pp. 233–242, May 1994.

[17] S. Z. Li, Markov Random Field Modeling in Computer Vision,Springer-Verlag, Tokyo, 1995.

[18] Z. Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli, “Im-age quality assessment: From error visibility to structural sim-ilarity,” IEEE Transactions on Image Processing, vol. 13, no.4, pp. 600–612, Apr 2004.

[19] Z. Wang and A. C. Bovik, “Mean squared error: love it or leave

it?,” IEEE Signal Processing Magazine, pp. 98–117, 2009.

526