A New Global Registration Algorithm for Image Mosaic

FULI WUZhejiang University of TechnologyCollege of Information Engineering

No.6 District Zhaohui, 310032 HangzhouChina

fuliwu@zjut.edu.cn

XIANYONG FANGAnhui of University

Key Lab. of Computing & Signal Processing230039 Hefei

Chinafangxianyong@ahu.edu.cn

Abstract: In this paper, we proposed a new bundle adjustment algorithm based on the distance minimization offeature matches, to eliminate the accumulated errors over a sequence for seamless mosaic. The new feature basedimage mosaic approach improved the stability of the global registration among images. Our experiments show thatthe new mosaic method is robust.

Key–Words: Feature based image mosaic, Bundle adjustment, Global registration

1 Introduction

Image mosaic tries to composite several narrow-angleimages into a wide-angle image and is widely used inaerial and satellite photographing [1], virtual touringand exhibition [2-3], photo edition [4], etc. Szeliski[5] reviewed the principles and advances of imagemosaic. As described in [5], there are two types ofmethod: direct method and feature based method, andthe latter becomes more robust than the former withthe advances of feature detection.

Feature based methods [6-9] mosaic the imagesby first automatically detecting and matching the fea-tures in the source images, and then warping theseimages together. Normally it consists of three steps:feature detection and matching, local and global reg-istration, and image composition. Feature detectionand matching aims to detect features and then matchthem. Local and global registration starts from thesefeature matches, locally registers the neighboring im-ages and then globally adjusts accumulated registra-tion error so that multiple images can be finely regis-tered. Image composition blends all images togetherinto a final mosaic. For more details on the currentstate of feature based mosaic, please refer to [5]. Reg-istration is still not finely solved with existing tech-niques and our focus in this paper is also on how toimprove the stability of registration for feature basedmosaic.

Many papers on wide baseline matching [10-12],object recognition [13-14] and image/video retrieval[15-17] consider how to improve the stability of fea-ture matching. In these works, feature matching is im-proved by spatial consistency which means the match

features of each feature and its every neighboring fea-ture should have the same spatial arrangement. Sivicet al. [15] used each region match in the neighbor-hood of each feature match to vote this feature match.The sum of votes of the whole frame decides the rankof the frame and match without vote is rejected. Fer-rari et al. [10, 16] iteratively applied a expansion andcontraction scheme to add new matches and removewrong matches while expansion is fulfilled based onthe similarity of affine transformations between neigh-boring region matches and contraction is reached bythe sidedness constraint which bases on the fact that,to a triple of region matches, the center of a first re-gion should be on the same side of the directed linegoing from the center of a second region to the centerof a third region. The median flow filter [18] is alsoused to remove wrong matches, which compares thelength and anger of each match vector with the medianlength and anger of its several neighboring match vec-tors respectively and selects the one whose length andanger below the thresholds.

But on the image mosaic side, there are few re-searches considering eliminating wrong matches be-fore robust registration. As far as we know, only Choet al. [6] applied the median flow filter to removewrong matches before registration for image mosaic.

For image mosaic, to locally register the neigh-boring images, 8-parameter homography can be ap-plied to accurately model the mapping between viewsunder general image condition. RANSAC [19] is acommonly accepted way to refine the homography be-tween images [7-9] because RANSAC can return thefinal inliers when getting the final homography. Ex-cept RANSAC, LMedS (the Least Median of Squares)

Proceedings of the 7th WSEAS International Conference on Signal, Speech and Image Processing, Beijing, China, September 15-17, 2007 136

[20] is also used for homography calculation, but itcan not return the final inliers.

For global registration, bundle adjustment is awidely acceptable idea. While decomposing the pro-jective transformation matrix into rotation angles andthe focal length is very sensitive to image noise [7],the iterative registration methods [21-22] is complexand weak. Another more robust way is presented byMarzotto et al. [23] where the homography of eachsource image to the reference mosaic surface is ad-justed by minimizing the total misalignment of a pre-defined set of m grid points on the mosaic. With thelocal registration strategy in the previous paragraph,the global registration can be build upon inliers andthe homography between neighboring views. In thispaper, a new bundle adjustment based on the fea-ture matches is proposed so that the homographies be-tween all images and the mosaic surface are adjustedby minmizing the distances of feature matches.

With above discussions, a new robust featurebased image mosaic method can be proposed. First,features are localized with SIFT [24], described byPCA-SIFT [25] and matched with priority search ink-d tree structure [26]. Then the RANSAC homogra-phy algorithm is applied to locally register neighbor-ing images and the bundle adjustment algorithm basedon the distance minimization of feature matches is ap-plied to globally register all the images together. Af-ter registration, image composition can be fulfilled bymulti band blending [27], featuring algorithm [2] orgradient domain fusion [28] and then a final mosaiccan be generated.

In the following paragraphs, the algorithm will bediscussed in detail in the Section 2. Then experimen-tal results will demonstrate the stability and efficiencyof our method. The first step, feature detection andmatching, and the last step, image composition, willnot be discussed here and interested readers can referto related papers.

2 RANSAC Homography for LocalRegistration

After feature detection and matching, the source im-ages need to be registered together. RANSAC algo-rithm can be applied to get the homography of eachimage pair. Four initial putative feature matches areselected in the random selection step of each iterationin RANSAC [19], and a correct homography can begot after one iteration.

3 Bundle Adjustments for GlobalRegistration

With the ransac algorithm, the pair wise homographyof all input images can be got. To register all theseimages together, we must adjust all these calculatedhomographies together. Or there will be accumulatederrors if simply selecting one reference image (the im-age selected as the mosaic surface) and warping allothers to this reference image. So a bundle adjustmentalgorithm based on the minimization of the distancesof feature matches is proposed to remove the accumu-lation of errors.

The reference image should be defined first beforeapplying the bundle adjustment. In our current imple-mentation, the middle image is selected as the refer-ence image so that all images are trying to warp to thecenter and thus will have less accumulated errors thanselecting other images. The best match image of eachimage should also be selected before applying bundleadjustment. In our current implementation, it is theimage which has the maximum number of inliers withthis image and which is processed earlier than this im-age.

The homography of each image to the mosaic sur-face should also be initialized and it can calculated ina recursive way. If the best match image of Image Ia isIb, the homography of Ib to mosaic surface is Hb andthe homography of Ia to Ib is Hab, then the homogra-phy of Ia to the mosaic surface, Ha, is then calculatedby

Ha = Hab ·Hb (1)

After above preparations, the bundle adjuster pro-cessed each source image one by one. For the currentprocessing image In:

◦ Update the homographies of all existing images totheir best match images under the following crite-rion of distance minimization of feature matches:

e =n∑

i=1

∑

j∈L(i)

∑

k∈F (i,j)

f2(rkij) (2)

L(i) is the set of images overlapping with Ia.F (i, j)are the inliers between Ii and Ij . f(x) = xand rij is calculated by warping the feature point inIi with Hi and Hj :

rkij =

∥∥∥xkj −H−1

j Hixki

∥∥∥ (3)

Figure 1 explains Equation(3). For matching fea-tures xi and xj which are in Ii and Ij respectively,xi is warped to the mosaic surface first and then

Proceedings of the 7th WSEAS International Conference on Signal, Speech and Image Processing, Beijing, China, September 15-17, 2007 137

warped to the image surface of Ij . After this twowarps, we can get x

′i and r which equals to xj − x

′i

or xj −H−1j Hixi.

◦ Update the homography of each existing image tothe mosaic surface with the homography of it to itsbest match image. For example, assuming the up-dated homography of Ia to its best match image Ib

is H′ab, then H

′ab ·Hb .

◦ If not converge, go to step 1; else, end.

Figure 1: Distance minimization of feature matches.

The bundle adjuster is implemented by L-M al-gorithm and the derivatives can be computed by thechain rule.

4 ExperimentsFigure 2,3,4 show the bundle adjustment results withfour source images. These images are composed fi-nally by featuring algorithm. In figure 2, there is nobundle adjustment. Ghosting is existed in the image.Figure 3 shows the bundle adjustment result if we donot update the homography of each image with its bestmatch image but the homography of each image tothe mosaic surface. Still there is ghosting existed al-though less than Figure 2. Figure 4 shows the mosaicafter fully applying the bundle adjustment algorithmand there is no ghosting.

Figure 2: Mosaic without bundle adjustment.

Figure 5 shows another example of our mosaicmethod where nine images are finely mosaiced withthe featuring algorithm after applying bundle adjust-ment to these images.

Figure 3: Mosaic after partially applying the bundleadjustment.

Figure 4: Mosaic after fully applying the bundle ad-justment.

5 ConclusionIn this paper, we put forward a robust feature basedimage mosaic method. First features are detectedwith SIFT, described by PCA-SIFT and matched withpriority search. Then images are locally registeredwith the RANSAC homography algorithm. Finallythe images are fused to be one image with a blend-ing method. In the new bundle adjustment algorithm,the homographies of all images to the mosaic surfaceare updated by the distance minimization of featurematches. Experiments show that our feature based im-age mosaic method is highly effective.

Acknowledgements: We would like to express ourthanks to Zhigeng Pan at Zhejiang University and BinLuo at Anhui University for much helpful advice. Thework is under the supports of Nature Science Founda-tion of Zhejiang Province (Grant No. Y105095) andScience Fund for Higher Education of Anhui Province(Grant No. KJ2007B005)

References:

[1] F.H.Moffitt and E.M.Mikhail, Photogrammetry.Harper & Row, New York, 1980.

[2] Richard Szeliski, Video Mosaics for Virtual En-vironments, IEEE Computer Graphics & Appli-cations 16(2)1996, pp. 22–30.

[3] Zhigeng Pan, Xianyong Fang, Jiaoying Shi andetc. EasyTour: a New Image-Based Virtual Tour

Proceedings of the 7th WSEAS International Conference on Signal, Speech and Image Processing, Beijing, China, September 15-17, 2007 138

Figure 5: Example bundle adjustment with 9 Images.

System, In Proceedings of ACM SIGGRAPH In-ternational Conference on Virtual Reality Con-tinuum and Its Applications in Industry, 2004,pp. 467-471.

[4] Shenchang Eric Chen, QucikTime VR-anImage-Based Approach to Virtual EnvironmentNavigation, Computer Graphics,1995, pp. 29-38.

[5] Richard Szeliski, Image Alignment and Stitch-ing: A Tutorial, Technical Report, MSR-TR-2004-92, Microsoft Research, 2004.

[6] Soo-Hyun CHO, Yun-Koo CHUNG and JaeYeon LEE, Automatic Image Mosaic SystemUsing Image Feature Detection and Taylor Se-ries, In Proceedings of the 7th InternationalConference on Digital Image Computing: Tech-niques and Applications, Sydney, Australia,2003, pp. 549-556.

[7] Matthew Brown and David Lowe, RecognisingPanorama, In Proceedings of the 9th Interna-tional Conference on Computer Vision, Nice,France, 2003, pp. 1218-1225.

[8] Philip H. S. Torr and Andrew Zisserman. Fea-ture Based Methods for Structure and MotionEstimation, In Proceedings of the InternationalWorkshop on Vision Algorithms: Theory andPractice,Springer-Verlag,London, United King-dom, 1999, pp. 298-373 .

[9] David Capel and Andrew Zisserman. AutomatedMosaicing with Super-Resolution Zoom, In Pro-ceedings of IEEE Computer Society Conferenceon Computer Vision and Pattern Recognition,Santa Barbara, United States, 1998, pp. 885-891.

[10] Vittorio Ferrari, Tinne Tuytelaars and Luc VanGool, Wide-Baseline Multiple view Correspon-dences, In Proceedings of IEEE Computer So-

ciety Conference on Computer Vision PatternRecognition, Madison, USA, 2003, pp. 718-725.

[11] Jir Matas, Ondrej Chum, Martin Urban andetc, Robust Wide Baseline Stereo from Maxi-mally Stable Extremal Regions, In Proceedingsof British Machine Vision Conference, Cardiff,UK, 2002, pp. 384-393.

[12] Tinne Tuytelaars and Luc Van Gool, Wide Base-line Stereo Matching Based on Local, AffinelyInvariant Regions, In Proceedings of British Ma-chine Vision Conference, Bristol, UK, 2000 ,pp. 412-425.

[13] Vittorio Ferrari, Tinne Tuytelaars and Luc VanGool, Integrating Multiple Model Views for Ob-ject Recognition, In Proceedings of IEEE Com-puter Society Conference on Computer Visionand Pattern Recognition, Washington, USA,2004, pp. 105-112.

[14] Stepan Obdrzalek and Jin Matas, Object Recog-nition Using Local Affine Frames on Distin-guished Regions, In Proceedings of British Ma-chine Vision Conference, Cardiff, UK, 2002,pp. 113-122.

[15] Josef Sivic and Andrew Zisserman, VideoGoogle: a Text Retrieval Approach to ObjectMatching in Videos, In Proceedings of Inter-national Conference on Computer Vision, Nice,France, 2003, pp. 1470-1477.

[16] Vittorio Ferrari, Tinne Tuytelaars and Luc VanGool, Retrieving Objects from Videos Basedon Affine Regions, In Proceedings of EuropeanSignal Processing Conference, Vienna, Austria,2004, pp. 128-131.

[17] Josef Sivic and Andrew Zisserman, Video DataMining Using Configurations of Viewpoint In-variant Region, In Proceedings of IEEE Com-puter Society Conference on Computer Visionand Pattern Recognition, Washington, USA,2004, pp. 488-495.

[18] P. Smith, D. Sinclair, R. Cipolla and etc,Effective Corner Matching, In Proceedingsof the Ninth British Machine Vision Confer-ence, Southampton, United Kingdom, Septem-ber 1998.

[19] Richard Hartley, Andrew Zisserman, MultipleView Geometry in Computer Vision, CambridgeUniversity Press, Cambridge, United Kingdom,2000.

[20] Rachid Deriche, Zhengyou Zhang, Quang-TuanLuon and etc, Robust Recovery of the Epipo-lar Geometry for an Uncalibrated Stereo Rig, InProceedings of European Conference on Com-puter Vision, Stockhom, Sweden, 1994, pp. 567-576.

Proceedings of the 7th WSEAS International Conference on Signal, Speech and Image Processing, Beijing, China, September 15-17, 2007 139

[21] Yaser Sheikh, Yun Zhai and Mubarak Shah,An Accumulative Framework for the Alignmentofan Image Sequence, In Asian Conference onComputer Vision, Jeju, Korea, 2004.

[22] Harpreet S. Sawhney, Steve Hsu and Rakesh Ku-mar, Robust Video Mosaicing through Topol-ogy Inferenceand Local to Global Alignment,In Proceedings of the 5th European Conferenceon Computer Vision,2,Freiburg, Germany, 1998,pp. 103-119.

[23] Roberto Marzotto, Andrea Fusiello and Vitto-rio Murino, High Resolution Video Mosaicingwith Global Alignment, In Proceedings of IEEEComputer Society Conference on Computer Vi-sion and Pattern Recognition,1, Washington,USA, 2004, pp. 692-298.

[24] David Lowe, Distinctive Image Features fromScale-Invariant Interest Points, InternationalJournal of Computer Vision, 60(2), 2004, pp. 91-110.

[25] Yan Ke and Rahul Sukthankar, PCA-SIFT: AMore Distinctive Representation for Local Im-age Descriptors, In Proceedings of IEEE Com-puter Society Conference on Computer Visionand Pattern Recognition, Washington, UnitedStates, 2004, pp. 511-517.

[26] Sunil Arya and David Mount, ApproximateNearest Neighbor Queries in Fixed Dimensions,In Proceedings of the 4th Annual ACM-SIAMSymposium on Discrete Algorithms, Austin,United States, 1993, pp. 271-280.

[27] Peter J. Burt and Edward H. Adelson, A Mul-tiresolution Spline With Application to ImageMosaics, ACM Transactions on Graphics, 2,1983, pp. 217-236.

[28] Anat Levin, Assaf Zomet, Shmuel Peleg and etc,Seamless Image Stitching in the Gradient Do-main, In Proceedings of 8th European Confer-ence on Computer Vision,2, Prague, Czech Re-public, 2004, pp. 377-389.

Proceedings of the 7th WSEAS International Conference on Signal, Speech and Image Processing, Beijing, China, September 15-17, 2007 140