Panoramas from Partially Blurred Video

Jani Boutellier1 and Olli Silven1

1Machine Vision GroupDepartment of Electrical and Information EngineeringP.O.Box 4500, FI-90014 University of Oulu, Finland

{bow, olli.silven}@ee.oulu.fi

Abstract. Numerous high-quality image stitching algorithms have beenpublished in the recent years. Mosaics created by these methods are ofhigh quality if the input images are not distorted. However, if the sourceimages are blurred, parts of the resulting mosaic will be blurred also.In this paper we propose a method to create high-quality panoramasfrom video sequences that contain also low-quality frames. Moreover,our method is computationally efficient, which makes it attractive forhand-held devices. The algorithm uses motion detection to display cor-rectly moving objects in the sequence. The colors of the mosaic are alsobalanced to handle changes in camera exposure times.

1 Introduction

Image stitching is used to combine several images into one wide-angled mosaicimage. Traditionally mosaic images have been constructed from a few separatephotographs, but nowadays that video recording has become commonplace, it ispossible to consider also video sequences as a source for mosaic images. When amosaic image is constructed from single photographs, the procedure is straight-forward because the amount of images is rather limited. With video sequences,the situation is different. Even a short video clip contains vast amounts of data,that is mostly redundant due to large overlaps between frames.

Because of the overlap, it is clear that not all frames from the sequence areneeded to construct a mosaic that covers the whole scene. The frame selectionprocess has been researched only little [1],[2] and until now, it has been assumedthat the video frames are of good quality. Practically this is not the case, sincepictures are often taken freehand, which leads to blurred images in many occa-sions. In this paper we will propose a new method to select the best frames froma video sequence and then create a high-quality panorama from those images.

2 Related Work

The whole process of creating a mosaic image consists of several smaller tasks,of which image registration is the most important one. Stitching algorithms useboth feature-based [2] and direct approaches [3], [4] in the registration process.

The definitions and properties of these two fundamentally different methods areexplained well in the survey of Zitova and Flusser [5].

An image stitching algorithm also needs to use some kind of motion detectionto avoid the distortion of moving objects [3]. This problem and its solutions arecovered extensively in the survey of Radke [6]. In stitching applications, Davis [3]solved the problem by drawing the seams around moving objects by Dijkstra’salgorithm. In the method created by Zhu [4] the moving objects were extractedby differencing three successive frames and defined further by calculating anactive contour for each object.

Once the registration and motion detection is done, the images can be stitched.The stitching process consists of local blending operations and of radiometric ad-justments. The paper of Zomet [7] contains a good comparison of image blendingmethods and also proposes a new approach of optimizing the stitching result bya gradient-based cost function. Szeliski proposed a simple, local blending methodto eliminate seam artifacts [8]. Xiao [9] reduced the exposure differences of animage pair by setting the mean and standard deviation of the registered imageto be the same as that of the reference image.

Fig. 1. Block diagram of our algorithm

3 The Algorithm

The issue of selecting the best frames for an image mosaic has not been addresseduntil now. The work closest to ours has been made by Li [1]. In the method ofLi the amount of distortion caused by rotation and perspective is evaluated anda suitable subset of images is chosen to form a mosaic. Also Hsu [2] has selectedframes for the construction of a panorama. He has only considered the criteriaof suitable overlap.

Our method relies on the recent image registration method of Vandewalle[10], that can register blurred, rotated and translated frames. After registration,

motion detection is performed to each frame. The frames used for stitching areselected by estimating their quality by a couple of different parameters. Ouralgorithm tries to use as few frames as possible, since the seams are the placeswhere the image quality is most probably degraded.

The image sequence is processed in three phases, as depicted in Figure 1. Inthe first phase the frames are registered consecutively, so that each frame n isregistered against frame n-1. The consecutive registration approach ensures thatthe overlap between frames is as large as possible. This makes registration andmotion detection more robust. Simultaneously with the registration, the amountof motion blur is calculated. The best frames are selected for the mosaic based ontheir quality. The choices depend on the amount of moving objects and motionblur in the frames. Also, the frames are selected so that their mutual offset is aslarge as possible.

After the frames have been chosen, the centermost frame is selected as theroot frame. A shortest spanning tree is constructed between frames based ontheir mutual translations. Then each selected frame is re-registered against itsparent [11]. This ensures that the accumulated registration errors from the firstphase disappear and that the registration is performed optimally.

In the final phase of the algorithm, the selected frames are stitched together.In the stitching process we use the method of Xiao [9] to minimize color dif-ferences caused by variable exposure times. Finally, the small geometric andphotometric misalignments are compensated by a bilinear weighting function[8].

3.1 Projection and Camera Motion Model

We used the idea of manifold projection, originally introduced by Peleg [12]. Inmanifold projection a thin strip is taken from the center of each frame to beused in the construction of the mosaic image. Peleg states that the frames canbe registered accurately by a rigid camera motion model if it is assumed thatsignificant motion parallax or change of scale does not occur. Manifold projectionoffers excellent image quality, since frames do not have to be projected to adifferent surface for the construction of the mosaic. Manifold projection is alsoquick to process.

3.2 Motion Blur Estimation

Motion blur is determined simultaneously with the first-phase registration. Theimage registration method of Vandewalle requires the calculation of the ampli-tude spectrum for each image to be registered. The spectrum is now also usedfor blur estimation by calculating the amount of high-frequency components init. Vandewalle states that the frequencies above a certain limit ρmax need to bediscarded in the registration process, since those frequencies contain alias if theimage is blurred. The darkened area in Figure 2 depicts the frequency area fromρmax to the maximum frequency, which we use to estimate the amount of blur.If the sum of the frequencies in the area is small, it means that sharp image

Fig. 2. An amplitude spectrum of a fictional image. The darkened area depicts thehigh-frequency area that is used to estimate the image blur

details are absent due to blurring. A large sum tells us that the frame is free ofblur.

Of course, these values can only be used for comparing frames that depictroughly the same scene, since the scene contents also affect the result.

3.3 Frame Selection

The frame selection process is a matter of weighting the importance of differentframe features. According to experiments, the presence of moving objects isthe most important criteria, since the most severe artifacts are created in thestitching process by moving objects that get clipped. The factor of second highestimportance was chosen to be the amount of blur. Finally, frames that are fartheraway from the previous selected frame, are preferred.

In practice each frame gets a quality value that is calculated from the afore-mentioned factors and this can be implemented successfully in many differentways. The quality values are computed for a certain scope of frames at a time.The scope encompasses the frames that have a suitable translation with respectof the previous selected frame, as depicted in Figure 3.

Fig. 3. The scope of frame selection

3.4 Stitching Phase

Once the best frames have been selected, the mosaic may be created. First ofall, the colors are balanced between adjacent frames in the way that Xiao usedin his paper [9]. The merging of two images is most critical in the area where theimages meet. This seam area can be processed in many ways, but we have chosenbilinear weighting [8], since it produces good results compared to the amountof calculations it requires. It is important to remember that more sophisticatedseam handling is beneficial only when the registration step produces erroneousresults.

In our method blending is done with a gaussian weighting mask similarly to[13] if no moving objects are present. If there are moving objects, the seam isdrawn outside the boundaries of moving objects.

Fig. 4. The images in the top row are a small detail from a mosaic that was createdwithout frame selection. The images in the bottom row show the corresponding areasfrom a mosaic that was constructed with frame selection. All of the images are magnifiedand even the best frames in this sequence were somewhat blurred

4 Experimental Results

The algorithm was tested with many different video sequences containing mov-ing objects and motion blur. The sequences we used were recorded freehand bya couple of different cameras. The consequence of freehand recording was thatthe frames were rotated slightly to unpredictable directions and that some unin-tentional camera tilt was also present. In other words, the sequences were takenin very practical conditions. Mosaic results can be seen in Figure 5 and Figure 6.

The impact of frame selection to the mosaic quality was tested by creating thesame mosaic with blur detection and without it. Corresponding areas from bothmosaics were compared against a blur-free input frame with the standard PSNR[14] measure. The results can be seen in Table 1 and magnified and croppeddetails from both examples are shown in Figure 4.

According to these tests the algorithm produced high-quality results withvery good computational efficiency. On a 3.0 GHz desktop computer runningMatlab 7.1 the execution speed was over 4 frames per second (with frame size352x288).

We used rather broad strips (34% of the frame width) from frames to con-struct the panorama images with manifold projection.

5 Conclusion

We have presented a stitching algorithm with the novel idea of evaluating theframes and selecting the best ones for the mosaic. The presented algorithm iscapable of handling moving objects in the video sequence and can correct prob-lems caused by varying lighting conditions. The method is also computationallyefficient, which makes it attractive for hand-held devices.

Table 1. The effect of blur detection to image quality (PSNR)

Sequence With blur detection No blur detectionExample A 27.0 25.0Example B 29.1 26.8

References

1. Li, J.S., Randhawa, S.: Improved video mosaic construction by selecting a suit-able subset of video images. In: CRPIT ’04: Proceedings of the 27th conferenceon Australasian computer science, Darlinghurst, Australia, Australian ComputerSociety, Inc. (2004) 143–149

2. Hsu, C.T., Cheng, T.H., Beuker, R.A., Horng, J.K.: Feature-based video mosaic.In: Proceedings of the International Conference on Image Processing. Volume 2.,Vancouver, Canada (2000) 887–890

3. Davis, J.: Mosaics of scenes with moving objects. In: CVPR ’98: Proceedings of theIEEE Computer Society Conference on Computer Vision and Pattern Recognition,Washington, DC, USA, IEEE Computer Society (1998) 354–360

4. Zhu, Z., Xu, G., Riseman, E., Hanson, A.: Fast generation of dynamic and multi-resolution 360 degrees panorama from video sequences. In: Proceedings of theIEEE International Conference on Multimedia Computing and Systems. Volume 1.,Florence, Italy (1999) 400–406

5. Zitova, B., Flusser, J.: Image registration methods: a survey. Image and VisionComputing 21(11) (2003) 977–1000

6. Radke, R.J., Andra, S., Al-Kofahi, O., Roysam, B.: Image change detection algo-rithms: a systematic survey. IEEE Transactions on Image Processing 14(3) (2005)294–307

7. Zomet, A., Levin, A., Peleg, S., Weiss, Y.: Seamless image stitching by minimizingfalse edges. IEEE Transactions on Image Processing 15(4) (2006) 969–977

8. Szeliski, R.: Video mosaics for virtual environments. IEEE Computer Graphics &Applications (1996) 22–30

9. Xiao, F., Wu, H.Z., Xiao, L., Tang, Y., Ma, W.J.: Auto method for ambientlight independent panorama mosaics. In: Proceedings of International Conferenceon Machine Learning and Cybernetics. Volume 6., Shanghai, China, IEEE (2004)3851–3854

10. Vandewalle, P., Susstrunk, S., Vetterli, M.: A Frequency Domain Approach toRegistration of Aliased Images with Application to Super-Resolution. EURASIPJournal on Applied Signal Processing (special issue on Super-resolution) (2005)

11. Marzotto, R., Fusiello, A., Murino, V.: High resolution video mosaicing with globalalignment. In: Proceedings of the IEEE Computer Society Conference on Com-puter Vision and Pattern Recognition. Volume 01., Los Alamitos, CA, USA, IEEEComputer Society (2004) 692–698

12. Peleg, S., Herman, J.: Panoramic mosaics by manifold projection. In: CVPR ’97:Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition(CVPR ’97), Washington, DC, USA, IEEE Computer Society (1997) 338–343

13. Heikkila, M., Pietikainen, M.: An image mosaicing module for wide-area surveil-lance. In: VSSN ’05: Proceedings of the third ACM international workshop onVideo surveillance & sensor networks, New York, NY, USA, ACM Press (2005)11–18

14. Bovik, A., Gibson, J., Bovik, A., eds.: Handbook of Image and Video Processing.Academic Press, Inc., Orlando, FL, USA (2000)

Fig. 5. Mosaic images created by our algorithm. The cut in the trees in the top imageis real, not a stitching error

Fig. 6. A 360 degree panorama constructed by our method. For viewing purposes thepanorama is cut in two pieces