SEMI-ONLINE VIDEO STABILIZATION USING PROBABILISTIC KEYFRAME UPDATEAND INTER-KEYFRAME MOTION SMOOTHING

Juhan Bae1,2, Youngbae Hwang1 and Jongwoo Lim2

1Multimedia IP Center, Korea Electronics Technology Institute2Department of Computer Science and Engineering, Hanyang University

jhbae@keti.re.kr1,2, ybhwang@keti.re.kr1, jlim@hanyang.ac.kr2

ABSTRACT

In this paper, we propose a video stabilization method thattakes advantages of both online and offline video stabiliza-tion methods in a semi-online framework. Our approach takesthe fixed length incoming frames for having the advantagesof offline methods with the same length of delayed results.We stabilize input frames by warping to the keyframe to in-crease the visual stability. For preserving user intent cameramotion correctly, we determine the next keyframe update bymeasuring inconsistency between a current keyframe and in-coming frames. Moreover, inter-frame motion smoothing byquadratic fitting bridges the keyframes smoothly for pleasantviewing experiences. Our algorithm not only handles rapidcamera motion changes, but also stabilizes the input camerapath smoothly in real-time. Experimental results show thatthe proposed algorithm is comparable to the state-of-the-artoffline video stabilization methods with only fixed length ofincoming frames.

Index Terms— Semi-online, Video Stabilization, Adap-tive Keyframe Update, Inter-keyframe Smoothing

1. INTRODUCTION

Video camera systems are widely installed in many daily andintelligent applications such as security systems and robotics.Especially in intelligent security systems, it is necessary tohave a stable image sequence directly from the camera forprompt response. Accordingly, both the immediate stabiliza-tion result and pleasant visual quality are key factors for eval-uating the quality of video stabilization.

Video stabilization is categorized into online and offlineapproaches by whether to stabilize the current frame directly.In the previous offline methods, Matsushita et al.[1] stabilizeda video by smoothing an estimated 2D affine camera pathwith motion inpainting technique. Grundmann et al. [2][3]synthesized a stabilized video by smoothing 2D camera mo-tion based on cinematography rules using linear programmingframework and mixture of homographies for rolling shutterremoval. Offline approaches show visually robust due to theoptimized estimation using the full motion chain. However,

offline methods are not applicable for real-time applicationssince full motion chain should be computed prior to the mo-ment of displaying stabilization results.

For online methods, simultaneous motion estimation andsmoothing are performed for immediate stabilization. Previ-ously Kalman filter [4][5], low-pass filtering [6] and motionvector integration [7] are applied for smoothing the originalmotion trajectory. Regarding the direct stabilization results,they are suitable for real-time applications. However, onlinemethods are not able to utilize further motion chain at the mo-ment of displaying and users may experience the stabilizationresults not robust to severe motion and parallax changes.

In this paper, we propose a semi-online video stabiliza-tion method that combines the advantages of both online andoffline approaches. It stabilizes the input frames by warpingto the keyframe for piecewise static motion. Our algorithmupdates the keyframe at the maximum discrepancy betweenincoming frames and the current keyframe for a moving cam-era. This enables the proposed method to be robust to rapidmotion changes with varying update frequency. We smoothlyconcatenate between the previous and current keyframe byquadratic fitting with the continuity constraint for the visu-ally comfortable conversion. In the experimental results, theeffectiveness of the proposed method is shown by the com-parisons of the smoothed motion trajectory and InterframeTransform Fidelity.

2. SEMI-ONLINE VIDEO STABILIZATION

The major difference between online and offline video stabi-lization is whether to use future information against the timeof displaying a stabilized result td. If the incoming camerapath is not known, the large motion drift that is the gap be-tween the original path and the smoothed path is easily ap-peared at online methods when the motion change is consid-erable. For the offline approaches, stabilized results are notlive at td, but visually robust against for the severe and unex-pected camera motion.

To combine the benefits from both online and offlinemethods, we propose a semi-online video stabilization method

Fig. 1. Semi-online video stabilization with inter-keyframemotion smoothing.as shown in the Fig.1. We take N incoming frames for thefuture motion at td. An initial keyframe is set as the firstframe and the proposed algorithm estimates the differencebetween the keyframe and incoming frames. If the maximumdifference is lower than a certain threshold, we consider thatthe input motion is close to static and perform the keyframebased stabilization. Otherwise, we update the keyframe at themaximum difference and perform inter-keyframe smoothingby quadratic fitting for reducing large deviation compared tothe input motion. If the proposed method is reached at thenewly updated keyframe, it resumes the process of determin-ing the next keyframe within N incoming frames, or remainsstatic.

2.1. Inter-keyframe motion smoothing

Video stabilization based on cinematographic principles [2]shows the most pleasant viewing experiences by modifyingthe input camera path into a static, constant velocity and ac-celeration path. Our algorithm approximates the cinemato-graphic rules in a semi-online framework. Our algorithm uti-lizes N future motions at td that cause N frame dealy of sta-bilization result. For trivial camera motion changes, our ap-proach warps the current frame I(t) to the keyframe Ik(t)for a static camera motion where k is the keyframe index. Ifthe camera path is drifted apart from the keyframe, the algo-rithm determines the next keyframe at the maximum differ-ence. Therefore, it is crucial to bridge between the previousand the current keyframe smoothly.

SupposeA(t) is inter-frame motion from frame t−1 and t.C(t) is the input camera pose at frame t and it can be writtenas :

C(t) = A(t)C(t−1),⇒ C(t) = A(t)A(t−1) . . . A(1) (1)

Let M(t) be the smoothed camera path and it can be de-scribed as :

M(t) = C(t)−1B(t), (2)

where B(t) is the update motion.To approximate the cinematographic rules, the algorithm

estimates B(t) as the quadratic motion. Static camera mo-tion can be covered in the keyframe based stabilization. If thecamera motion is in constant velocity with jitters, quadratic

estimation is considered as linear motion as shown in theFig.1. The estimated motion is approximated as quadraticwhen the camera motion is in constant acceleration. Accord-ingly, the algorithm performs the least-squares quadratic fit-ting on each of the affine motion parameter chain (horizontaland vertical scale, translation, skew and rotation) with N in-coming frames. For visual smoothness of the keyframe con-version, the algorithm satisfies the continuity constraint thatthe beginning slope of the newly smoothed chain is the sameas the end slope of the previously smoothed chain.

2.2. Keyframe update

Adequate keyframe update on our framework can prevent thepropagation of accumulated errors and large inconsistency be-tween the input and smoothed path. Consequentially, esti-mating the moment of proper keyframe update is essentialfor minimizing the path drift compared to the keyframe. Ifthe frequency of keyframe update is too long, the smoothedpath is far deviated from the original path when the motionchanges are considerably large. When the update frequency istoo short, the effect of stabilization is decreased since the dis-crepancy between the input and the smoothed path is reducedwhile the algorithm tracks the rapid camera motion well. Tomake an appropriate keyframe update, the inconsistency withthe keyframe should be measured. Our algorithm aims to findthe maximum change from the keyframe by measuring thedifference of time, motion and photometric error.

If we set r ∈ Ωk, where Ωk consists of N incomingframes of Ik(t) and r is the index of incoming frames, themotion discontinuity wd(r) consists of two zero-mean Gaus-sian function:

wd(r) = Gm(‖M(r)− C(k)‖2) ·Gt(‖r − k‖), (3)

where Gt() and Gm() measures the difference of a motionof frame r and time compared to the keyframe respectively.However, the term of motion changes (‖M(r) − C(k)‖2)mainly depends on the translational motion while the rest ofcomponents are negligible.

To make a correct measurement of the deviation, wetake into account the photometric difference wp(r) by thezero-mean Gaussian of photometric errors Gp() against thekeyframe and it is shown as below

wp(r) = Gp(R(r)), (4)

where R(r) is the root mean square error between I(r) andthe current keyframe Ik(t). R(r) is defined as

R(r) =

√√√√ ∑j=R,G,B

Dj(r)2

3, Dj(r) = Ijk(t)− Ij(r), (5)

where j and Dj are the color channel variations and channelcolor difference respectively. Adaptive keyframe selection

(a) Video 1 (slow and periodic motion) (b) Video 2 (rapid horizontal motion) (c) Video 4 (static camera with vertical motion)

Fig. 2. Motion smoothing results

decides the next keyframe at the maximum difference fromthe keyframe. The index of next keyframe is set as

r = argminr

wd(r) · wp(r), (6)

and the next keyfrmae is set as Ik+1(t) = I(r). The algo-rithm sets the next keyframe when the wd(r) · wp(r) exceedsa certain threshold Tp. In our implementation, we set N=30as acceptable delay, the standard deviation of Gt() as 20 andthe standard deviation ofGm() andGp() as 10 for higher sen-sitivity on motion changes and photometric errors. We set Tpas 2 · 10−5 for the balanced stabilization result.

3. EXPERIMENTAL RESULT

We run our algorithm on the Intel i7 3.4 GHZ Quad-Coremachine with 8G RAM. We extract good features to trackfeatures [9] and use KLT tracker [10] with Homography-RANSAC based rejection for estimating inter-frame motion.The proposed algorithm is applied on 5 selected from videostabilization dataset [2] and our own CCTV videos. It runsin real-time with a single core on 640x480 resolution inputvideo (average 34 fps). We use N=60 that 30 incomingframes are used as the future motion and previous 30 framesas the past motion for Offline Gaussian method [1]. OnlineGaussian methods takes previous 30 frames. We use σ = 20for both online and offline Gaussian methods for strong sta-bilization effect. Deshaker [8] and L1-optimal camera path[2] implemented in OpenCV that stabilize the video by fullyoptimized path with automatic trimming are the one of thestate-of-the-art video stabilization algorithms in the offlineframework and they are used as the ground truth withouttrimming for comparison.

Fig.2 shows the stabilized translational camera path. Forslow and periodic camera motion in Fig.2(a), the proposed

Table 1. Mean and standard deviation of absolute differenceof affine parameters(horizontal and vertical scale, rotation,translation and skew) on stabilization results compared to theground truth. We use Deshaker for slow and periodic cam-era motion and L1-optimal camera path for the rapid cameramotion as the ground truth.

Deshaker [8] L1 [2]Offline Online

ProposedOffline Online

ProposedGaussian[1] Gaussian Gaussian[1] Gaussian

sx 0.004 0.051 0.009 0.271 0.124 0.009(0.003) (0.028) (0.008) (0.256) (0.114) (0.005)

sy 0.004 0.045 0.013 0.033 0.04 0.01(0.003) (0.031) (0.009) (0.24) (0.036) (0.006)

angle 0.009 0.009 0.008 0.041 0.013 0.003(0.005) (0.005) (0.005) (0.037) (0.009) (0.002)

tx 11.05 33.21 8.91 54.41 203.51 32.67(7.03) (19.48) (4.93) (67.31) (163.9) (31.16)

ty 5.89 7.26 5.88 8.41 16.15 6.23(2.92) (5.22) (4.17) (5.99) (16.01) (6.26)

sk 0.014 0.014 0.012 0.099 0.031 0.013(0.007) (0.006) (0.008) (0.088) (0.022) (0.012)

method shows less deviation from the ground truth and thestabilized path is passing between online and offline Gaussianmethods. The algorithm is robust for rapid motion in Fig.2(b)by adaptive keyframe update that maintains more consistencybetween the input and smoothed paths than online and offlineGaussian methods as shown in the Fig.3. The effectiveness ofthe proposed method becomes better when the static camerawith trembling motion is used. The algorithm effectively in-creases the stability by reducing the effect of periodic motioneven compared to L1 and Deshaker in Fig.2(c).

In Table.1, quantitative evaluation of the stabilizationquality is shown compared to the ground truth. Preservinguser intent camera motion is evaluated by mean difference andstability is measured by standard deviation with the ground

(a) Input (b) Offline Gaussian[1] (c) Online Gaussian (d) Deshaker[8] (e) L1 [2] (f) Proposed

Fig. 3. Stabilzation results. From the video 1-3 (V1-V3 in row 1-3), videos from hand-held camera are used and CCTV videos4-5 (V4-V5 in row 4-5) with vertical the unwanted motion are used for stabilization comparison.

Table 2. ITF comparison (dB)Input Deshaker L1[2]

Offline OnlineProposedGaussian[1] Gaussian

V1 20.01 24.23 24.81 23.66 24.29 24.55V2 18.22 24.87 23.59 24.56 24.30 24.66V3 17.40 21.57 21.56 18.32 19.05 21.91V4 25.09 31.15 31.17 29.04 30.34 31.76V5 26.94 33.61 33.32 32.32 32.36 33.07

Fig. 4. Stabilization results with varying N

truth. For slow camera motion, translation is the most varyingparameters and the proposed method preserves the user in-tent motion better than online and offline Gaussian cases. Inthe case of rapid motion, the proposed method shows bettertracks and stability as well while online and offline Gaussianshows large deviation from the input path.

In addition, quantitative results by Interframe Transfor-mation Fidelity(ITF ) [11] that measures the inter-frame sim-ilarity shows that our approach are comparable with offlinemethods for both moving and static cameras in spite of onlyusing N future frames information (Table.2). Especially forthe static CCTV camera, the proposed methods obtains even

better stability than Deshaker and L1-optimal camera pathdue to the keyframe warping with fewer updates.

The effect of a window size on motion smoothing isshown in Fig.4. Online Gaussian method gains more stabilityas higher N is applied. The proposed method shows bet-ter stability by suppressing small bounces of the smoothedpath effectively than the online Gaussian method with strongmotion smoothing with larger N .

4. CONCLUSION

We have presented a semi-online algorithm to stabilize theinput video in real-time. The essence of our algorithm isto stabilize the video with only N future frame. Adaptivekeyframe update and interframe motion smoothing enablesthe proposed method comparable to state-of-the-art offlinemethods as Deshaker and L1-optimal camera path.

5. ACKNOWLEDGMENT

This work was partially supported by the IT R&D program ofMKE/KEIT. [10040246, Development of Robot Vision SoC/Modulefor acquiring 3D depth information and recognizing objects/faces]

This work was partially supported by the IT R&D program ofMKE/KEIT (No. 10040246) and the Center for Integrated SmartSensors as Global Frontier (CISS-2011-0031868).

This research was partially supported by the Cross-MinistryGiga KOREA Project of the Ministry of Science, ICT and FuturePlanning, Republic of Korea(ROK). [GK130100, Development ofInteractive and Realistic Massive Giga-Content Technology]

6. REFERENCES

[1] Y. Matsushita, E. Ofek, Weina Ge, Xiaoou Tang, and Heung-Yeung Shum, “Full-frame video stabilization with motion in-painting,” PAMI, IEEE Transactions on, vol. 28, no. 7, pp.1150–1163, 2006.

[2] M. Grundmann, V. Kwatra, and I. Essa, “Auto-directed videostabilization with robust l1 optimal camera paths,” in CVPR,2011 IEEE Conference on, 2011, pp. 225–232.

[3] M. Grundmann, V. Kwatra, D. Castro, and I. Essa,“Calibration-free rolling shutter removal,” in ICCP, 2012 IEEEInternational Conference on, 2012, pp. 1–8.

[4] Chuntao Wang, Jin-Hyung Kim, Keun-Yung Byun, JiangqunNi, and Sung-Jea Ko, “Robust digital image stabilization usingthe kalman filter,” Consumer Electronics, IEEE Transactionson, vol. 55, no. 1, pp. 6–14, 2009.

[5] Junlan Yang, D. Schonfeld, Chong Chen, and M. Mohamed,“Online video stabilization based on particle filters,” in ImageProcessing, 2006 IEEE International Conference on, 2006, pp.1545–1548.

[6] A. J. Crawford, H. Denman, F. Kelly, F. Pitie, and A.C.Kokaram, “Gradient based dominant motion estimation withintegral projections for real time video stabilisation,” in ImageProcessing, 2004. ICIP ’04. 2004 International Conference on,2004, vol. 5, pp. 3371–3374 Vol. 5.

[7] S. Battiato, G. Gallo, G. Puglisi, and S. Scellato, “Sift featurestracking for video stabilization,” in ICIAP 2007. 14th Interna-tional Conference on, 2007, pp. 825–830.

[8] Gunnar Thalin, “Deshaker,” http://www.guthspot.se/video/deshaker.htm.

[9] Jianbo Shi and Carlo Tomasi, “Good features to track,” in 1994IEEE Conference on Computer Vision and Pattern Recognition(CVPR’94), 1994, pp. 593 – 600.

[10] Carlo Tomasi and Takeo Kanade, “Detection and tracking ofpoint features,” Tech. Rep., International Journal of ComputerVision, 1991.

[11] C. Morimoto and R. Chellappa, “Evaluation of image stabiliza-tion algorithms,” in Acoustics, Speech and Signal Processing,1998. Proceedings of the 1998 IEEE International Conferenceon, May 1998, vol. 5, pp. 2789–2792 vol.5.