Digital Video/Image Stabilization - unict.itbattiato/mm1213/Part 10 Digital_Video...Optical Image Stabilization • Optical Image Stabilization (OIS), by using some mechanical or electronic

Digital Video/Image Stabilization

Introduction

• In the last decade multimedia devices (camcorders, PDAs, mobile phones) have been dramatically diffused. Moreover the increasing of their computational performances combined with an higher storage capability permits them to elaborate large amount of data. These devices, typically small and thin, usually have video acquisition capability.

• However making a stable video with these devices is a very challenging task especially when a zoom lens or a digital zoom is used. Due to user’s hands shake, the recorded videos suffer from annoying perturbations.

• The same problem arises in presence of cameras placed on mobile supports (car, airplane) or fixed cameras operating outdoors. The atmospheric conditions (e.g. the wind) and the vibrations created by vehicles passing make unstable the recorded video.

Video Stabilization

• Video stabilization allows to acquire video sequences without disturbing jerkiness, removing unwanted camera movements.

• Videos quality is then improved and the higher level algorithms present in the device (segmentation, tracking, recognition) can also work properly.

• Moreover higher bit rate compression can be obtained from stabilized video with respect to the unstable one.

Optical Image Stabilization

• Optical Image Stabilization (OIS), by using some mechanical or electronic tools, measures camera shake and then control the jitter acting on lens.

• Horizontal and vertical vibration are usually detected using gyroscopic sensor.

• Image stabilization is performed by using floating lens (Canon, Nikon) element that vary the optical path to the sensor.

• Both steps are applied before the acquisition avoiding any post-processing computation and image deformation.

• Requires high expensive optical systems and enough space around the objective of the camera making difficult the integration in very small and thin systems like imaging phones.

Digital Video Stabilization (1)

• Digital Video Stabilization (DIS) techniques make use only of information retrieved from the analysis of the video to estimate physical camera motion.

• Some extra computational cost and the risk of generating image deformation.

• No additional mechanical apparatus (gyroscopic sensor, floating lens, etc.).

• These approaches may be implemented easily both in real-time and post-processing systems.


IMAGE

WARPING

MOTION

FILTERING

MOTION

ESTIMATION

Stable Frame Unstable Frame

Digital video stabilization algorithms, in general, are made up of three

stages:


IMAGE

WARPING

MOTION

FILTERING

MOTION

ESTIMATION



stages:

Motion estimation is devoted to find the parameters relative to the

transformation occurred between adjacent frames. Translational, similarity and

affine are the most common adopted motion models.


IMAGE

WARPING

MOTION

FILTERING

MOTION

ESTIMATION



stages:

Motion filtering discriminates intentional motion (panning) from the unwanted motion (jitter). Typically motion smoothness consideration are taken into account in the filtering process (jitter is an high frequency signal).


IMAGE

WARPING

MOTION

FILTERING

MOTION

ESTIMATION



stages:

Image warping consists of the reconstruction of the stabilized image through a properly warping. In order to do so, sensor dimensions must be higher than the final produced video.


• Motion estimation is devoted to find the parameters relative to the transformation occurred between adjacent frames. Translational, similarity and affine are the most common adopted motion models.

• Motion filtering discriminates intentional motion (panning) from the unwanted motion (jitter). Typically motion smoothness consideration are taken into account in the filtering process (jitter is an high frequency signal).

• Image warping consists of the reconstruction of the stabilized image through a properly warping. In order to do so, sensor dimensions must be higher than the final produced video.

Motion Estimation: camera

movements

Taken from: “Qualitative estimation of camera motion parameters from the linear composition of optical flow”, S. Park, H. Lee, S. Lee. Pattern Recognition 37 (2004) 767-779.

Motion Estimation: camera

movements (2) Even if the motion in the video is three-dimensional, global motion between

frames is typically estimated with two-dimensional models:

yif

xif

Tyy

Txx

yiif

xiif

Tyxy

Tyxx

cossin

sincos

yiif

xiif

Tdycxy

Tbyaxx

2D Translational

model

2D Similarity

model

2D Affine

model

2 parameters: horizontal

(Tx) and vertical shift (Ty)

4 parameters: 2 shifts

(Tx,Ty), a rotation () and

an isotropic scale variation

()

6 parameters: shifts,

rotation, anisotropic scale

variation, shear

Motion Estimation

• The problem of motion estimation has been widely investigated and many solutions have been proposed. The existing approaches can be classified in two categories:

• Direct methods aim to recover the unknown parameters through global minimization criteria based on direct image information. Some assumptions (e.g., brightness constancy) are typically used as starting point.

• Feature based approaches first locate a sparse set of reliable features in the image and then recover the motion parameters considering their correspondences.

Optical Flow (1)

• Human eyes perceive motion matching

corresponding points in different time.

• The correspondences are typically done

assuming no (or small) variation of color or

brightness after the motion.

• In computer vision, the apparent 2D

motion of each pixel is know as optical

flow.

Optical Flow (2)

Examples of optical flow. Taken from: Determining Optical Flow, Berthold K.P. Horn and

Brian G. Schunck.

Optical Flow (3)

The optical flow sometimes differs from the true motion field. On the left a sphere is rotating

under a constant ambient illumination (the observed image does not change). On the right a

point light source is rotating around a stationary sphere causing an apparent motion.

Taken from: “Video Processing and Communication”, Yao Wang, John Ostermann, Ya-Qin

Zhang.

Block Matching (1)

• Image partitioning into square blocks;

• Only translational movements of each

block;

• The correspondence of blocks is done

within a search windows;

• A matching criteria drives the translational

parameters estimation.

Block Matching (2)

Block matching approaches divide the image in square blocks (in

black) and search the corresponding one in the consecutive

frame. The matching procedure is performed within a search

window (dark gray).

Block Matching: Full Search

• Let NxN be the image size, MxM the block size

and R the search range.

• By using an exhaustive search strategy we have

to perform about N2(2R+1)2 operations. Notice

that we have supposed N multiple of M and SAD

(sum of absolute differences) has been used as

matching criterion.

• If N=512 and R=16 for each frame 2.85*108

operations are needed.

Block Matching: Three Step

Search (1) • The search starts with a step equal to half

search range;

• It evaluates 9 points (one in the centre and

8 in the search area boundaries);

• After each step the stepsize is halved;

• The minimum error point become the

novel centre;

• The iterations end when stepsize=1.

Block Matching: Three Step

Search (2)

TSS Example.

Number of operations

required for each block

Phase correlation (1)

Let fk(x,y) and fk+1(x,y) be two function that satisfy the following relation:

),(),( 211 dxdxfyxf kk

According to the Fourier shift property we have:

)(2

121),(),(

vdudj

kk evuFvuF

Where F(u,v) represents the Fourier transform of f(x,y).

Phase correlation (2)

The normalized cross power spectrum is:

)(2

*

*

1 21

),(),(

),(),( vdudj

kk

kk evuFvuF

vuFvuF

The translational parameters can be easily recovered performing the

inverse Fourier transform of the normalized cross power spectrum.

Inverse Fourier transform of the normalized cross power spectrum (c)

corresponding to the image pair (a) and (b). The peak coordinates quantify images

displacement ((b) is a translated version of (a)).

Feature Matching (1)

• In computer vision a lot of feature points have

been developed (Harris, SIFT, etc.). They have

been used in many applications: wide baseline

matching, object recognition, image retrieval,

robot localization, building panoramas and

recognition of object categories.

• Due to the local feature properties, even video

stabilization approaches make use of them

trying to find a good compromise between

accuracy and complexity.

Feature Matching (2)

An example of feature matching.

In order to properly work, the interest points must be reliable detected (by

means of detectors) in the image even if have been applied some

transformation (rotation, scale, viewpoint, etc.). These points are typically

recognized in image regions with significative two dimensional intensity

changes.

Once the points have been detected, the local image structure they

represent must be properly codified through descriptors. These

descriptors must be invariant, as much as possible, with respect to both

geometric and photometric image transformation.

Critical Conditions (1)

• Very often, in real videos, there are many conditions that degrade, if not properly managed, the performances of video stabilization algorithms.

• In presence of homogeneous regions, periodic patterns and fast illumination changes the local motion estimators sometimes produce wrong vectors.

• Moreover the movement of the objects in the scene can mislead the global motion vector estimation. Although their vectors are correctly computed they do not describe camera movements.

• Finally zooming and forward walking of the user can create some problem if they are not taken into account in the motion model (e.g., a translational motion model).


homogeneous

regions moving objects

periodic patterns


• In order to cope with wrong vectors a lot of robust estimator methods have been developed (Ransac, Iterative Least Squares, etc.).

An example of outliers filtering. On the left all the vectors produced by a

Block matching algorithm. After the filtering step only the reliable

vectors remain (on the right).

Motion Filtering

Discriminates intentional motion (panning) from the

unwanted motion (jitter):

• Motion Vector Integration;

• Frame Position Smoothing;

• Kalman Filtering.

An example of motion filtering. Taken from Erturk, S. (2001). Image sequence stabilization based on

Kalman filtering of frame positions. IEE Electronics Letters, 37(20), 1217 - 1219.

Motion Vector Integration (MVI)

• The Global Motion Vector is integrated with a damping

coefficient δ.

• The Integrated Motion Vector (IMV) represents the

correction to be applied in order to stabilize the video

sequence.

• The damping factor is chosen between 0 and 1

depending on the desired trade off between canceling

low amplitude jitter and following intentional motion with

low delay.

Frame Position Smoothing

• The absolute displacement curve is smoothed by using a lowpass filter (Gaussian, Butterworth, etc.);

• The correction vector is given by the difference between the original and the smoothed curve;

• Successful stabilization is achieved;

• Requires offline processing not suited for real-time applications.

Kalman filtering

• Kalman filter provides the estimation of the state of a dynamic system from a set of noisy measure;

• Typically a constant velocity model is used to represent the intentional motion;

• The unwanted movement is considered as noise.

kkk

kkk

vHxz

wFxx

1

F is the state transition model;

wk is the process noise;

H is the observation model;

vk is the observation noise.

Image Stabilization (1)

• Image stabilization is a key element introduced in the still cameras in order to reduce the motion blur.

• This kind of degradation plagues photography since its early days and is due to the relative movement between the camera and the scene during the exposure (integration) time.

• Image stabilization can be very useful in low light conditions and in the case of small mobile cameras, particularly with high zoom.

• Moreover due to the sensor miniaturization (smaller pixel area) longer integration time is needed.

Image Stabilization (2)

• Some approaches (optical image

stabilization) effectively remove motion

blur but involve extra hardware, hence

extra costs.

• On the other hand, digital stabilization,

considering only information related to

acquired images is a low expensive

solution of the problem.

Digital Image Stabilization

Techniques • Many digital image stabilization techniques have been developed. If the

PSF (Point Spread Function) is known, the original image can be restored applying an image de-convolution approach.

• However in real cases PSF is unknown and blind de-convolution approaches have been studied. These approaches, considering very simple motion model (constant velocity motion or linear harmonic motion) obtain poor result in real cases and are very time consuming.

• Recently some techniques synthesize blur free images considering only several images acquired with different exposure time. In these approaches image stabilization is performed through the fusion of a series of low-exposed images. These images usually have no blur (low exposure time) but contain high noise level.

• In order to minimize the computational efforts and memory requirement some approaches consider only a pair of images (instead of a sequence), typically a low-exposed image (no blur but high noise level) together with a normal exposed image (high blur degradation but low noise level).

Digital Image Stabilization

Example

Image Stabilization on a real image: (a) low exposed image shot

(1/100 sec), (b) high exposed image shot (1 sec), (c) the

restored image. Taken from “Motion Blur Identification Based on

Differently Exposed Images”, Marius Tico, Mejdi Trimeche,

Markku Vehvilainen.

References (1)

http://www.canon.com/bctv/faq/vari.html

Adams, A., Gelfand, N. & Pulli, K. (2008). Viewfinder alignment. Computer Graphics Forum, 2, 597–606.

Digital video stabilization architecture for low cost devices. In Proceedings of International Symposium on Image and Signal Processing and Analysis, 474–479, Zagreb (Croatia).

Baker, S. & Matthews, I. (2004). Lucas-kanade 20 years on: A unifying framework. International Journal of Computer Vision.

Battiato, S. & Lukac, R. (2008). Encyclopedia of Multimedia, chap. Video Stabilization Techniques, 941–945. Springer-Verlag.

Battiato, S., Gallo, G., Puglisi, G. & Scellato, S. (2007). SIFT features tracking for video stabilization. In Proceeding of International Conference on Image Analysis and Processing (ICIAP), 825–830, Modena (Italy).

Battiato, S., Bruna, A.R., & Puglisi, G. (2008). A Robust Video Stabilization System by Adaptive Motion Vectors Filtering. Proceedings of ICME, Hannover Germany.

Battiato, S., Puglisi, G., & Bruna, A.R. (2008). Regular Texture Removal for Video Stabilization, International Conference on Pattern Recognition (ICPR), Florida USA.

References (2)

Censi, A., Fusiello, A. & Roberto, V. (1999). Image stabilization by features tracking. In Proceeding of International Conference on Image Analysis and Processing (ICIAP), 665–667, Venice (Italy).

Chang, H.C., Lai, S.H. & Lu, K.R. (2004). A robust and efficient video stabilization algorithm. In Proceedings of International Conference on Multimedia and Expo (ICME), 29–32, Taipei (Taiwan).

Chang, J., Hu, W., Cheng, M. & Chang, B. (2002). Digital image translational and rotational motion stabilization using optical flow technique. IEEE Transactions on Consumer Electronics, 48.

del Blanco, C.R., Jaureguizar, F., Salgado, L. & Garc´ıa, N. (2008). Automatic feature-based stabilization of video with intentional motion through a particle filter. In Proceedings of Advanced Concepts for Intelligent Vision Systems (ACIVS), vol. 5259 of Lecture Notes in Computer Science, 356–367, Juan-les-Pins (France), Springer.

Erturk, S. & Dennis, T.J. (2000). Image sequence stabilisation based on dft filtering. In IEE Proceedings of Vision, Image and Signal Processing, vol. 147, 95–102.

Erturk, S. (2001). Image sequence stabilization based on Kalman filtering of frame positions. IEE Electronics Letters, 37(20), 1217 - 1219.

References (3)

Fischler, M.A. & Bolles, R.C. (1981). Random sample consensus: a paradigm model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381–395.

Harris, C.G. & Stephens, M. (1988). A combined corner and edge detector.In In Proceeding of Alvey Vision Conference, 147–151, Manchester, UK.

Horn, B. & Schunck, B. (1981). Determining optical flow. Artificial Intelligence, 185–203.

Irani, M. & Anandan, P. (1999). About direct methods. In Proceedings of International Workshop on Vision Algorithms, held during ICCV , 267–277, Corfu, Greece.

Jang, S.W., Pomplun, M., Kim, G. & Choi, H.I. (2005). Adaptive robust estimation of affine parameters from block motion vectors. Image and Vision Computing, 1250–1263.

Jim´enez, H. & Salas, J. (2008). Dynamic selection of characteristics for feature based image sequence stabilization. In Proceedings of Advanced Concepts for Intelligent Vision Systems (ACIVS), vol. 5259 of Lecture Notes in Computer Science, 290–297, Juan-les-Pins (France), Springer.

References (4)

Koga, T., Iinuma, K., Hirano, A., Iijima, Y. & Ishiguro, T. (1981). Motion compensated interframe coding for video conferencing. In Proceedings of National Telecommunications Conference, vol. 4, G5.3.1–G5.3.5.

Kwon, O., Shin, J. & Paik, J.K. (2005). Video stabilization using kalman filter and phase correlation matching. In Proceeding of International Conference on Image Analysis and Recognition (ICIAR), vol. 3656 of Lecture Notes in Computer Science, 141–148, Springer.

Lowe, D. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60, 91–110.

Lukac, R. (2008). Single-Sensor Imaging: Methods and Applications for Digital Cameras. CRC Press.

Matsushita, Y., Ofek, E., Ge, W., Tang, X. & Shum, H.Y. (2006). Full-frame video stabilization with motion inpainting. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 28, 1150–1163.

Meer, P., Stewart, C.V. & Tyler, D.E. (2000). Robust computer vision:an interdisciplinary challenge. Computer Vision and Image Understanding, 78, 2000.

Mikolajczyk, K. & Schmid, C. (2004). Scale & affine invariant interest point detectors. International Journal of Computer Vision (IJCV), 60, 63–86.

References (5)

Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., Kadir, T. & Gool, L.V. (2005). A comparison of affine region detectors. International Journal of Computer Vision (IJCV), 65, 43–72.

Park, S., Lee, H., Lee, S. (2004) Qualitative estimation of camera motion parameters from the linear composition of optical flow. Pattern Recognition 767-779.

Tico, M., Alenius, S. & Vehvilainen, M.. (2005). Constraint motion filtering for video stabilization. In Proceeding of International Conference on Image Processing (ICIP), Genova, Italy.

Tico, M., Alenius, S. & Vehvilainen, M. (2006). Method of motion estimation for image stabilization. In Proceeding of International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toulouse (France).

Torr, P.H.S. & Zisserman, A. (1999). Feature based methods for structure and motion estimation. In Proceedings of International Workshop on Vision Algorithms, held during ICCV , 278–294, Corfu, Greece.

Wang, Y., Ostermann, J., Zhang, Y. Video Processing and Communication. Prentice Hall.

Yang, J., Schonfeld, D., Chen, C. & Mohamed, M. (2006). Online video stabilization based on particle filters. In Proceedings of International Conference on Image Processing (ICIP), 1545–1548, Atlanta (USA).

References (6)

http://www.canon.com/bctv/faq/optis.html

http://www.canon.com/bctv/faq/vari.html

http://www.nikon.co.jp/main/eng/portfolio/about/technology/nikon_technology/vr_e/index.htm

P.A. Jansson, Deconvolution of image and spectra, Academic Press, 1997.

Tony F. Chan and Chiu-Kwong Wong, “Total Variation Blind Deconvolution,” IEEE Transactions on Image Processing, vol. 7, no. 3, pp. 370–375, 1998.

Yu-Li You and M. Kaveh, “A regularization approach to joint blur identification and image restoration,” IEEE Trans. on Image Processing, vol. 5, no. 3, pp. 416–428, Mar. 1996.

Marius Tico and Markku Vehvilainen, “Robust image fusion for image stabilization,” in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Honolulu, USA, 2007.

Marius Tico, and Markku Vehvilainen, "Image stabilization based on fusing the visual information in differently exposed images," in Proc. of the IEEE International Conference of Image Processing (ICIP), San Antonio, Texas, USA, Sept. 2007.

J. Jia, J. Sun, C. Tang, and H. Shum. Bayesian correction of image intensity with spatial consideration. In European Conference on Computer Vision (ECCV’04), pages 342–354, 2004.

Documents

Digital Video/Image Stabilization - unict.itbattiato/mm1213/Part 10 Digital_Video...Optical Image Stabilization • Optical Image Stabilization (OIS), by using some mechanical or electronic