Upload
vukien
View
226
Download
5
Embed Size (px)
Citation preview
Full-Frame Video Stabilization with Motion Inpainting
Yasuyuki Matsushita,E l Of kEyal OfekWeina Ge
Xiaoou TangHeung-Yeung Shumg g
IEEE Trans on PAMI, July 2006
OutlineOutline
I t d ti• Introduction• Proposed Method• Experimental results• Quantitative EvaluationQuantitative Evaluation• Computation Cost
C l i• Conclusion
2
IntroductionIntroduction
St bili ti• Stabilization:– Remove undesirable motion caused by
i t ti l h k f h h dunintentional shake of a human hand.• remove high frequency camera motion vs.
completely remove camera motioncompletely remove camera motion.• full frame vs. trimming• motion inpainting vs. mosaicingp g g
3
Input Video
Global Motion Estimation
Motion Smoothing
Local Motion Estimation
Motion Inpainting Image Deblurring
Completion
5Output Video
Global Motion EstimationGlobal Motion Estimation
GM i ti t d b li i i i• GM is estimated by aligning pair-wise adjacent frames.
min( ( ) ( ))I Tp I p–• Hierarchical motion estimation
– construct an image pyramid
'min( ( ) ( ))t t t tTI Tp I p−
– construct an image pyramid– start from the coarsest level
• By applying the parameter estimation forBy applying the parameter estimation for every pair of adjacent frames, a global transformation chain is obtained.j
iT6
iH.-Y. Shum and R. Szeliski, “Construction of Panoramic Mosaics with Global and Local Alignment,” Int’l J. Computer Vision, vol. 36, no. 2, pp. 101-130, 2000.
Input Video
Global Motion Estimation
Motion Smoothing
Local Motion Estimation
Motion Inpainting Image Deblurring
Completion
7Output Video
Image DeblurringImage Deblurring
T f i h i i l• Transferring sharper image pixels from neighboring frames.– evaluates the “relative blurriness”
1b = 2 2y{(( )( )) (( )( )) }
t
tx t t t t
p
bf I p f I p
=⊗ + ⊗∑
– evaluates the “alignment error”'
' '( ) | ( ) ( ) |t t tt t t t t tE p I T p I p= −
8
( ) | ( ) ( ) |t t t t t tp p p
Image DeblurringImage Deblurring
Bl i l l d b i t l ti• Blurry pixel are replaced by interpolating shaper pixels. '
' '( ) ( ) ( )t tt t t t t t tI p w p I T p+ ∑
w is the eight factor hich consists of
'
''
( )1 ( )
t Nt t t
t tt N
I pw p
∈
∈
=+ ∑
• w is the weight factor which consists of the pixel-wise alignment error and relative blurrinessblurriness
'
'
'( )
0 1( )
t
tt
tt
bbt
bt tb E p
ifw p
otherwiseα
α+
⎧ <⎪= ⎨⎪⎩
9
' ' ( )t t tb E p otherwiseα+⎪⎩
Input Video
Global Motion Estimation
Motion Smoothing
Local Motion Estimation
Motion Inpainting Image Deblurring
Completion
11Output Video
Motion SmoothingMotion Smoothing( )iS T G k= ⊗∑
2 2
t
/ 212
( )
{ : }, ( ) ,t
ti N
kt
S T G k
N j t k j t k G k e kσπσ
σ
∈
−
⊗
= − ≤ ≤ + = =
∑
12
Input Video
Global Motion Estimation
Motion Smoothing
Local Motion Estimation
Motion Inpainting Image Deblurring
Completion
14Output Video
Local Motion EstimationLocal Motion Estimation
A id l i f L K d• A pyramidal version of Lucas-Kanade optical flow computation is applied to bt i th l l ti fi ldobtain the local motion field.
15J. Bouguet, “Pyramidal Implementation of the Lucas Kanade Feature Tracker: Description of the Algorithm,” OpenCV Document, Intel, Microprocessor Research Labs, 2000.
Input Video
Global Motion Estimation
Motion Smoothing
Local Motion Estimation
Motion Inpainting Image Deblurring
Completion
16Output Video
Motion InpaintingMotion Inpainting
M i i ith i t t i t• Mosaicing with consistency constraint.'
' ' ( )( ( ))( )
tt tt t t t if v p Tmidian I T p
I p<⎧
= ⎨
where
( )t tI potherwisekeep it as missing
⎨⎩
2' '' '
'
1( ) ( ( ) ( ))11
t
t tt t t t t t t t
t Mv p I T p I T p
n ∈
= −− ∑
' '' '
'
1( ) ( )t
t tt t t t t t
t MI T p I T p
n ∈
= ∑
17
Motion InpaintingMotion Inpainting
18A. Telea, “An Image Inpainting Technique Based on the Fast Marching Method,” J. Graphics Tools, vol. 9, no. 1, pp. 23-34, 2004.
Motion InpaintingMotion Inpainting
Th ti l f i l i• The motion value for pixel pt is generated by a weighted average of th ti t f th i l H( )the motion vectors of the pixels H(pt)
( )( , ) ( | )
( ) t t
t t t tq H p
w p q F p qF ∈
∑
where
( )
( )
( )( , )
t t
t t
q H pt
t tq H p
F pw p q
∈
∈
=∑
( ) ( )
( ) ( )( | ) ( ) ( )( ) ( )
x t x t
y t y t
F q F qx y
t t t t t t t F q F qx y
xF p q F q F q p q F q
y
∂ ∂∂ ∂
∂ ∂∂ ∂
⎡ ⎤ Δ⎡ ⎤⎢ ⎥= + ∇ − = + ⎢ ⎥Δ⎢ ⎥ ⎣ ⎦⎣ ⎦
19' ' ' '
1 1( , )|| || || ( ) ( ) ||t t
t t t t t t t t
w p qp q I q p q I q ε
=− + − − +
Input Video
Global Motion Estimation
Motion Smoothing
Local Motion Estimation
Motion Inpainting Image Deblurring
Completion
20Output Video
Experimental ResultsExperimental Results
30 id li ( b t 80 i t )• 30 video clips (about 80 minutes) with different types of scenes
• k = 6 for motion smoothing• 5x5 filter for motion inpaintingp g
22
Quantitative EvaluationQuantitative Evaluation
D i ti f th G d T th• Deviation from the Ground Truth.• MAD of intensity
28
Quantitative EvaluationQuantitative Evaluation
E l ti f S ti T l S th• Evaluation of Spatio-Temporal Smoothness.– The normalized discontinuity measure D is
defined asdefined as1 1|| ||
n n
i i ii i
I
D I I In n
∂
= ∇ = ∇ ⋅∇
⎡ ⎤ ⎡ ⎤
∑ ∑( 1, , ) ( 1, , )( , 1, ) ( , 1, )( 1) ( 1)
IxIy
I
I x y t I x y tI I x y t I x y t
I x y t I x y t
∂∂∂∂
∂
⎡ ⎤ + − −⎡ ⎤⎢ ⎥ ⎢ ⎥∇ = ≈ + − −⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥+⎣ ⎦⎣ ⎦
– The relative smoothness is evaluated by(D D )/(D D )
( , , 1) ( , , 1)It
I x y t I x y t∂∂⎢ ⎥ ⎢ ⎥+ − −⎣ ⎦⎣ ⎦
29
(DM-DO)/(DM-DA)– 5.9%~23.5% smoother than mosaicing
Computation CostComputation Cost
2 2 f / f 720 486 id ith• 2.2 frames/s for 720x486 video withP4 2.8GHz CPU
30
ConclusionConclusion
Motion inpainting instead of cropping• Motion inpainting instead of cropping.• Deblurring without estimating PSFs.• Spatial smoothness is indirectly
guaranteed by the smoothness of the extrapolated motionextrapolated motion.
• Temporal consistency on both static and dynamic areas is given by opticaland dynamic areas is given by optical flow from the neighboring frames.
31