View
224
Download
0
Tags:
Embed Size (px)
Citation preview
Background vs foreground segmentation of video sequences
= +
The Problem
bull Separate video into two layersndash stationary backgroundndash moving foreground
bull Sequence is very noisy reference image (background) is not given
Simple approach (1)
temporal mean
background
temporal median
Simple approach (2)
threshold
Simple approach noise can spoil everything
Variational approach
Find the background and foregroundsimultaneously by minimizing energy functional
Bonus remove noise
Notations
[0t
max ]
N(xt) original noisy sequence
B(x) background image
C(xt) background mask(1 on background 0 on foreground)
give
n
need
to fi
nd
Energy functional data term
B N
B - N C
Energy functional data term
Degeneracy can be trivially minimized bybull C 0 (everything is foreground)
bull B N (take original image as background)
Energy functional data term
C 1
Energy functional data term
there should be enough of background
original images should be close to the restored background image
in the background areas
Energy functional smoothness
For background image B
For background mask C
Energy functional
Edge-preserving smoothnessRegularization term
Quadratic regularization [Tikhonov Arsenin 1977]
ELE
Known to produce very strong isotropic smoothing
Edge-preserving smoothnessRegularization term
Change regularization
ELE
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
The Problem
bull Separate video into two layersndash stationary backgroundndash moving foreground
bull Sequence is very noisy reference image (background) is not given
Simple approach (1)
temporal mean
background
temporal median
Simple approach (2)
threshold
Simple approach noise can spoil everything
Variational approach
Find the background and foregroundsimultaneously by minimizing energy functional
Bonus remove noise
Notations
[0t
max ]
N(xt) original noisy sequence
B(x) background image
C(xt) background mask(1 on background 0 on foreground)
give
n
need
to fi
nd
Energy functional data term
B N
B - N C
Energy functional data term
Degeneracy can be trivially minimized bybull C 0 (everything is foreground)
bull B N (take original image as background)
Energy functional data term
C 1
Energy functional data term
there should be enough of background
original images should be close to the restored background image
in the background areas
Energy functional smoothness
For background image B
For background mask C
Energy functional
Edge-preserving smoothnessRegularization term
Quadratic regularization [Tikhonov Arsenin 1977]
ELE
Known to produce very strong isotropic smoothing
Edge-preserving smoothnessRegularization term
Change regularization
ELE
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Simple approach (1)
temporal mean
background
temporal median
Simple approach (2)
threshold
Simple approach noise can spoil everything
Variational approach
Find the background and foregroundsimultaneously by minimizing energy functional
Bonus remove noise
Notations
[0t
max ]
N(xt) original noisy sequence
B(x) background image
C(xt) background mask(1 on background 0 on foreground)
give
n
need
to fi
nd
Energy functional data term
B N
B - N C
Energy functional data term
Degeneracy can be trivially minimized bybull C 0 (everything is foreground)
bull B N (take original image as background)
Energy functional data term
C 1
Energy functional data term
there should be enough of background
original images should be close to the restored background image
in the background areas
Energy functional smoothness
For background image B
For background mask C
Energy functional
Edge-preserving smoothnessRegularization term
Quadratic regularization [Tikhonov Arsenin 1977]
ELE
Known to produce very strong isotropic smoothing
Edge-preserving smoothnessRegularization term
Change regularization
ELE
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Simple approach (2)
threshold
Simple approach noise can spoil everything
Variational approach
Find the background and foregroundsimultaneously by minimizing energy functional
Bonus remove noise
Notations
[0t
max ]
N(xt) original noisy sequence
B(x) background image
C(xt) background mask(1 on background 0 on foreground)
give
n
need
to fi
nd
Energy functional data term
B N
B - N C
Energy functional data term
Degeneracy can be trivially minimized bybull C 0 (everything is foreground)
bull B N (take original image as background)
Energy functional data term
C 1
Energy functional data term
there should be enough of background
original images should be close to the restored background image
in the background areas
Energy functional smoothness
For background image B
For background mask C
Energy functional
Edge-preserving smoothnessRegularization term
Quadratic regularization [Tikhonov Arsenin 1977]
ELE
Known to produce very strong isotropic smoothing
Edge-preserving smoothnessRegularization term
Change regularization
ELE
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Simple approach noise can spoil everything
Variational approach
Find the background and foregroundsimultaneously by minimizing energy functional
Bonus remove noise
Notations
[0t
max ]
N(xt) original noisy sequence
B(x) background image
C(xt) background mask(1 on background 0 on foreground)
give
n
need
to fi
nd
Energy functional data term
B N
B - N C
Energy functional data term
Degeneracy can be trivially minimized bybull C 0 (everything is foreground)
bull B N (take original image as background)
Energy functional data term
C 1
Energy functional data term
there should be enough of background
original images should be close to the restored background image
in the background areas
Energy functional smoothness
For background image B
For background mask C
Energy functional
Edge-preserving smoothnessRegularization term
Quadratic regularization [Tikhonov Arsenin 1977]
ELE
Known to produce very strong isotropic smoothing
Edge-preserving smoothnessRegularization term
Change regularization
ELE
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Variational approach
Find the background and foregroundsimultaneously by minimizing energy functional
Bonus remove noise
Notations
[0t
max ]
N(xt) original noisy sequence
B(x) background image
C(xt) background mask(1 on background 0 on foreground)
give
n
need
to fi
nd
Energy functional data term
B N
B - N C
Energy functional data term
Degeneracy can be trivially minimized bybull C 0 (everything is foreground)
bull B N (take original image as background)
Energy functional data term
C 1
Energy functional data term
there should be enough of background
original images should be close to the restored background image
in the background areas
Energy functional smoothness
For background image B
For background mask C
Energy functional
Edge-preserving smoothnessRegularization term
Quadratic regularization [Tikhonov Arsenin 1977]
ELE
Known to produce very strong isotropic smoothing
Edge-preserving smoothnessRegularization term
Change regularization
ELE
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Notations
[0t
max ]
N(xt) original noisy sequence
B(x) background image
C(xt) background mask(1 on background 0 on foreground)
give
n
need
to fi
nd
Energy functional data term
B N
B - N C
Energy functional data term
Degeneracy can be trivially minimized bybull C 0 (everything is foreground)
bull B N (take original image as background)
Energy functional data term
C 1
Energy functional data term
there should be enough of background
original images should be close to the restored background image
in the background areas
Energy functional smoothness
For background image B
For background mask C
Energy functional
Edge-preserving smoothnessRegularization term
Quadratic regularization [Tikhonov Arsenin 1977]
ELE
Known to produce very strong isotropic smoothing
Edge-preserving smoothnessRegularization term
Change regularization
ELE
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Energy functional data term
B N
B - N C
Energy functional data term
Degeneracy can be trivially minimized bybull C 0 (everything is foreground)
bull B N (take original image as background)
Energy functional data term
C 1
Energy functional data term
there should be enough of background
original images should be close to the restored background image
in the background areas
Energy functional smoothness
For background image B
For background mask C
Energy functional
Edge-preserving smoothnessRegularization term
Quadratic regularization [Tikhonov Arsenin 1977]
ELE
Known to produce very strong isotropic smoothing
Edge-preserving smoothnessRegularization term
Change regularization
ELE
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Energy functional data term
Degeneracy can be trivially minimized bybull C 0 (everything is foreground)
bull B N (take original image as background)
Energy functional data term
C 1
Energy functional data term
there should be enough of background
original images should be close to the restored background image
in the background areas
Energy functional smoothness
For background image B
For background mask C
Energy functional
Edge-preserving smoothnessRegularization term
Quadratic regularization [Tikhonov Arsenin 1977]
ELE
Known to produce very strong isotropic smoothing
Edge-preserving smoothnessRegularization term
Change regularization
ELE
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Energy functional data term
C 1
Energy functional data term
there should be enough of background
original images should be close to the restored background image
in the background areas
Energy functional smoothness
For background image B
For background mask C
Energy functional
Edge-preserving smoothnessRegularization term
Quadratic regularization [Tikhonov Arsenin 1977]
ELE
Known to produce very strong isotropic smoothing
Edge-preserving smoothnessRegularization term
Change regularization
ELE
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Energy functional data term
there should be enough of background
original images should be close to the restored background image
in the background areas
Energy functional smoothness
For background image B
For background mask C
Energy functional
Edge-preserving smoothnessRegularization term
Quadratic regularization [Tikhonov Arsenin 1977]
ELE
Known to produce very strong isotropic smoothing
Edge-preserving smoothnessRegularization term
Change regularization
ELE
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Energy functional smoothness
For background image B
For background mask C
Energy functional
Edge-preserving smoothnessRegularization term
Quadratic regularization [Tikhonov Arsenin 1977]
ELE
Known to produce very strong isotropic smoothing
Edge-preserving smoothnessRegularization term
Change regularization
ELE
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Energy functional
Edge-preserving smoothnessRegularization term
Quadratic regularization [Tikhonov Arsenin 1977]
ELE
Known to produce very strong isotropic smoothing
Edge-preserving smoothnessRegularization term
Change regularization
ELE
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Edge-preserving smoothnessRegularization term
Quadratic regularization [Tikhonov Arsenin 1977]
ELE
Known to produce very strong isotropic smoothing
Edge-preserving smoothnessRegularization term
Change regularization
ELE
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Edge-preserving smoothnessRegularization term
Change regularization
ELE
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Edge-preserving smoothnessRegularization term
ELE
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Edge-preserving smoothnessRegularization term
ELE
n
Change the coordinate system
across the edgealong the edge
Compare
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Edge-preserving smoothnessRegularization term
Weak edge (s 0)
Conditions on
Isotropic smoothings) is quadratic at zero
(s)
s
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Edge-preserving smoothnessRegularization term
Strong edge (s )
Conditions on
bull no smoothing across the edge
bull more smoothing along the edge
Anisotropic smoothings) does not grow too fast at infinity
(s)
s
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Edge-preserving smoothnessRegularization term
ConclusionUsing regularization term of the form
we can achieve both
isotropic smoothness in uniform regions
and anisotropic smoothness on edges
with one function
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
0 1 2 3 4 50
05
1
15
2
25
3
35
4
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
0 005 01 015 020
0005
001
0015
002
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Edge-preserving smoothnessSpace of Bounded Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only smooth functions we may not achieve the desired minimum
-1 -05 0 05 1-1
-05
0
05
1
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Edge-preserving smoothnessSpace of Bounded Variations
which one is ldquobetterrdquo
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Bounded Variation ndash ND caseBounded Variation ndash ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || )( ) ( )(
101
L
NN C
bounded open subset function NR )( 1 Lf
Variation of over f
where
φ
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Edge-preserving smoothnessSpace of Bounded Variations
integrable (absolute value) and with bounded variation
Functions are not required to have an integrable derivative hellip
What is the meaning of u in the regularization term
Intuitively norm of gradient |u| is
replaced with variation |Du|
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Total variation
Theorem (informally) if u BV() then
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Hausdorff measure
area gt 0area = 0
How can we measure zero-measure sets
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Hausdorff measure
1) cover with balls of diameter
2) sum up diameters for optimal cover (do not waste balls)
3) refine 0
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Hausdorff measureFormally
For A RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable
bull HN is just the Lebesgue measure
bull curve in image
its length = H1 in R2
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Total variation
Theorem (more formally) if u BV() then
u+
u-
u(x)
xx0
u+ u- - approximate upper and lower limits
Su = x u+gtu-the jump set
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Energy functional
data term
regularization for background image
regularization for background masks
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Total variation example
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
-1 -05 0 05 1 15 2
-1
0
1
2
0
05
1
= perimeter = 4
Divide each side into n parts
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation(= sum of perimeters)
Large total variation(= sum of perimeters)
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Edge-preserving smoothnessSpace of Bounded Variations
Small total variation Large total variation
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Edge-preserving smoothnessSpace of Bounded Variations
BV informally functions with discontinuities on curves
Edges are preserved texture is not preserved
original sequencetemporal median energy minimization
in BV
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Energy functional
Time-discretized problem
Find minimum of E subject to
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Existence of solution
Under usual assumptions
12 R+ R+ strictly convex nondecreasing
with linear growth at infinity
minimum of E exists in BV(BC1hellipCT)
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
(non-)Uniqueness
is not convex wrt (BC1hellipCT) Solution may not be unique
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Uniqueness
But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique
Interpretation if we are allowed to say that everything is foreground background image is not well-defined
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Finding solution
BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV
Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Approximating functionals
Recall 12(s) = s2 gives smooth solutions
Idea replace i with iwhich are quadratic
at s 0 and s
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Approximating functionals
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
Approximating problems
has unique solution in the space
ndash convergence of functionals if E -converge to Ethen approximate solutions of min E
converge to min E
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
More results Sweden
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
More results Highway
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
More results INRIA_1
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
More results INRIA_1Sequence restoration
More results INRIA_2Sequence restoration
More results INRIA_2Sequence restoration