View
0
Download
0
Category
Preview:
Citation preview
From local to global: Edge profiles to camera motion in blurred images
Subeesh Vasu1, A. N. Rajagopalan2
Indian Institute of Technology Madras
subeeshvasu@gmail.com1, raju@ee.iitm.ac.in
2
Abstract
In this work, we investigate the relation between the edge
profiles present in a motion blurred image and the underly-
ing camera motion responsible for causing the motion blur.
While related works on camera motion estimation (CME)
rely on the strong assumption of space-invariant blur, we
handle the challenging case of general camera motion. We
first show how edge profiles ‘alone’ can be harnessed to
perform direct CME from a single observation. While it is
routine for conventional methods to jointly estimate the la-
tent image too through alternating minimization, our above
scheme is best-suited when such a pursuit is either imprac-
tical or inefficacious. For applications that actually favor
an alternating minimization strategy, the edge profiles can
serve as a valuable cue. We incorporate a suitably de-
rived constraint from edge profiles into an existing blind de-
blurring framework and demonstrate improved restoration
performance. Experiments reveal that this approach yields
state-of-the-art results for the blind deblurring problem.
1. Introduction
Camera motion estimation (CME) plays a vital role in
many multi-image based applications such as structure from
motion, super-resolution, HDR imaging etc. These applica-
tions involve sharing of information across images, which is
typically achieved by establishing feature correspondences.
But this becomes a challenging task in the presence of mo-
tion blur. CME is also important in many single motion
blurred image based applications such as image restoration,
scene depth estimation ([18, 36]), splicing detection ([12]),
estimation of 3D structure of light sources ([38]) etc.
A common approach for CME from a motion blurred im-
age is to pose it as a blind deblurring problem, wherein both
latent image (or its features) and camera motion are esti-
mated within an alternating minimization (AM) framework.
In the blind deblurring literature, most approaches assume
space invariant (SI) blur (i.e. pure in-plane camera transla-
tions) ([3, 9, 27, 10, 11, 1, 35, 15, 17, 2, 4, 29, 20, 21]).
While this leads to very efficient blind deblurring algo-
rithms, this assumption does not hold true in practice. It is
well-known ([16, 13]) that natural camera shake often con-
tains rotational components resulting in spatially varying
(SV) blur in the captured images. As is evident from recent
works ([5, 31, 7, 34, 37, 22]), one needs to account for the
SV nature of blur for accurate estimation of general camera
motion. While this is an important problem, the increase in
number of unknowns makes it quite ill-posed, demanding
stronger priors for achieving quality performance. Very ill-
posed problems such as depth-aware deblurring and rolling
shutter motion deblurring, either avoid using AM altogether
([24]) or need a good initialization of the camera motion
([36, 28]) derived from local point spread functions (PSFs)
i.e., blur kernel estimates returned by an off-the-shelf blind
deblurring algorithm. However, their performance is lim-
ited because PSF estimates from small regions can be erro-
neous.
A recent trend in blind deblurring is to come up with
class specific or generic priors ([35, 2, 29, 37, 20, 21, 22])
to reduce the ill-posedness of the problem. While some of
these priors have shown great promise, their performance is
limited due to one or more of the following reasons: fail-
ure on images for which prior does not hold good, inability
to handle SV blur, computational complexity etc. In SV de-
blurring, [37] has been widely popular for a long time. They
employed an L0 prior on the gradients of the image while
performing CME. The state-of-the-art work in [22] employs
dark channel prior in addition to the L0 prior of [37] to sig-
nificantly improve the performance of SV deblurring.
In this work, we address the problem of CME from a
single motion blurred image by harnessing edge profiles.
There are only two notable works that have used edge pro-
files to derive useful information. Cho et al. [2] have mod-
eled the relation between PSF and edge profiles to perform
SI deblurring. Tai et al. [30] have used edge profiles for esti-
mation of camera response function (CRF) from SI blurred
images. We first propose an approach (called EpAlone)
for efficient and direct CME from edge profiles i.e., sans
the need for latent image estimation. We expand the scope
of edge profiles and employ them for estimation of general
camera motion. While problems such as blind deblurring,
4447
HDR imaging etc. perform joint estimation of the latent
image and camera motion, for applications where such a
scheme is impractical or inefficacious, EpAlone is apt. For
applications that actually rely on such a joint estimation
scheme, a wise choice is to use edge profiles as an addi-
tional regularizer. Recent works on blind deblurring have
([21, 22]) revealed that the performance can be improved
by adding strong priors onto existing methods. Following
these works, we propose a method (called Xu + Ep) that in-
corporates an efficient and effective constraint derived from
edge profiles into the existing method of Xu et al. [37]
to achieve state-of-the-art performance. Both the proposed
methods (EpAlone and Xu + Ep) are based on new theoret-
ical claims that we introduce in this paper. The correspond-
ing proofs are provided in the supplementary material due
to space constraints.
Our main contributions are summarized below.
• For the first time in literature, we derive and analyze
the relation between edge profiles present in a blurred image
and the underlying 6D camera motion.
• We propose a new constraint derived from edge profiles
and use it to perform efficient and direct CME from a single
motion blurred image.
• We propose a blind deblurring scheme by elegantly in-
corporating our new constraint into the formulation of [37]
to achieve state-of-the-art performance.
2. Edge profile and convolution
Edge profile is the alpha matte taken over a line along
the edge orientation (by treating the two colors of homoge-
neous regions on either sides of the edge as background and
foreground). In our work, the boundary between two homo-
geneous regions in an image is considered as an edge. Let
us denote the edge profile of length N across an edge as αi
where i = 1, 2, .., N . Then the intensity values along a line
L lying along the edge orientation can be expressed as
L(i) = c1αi + c2(1− αi) (1)
with c1 and c2 being the colors of the two homogeneous re-
gions on either side of the edge. Without loss of generality,
assume that the edge that we are working with is a vertical
edge. Consider a binary step edge S defined over a rectan-
gular region around the considered edge as
S(x, y) =
{
1 x ≥ 00 x < 0.
(2)
For a clean image, the intensity I at this region can be ex-
pressed as,
I = Sc1 + (1− S)c2 (3)
Blurring the above region with a kernel k results in
BI = k ∗ I = (k ∗ S)c1 + (1− k ∗ S)c2 (4)
= Skc1 + (1− Sk)c2 (5)
where Sk embeds the variation in intensity while moving
across the edge in the motion blurred image BI . By com-
paring Eq. (1), Eq. (2), and Eq. (5) we can observe that, the
edge profile α in the blurred image is equal to the middle
row of Sk. It is trivial to see that the rows of Sk and the cu-
mulative of the projection of k onto X axis are equivalent,
i.e.,
Sk(x, y) =
x∑
u=−∞
∞∑
v=−∞
k(u, v) =
x∑
u=−∞
P (u) (6)
where P is the projection of k onto the X axis. Note that the
final expression in Eq. (6) is not a function of y indicating
that all rows of Sk will be identical.
Similarly one can show that, in general, an edge profile is
equivalent to the cumulative distribution function (CDF) of
the projection of PSF onto the line along the edge orienta-
tion [30] (or CDF of the Radon transform of the PSF along
the direction orthogonal to the edge orientation [2]). If the
local region I in the image is not blurred (i.e., BI = I), then
Sk = S i.e., the edge profile in that region will be a step
response. Note that these properties of step edges are de-
fined over local regions. Since we can treat the blur at local
regions as SI even for general camera motion, we can use
the above results in SV blurred images too. This opens up
the possibility of exploiting edge profiles for general CME.
More specifically, one could solve for the camera motion
by looking for the solution which favors ideal step edges in
corresponding latent image estimates.
An illustrative example supporting the results discussed
in this section is displayed in Fig. 1. We have used a sample
PSF (Fig. 1(a)) from the dataset of [16] to blur an example
image, a region extracted from which is shown in Fig. 1(b).
The projection of PSF onto two different directions along
with the CDF of this projection is displayed in Fig. 1(a).
As is evident from Fig. 1(b), the projection of PSF is equal
to the differential of the edge profile extracted along the re-
spective directions.
3. Edge profile to camera motion
In this section, we derive the relation between the local
edge profiles present in a motion blurred image and the un-
derlying global camera motion. For general camera motion,
the blurred image (g) can be expressed as an aggregate of
warped instances of the latent image f as ([5, 34, 23])
g(x) =1
te
te∫
0
f((Ht)−1(x))dt (7)
where x = (x, y) represents the spatial coordinates, te is
the total exposure time of the image, Ht corresponds to the
homography relating the latent image to the projection of
4448
(a) (b)Figure 1. Equivalence between the CDF of projection of PSF and
its corresponding edge profile formed in the blurred image. (a) A
sample PSF and its projection along two directions and the CDF
of the projections. (b) An example region from a corresponding
blurred image along with the edge profile extracted from it (along
orthogonal directions) as well as the differential of the edge profile.
the scene onto the image plane at the time instant t. An
equivalent way to relate g and f is to use the warp which is
defined based on the poses that the camera undergoes during
motion.
g(x) =
∫
γ∈T
Ψ(γ)f((Hγ)−1(x))dγ (8)
where T is the set of different poses that the camera un-
dergoes during the exposure time, γ refers to a single pose
from T , Hγ is the homography warp corresponding to γ and
Ψ(γ) is the fraction of time over which the camera stays at
γ. Eq. (8) allow to express the same relation as Eq. (7)
but in a time-independent fashion. Ψ is referred to as mo-
tion density function (MDF) ([5]) or transformation spread
function (TSF) ([23]). For general camera motion, the PSF
at x in the blurred image can be related to the MDF ([23])
as
k(x, u) =
∫
γ∈T
Ψ(γ)δ(u − (Hγx − x))dγ (9)
where Hγx refers to Hγ(x), the coordinates obtained by
warping x with the homography Hγ . Here PSF k(x, u) is
defined as a function of two variables, since it is space vari-
ant (SV) in nature, where u = (u, v) represents the spatial
coordinates over which the PSF at x is defined. Eq. 9 is a
direct implication of the fact that a pose vector γ0 assign its
weight Ψ(γ0) to a point u0 in the PSF, and the PSF can be
obtained by integrating such assignments from all poses in
MDF. The pose γ0 can be related to u0 as
u0 = Hγ0x − x (10)
Let us denote k(x, u) as kx(u, v). The differential of edge
profile dEθ,x (along θ + π/2) at x can be expressed as the
Radon transform of the PSF at x along θ ([2]).
dEθ,x(ρ) =
∫ ∫
kx(u, v)δ(ρ− u cos θ − v sin θ)dudv
(11)
where ρ represents the radial coordinates of the Radon pro-
jection. Combining the relations in Eq. (9) and Eq. (11) we
are able to directly relate MDF to the edge profile as1
dEθ,x(ρ) =
∫
γ∈Γ
Ψ(γ)δ(ρ− (Hγxx − x) cos θ−
(Hγy x − y) sin θ)dγ (12)
Here Hγxx refers to the x coordinate of Hγ(x). Eq. (12)
relates the differential of edge profile (dE) directly with the
MDF or the underlying camera motion. By discretizing the
relation in Eq. (12) we obtain
dEθ,x(ρ) =
NT∑
p=1
w(p)δ(ρ− (Hpxx − x) cos θ−
(Hpyx − y) sin θ) (13)
where w is the MDF weight vector defined over a discrete
pose space P with the value being nonzero (and positive)
only for those poses p ∈ P over which the camera has
moved, and NT is the total number of poses in P . This
discretization allows us to relate the edge profile with the
MDF vector w in matrix-vector multiplication form as
dEθ,x = Mθ,xw (14)
where Mθ,x is the measurement matrix formed according to
the relation in Eq. (13), dEθ,x ∈ Nρ × 1,Mθ,x ∈ Nρ ×NT
and w ∈ NT × 1 with Nρ being the length of the edge
profile. Using such Ne edge profiles, we can write
dE = Mw (15)
where dE ∈ NρNe × 1,M ∈ NρNe ×NT . Thus a collec-
tion of edge profiles from a motion blurred image can be re-
lated to the MDF elegantly through Eq. (15). We can solve
for MDF w by minimizing the following cost function.
w = argminw
||dE −Mw||2 + λ||w||1 (16)
where L1 norm is used to enforce the natural sparsity of
camera motion trajectory in the pose space, and λ is the
prior weight on MDF. Note that this approach is similar to
the idea of recovering MDF from local PSF estimates as
suggested in [24]. Since the local PSF estimates obtained
through conventional blind deblurring schemes can be erro-
neous, the applicability of CME from PSF is limited. On
the other hand, our formulation in Eq. (16) directly relate
camera motion to the naturally available edge profiles in the
image, improving the scope of direct CME. In other words,
while the performance of CME from PSFs depends on the
performance of existing methods on PSF estimation, our
method does not suffer from such a limitation.
1Details of the derivation can be found in supplementary.
4449
4. Recovering camera motion from edges
If sufficient number of edge profiles is available, we can
use Eq. (16) to estimate camera motion accurately. How-
ever, there are some issues in attempting to use Eq. (16) in
practical situations. To extract edge profiles from different
locations in an image, we follow the procedure in [2]. They
have used a set of image analysis heuristics including de-
tection of edges in blurry images and a number of outlier
rejection schemes. However, a fundamental problem is that
the extracted edge profiles will not be aligned with respect
to each other, and hence cannot be directly used for CME in
Eq. (16). To align the extracted edge profiles, [2] exploits
the fact that the center of mass of the PSF corresponds to the
center of mass of their 1D projections. Since this was not
valid under the influence of CRF, [30] employed extreme
points to align the edge profiles along the same direction.
But this is not valid for edge profiles from different orienta-
tions, since the homographies which map to extreme points
can be different for projections along different directions.2
Hence, for our scenario we cannot use extreme points for
alignment.
4.1. Alignment and implications
In this section, we show that edge profiles can be aligned
using the center of mass, even for general camera motion.
We restrict our analysis to two different but commonly used
approximations for 6D camera trajectories. First, we an-
alyze the case of model1 which models 6D motion using
in-plane translations and in-plane rotations ([5]). Later we
extend the result to model2 which employs pure 3D rota-
tions ([34]) as an approximation to 6D motion.
Claim 1: For model1, if the in-plane rotational motion
undergone by the camera is small (as is typical for camera
shakes in handheld situations [13, 28]) then it can be shown
that the centroid of MDF vector will correspond to the cen-
troid of PSF i.e., the centroid pose of the camera obtained
from MDF will map points to the centroid of PSFs gener-
ated by the camera motion or equivalently to the centroid of
the corresponding edge profiles.
For small in-plane rotations one can show that the cen-
troid co-ordinates (uc, vc) of PSF at x in the image corre-
sponds to the centroid (tcx, tcy, θ
cz) of MDF.3 i.e.,
uc =∑
u
∑
v
ukx(u, v) = Hcxx − x (17)
vc =∑
u
∑
v
vkx(u, v) = Hcyx − y (18)
where Hc is the homography corresponding to the centroid
of MDF. The correspondence between the centroid of a PSF
2An illustrative example is provided in supplementary.3Detailed proof is provided in supplementary.
and its corresponding edge profiles is trivial to see from the
fact that the edge profile is a linear projection of PSF. Since
the correspondence between the centroids of edge profile
and PSF is always valid [2], the alignment of PSFs auto-
matically refers to alignment of edge profiles.
Furthermore, since in-plane translations are equivalent
to out-of-plane rotations (for large focal length) [34], we
can conclude that the centroid based alignment of edge pro-
files is sufficient for model2 also. This is also supported
by the work in [13] which shows that model1 and model2are equally good in approximating the original 6D camera
trajectories caused by natural handshakes.
Next, we analyze the effect of centroid-based alignment
on the estimation of camera motion and latent image. It
is well-known that the same blurred image can be pro-
duced by different valid (i.e., correct upto a homography)
latent image-camera motion pairs. The aligned edge pro-
files should correspond to a valid camera motion to ensure
that the estimate returned by Eq. (16) is correct. Next, we
will show that a collection of centroid aligned PSFs is con-
sistent with a valid camera motion and latent image that can
produce the same blurred image (implying that the aligned
edge profiles can be directly used for CME). Let us consider
a collection of m PSFs, k1, ..km obtained from locations
x1, ..xm in the blurred image.
Claim 2: The centroid aligned PSFs correspond to a
camera motion and a latent image both of which are con-
sistent with the given blurred image and correct upto a ho-
mography.
Let there be n active (non-zero) poses in the MDF w0,
with H1, .., Hn being the corresponding homographies,
and L be the latent image. We can show that4 the aligned
PSFs corresponds to another latent image (Lc) defined on
coordinates Hcx (i.e. related to the L through the homog-
raphy warp Hc) and a new MDF (formed of a collection
of homographies Hj(Hc)−1) both of which are consistent
with the same blurred image. Thus, using the centroid
aligned edge profiles we can estimate one of the valid cam-
era motion wc0 using Eq. (16).
To verify the results of claims 1 and 2, and also to an-
alyze the impact of the error introduced in the CME by
centroid alignment of edge profiles, we performed the fol-
lowing experiment. Using camera trajectories (ground truth
camera motion) from the database of [13], we produced
edge profiles at different locations in the image. These edge
profiles are aligned with respect to the centroid and then
used for estimation of corresponding MDF using Eq. (16).
This MDF is then used to generate PSFs at different loca-
tions. We generate another set of PSFs using the centroid
aligned camera trajectories obtained by mapping the ground
truth camera trajectory using (Hc)−1. From claims 1 and
2, these two PSFs should be identical. We computed the
4For detailed discussions on claim 2, refer to supplementary material.
4450
normalized cross correlation (NCC) values between these
two sets of PSFs to verify our observations. This also mea-
sures the accuracy (validity) of centroid alignment of PSFs
for different kinds of camera motions. In addition to this,
we computed the difference (say dc) between the centroid
of the edge profiles and the point to which the centroid of
camera trajectories maps. We carry out experiments in two
different ways. On the one hand, we tested the applicabil-
ity of our proposed method on model1 and model2 using
the camera trajectories from database of [13] (Figs. 2(b,
c, e, f)). On the other hand, we separately investigated the
variation of the accuracy with respect to in-plane rotations
alone using a single camera trajectory. The amount of rota-
tion spanned by the trajectory is increased to observe the
behavior of both error in centroid as well as NCC value
(displayed in Figs. 2(a, d)). We found that dc is always sub-
pixel (Figs. 2(a,b,c)) and the correlation values are consis-
tently high (Figs. 2(d,e,f)), which supports our theoretical
findings. Although the error increases with increase in the
degree of rotation, it is still quite small even for in-plane
rotations upto 7 degrees, which is well within the range in-
troduced by incidental camera shakes.
(a) (b)
(c) (d)
(e) (f)Figure 2. Analyzing the impact of centroid alignment on CME.
These measures are estimated from PSFs generated all over an
example image. Absolute value of maximum and average er-
ror between estimated centroid and ground truth centroid for the
camera motion using (a) in-plane rotation only, (b) model1, and
(c) model2. Minimum and average value of NCC between non-
blindly estimated PSFs and GT PSFs for (d) in-plane rotation only,
(e) model1, and (f) model2.
4.2. Computational considerations
On the basis of claims 1 and 2, we can align all the edge
profiles extracted from a given motion blurred image and
perform CME. However, direct use of Eq. (16) is computa-
tionally expensive. This is because both the construction of
the measurement matrix M and operations involving such a
large matrix are time-consuming. Therefore, we propose to
re-frame our constraint in Eq. (16) so as to exploit the ad-
vantages of the computationally efficient filter-flow frame-
work ([8]).
From claims 1 and 2, we observe that the centroid loca-
tions of edge profiles on the blurred image can be treated
as the location of the step edges in the corresponding latent
image. From this principle, we can generate a prediction
image (Ip) formed from all the step edges in the latent im-
age. We build an edge profile image (Iep) formed from the
differential of edge profiles at the respective locations. Us-
ing Eq. (6), Eq. (15) and claims 1 and 2, we note that
the step edges in Ip are related to the corresponding edge
profiles in Iep through a valid camera motion. Hence, we
propose to use the following estimation equation, formed
from Ip and Iep, to replace the computationally expensive
formulation of Eq. (16).
w = argminw
||Tw(Ip)− ˜Iep||2 + λ||w||1 (19)
Here Tw represents the set of warping operations defined by
w. This non-uniform blur relation between Ip and Iep can
be implemented efficiently using the filter-flow framework
[8]. We use our proposed constraint in Eq. (19) to perform
blind CME from edge profiles and name this as EpAlone.
The computational complexity of EpAlone can be reduced
further, as discussed next in the generation of Ip and Iep.
Claim 3: The absolute value of differential of edge pro-
file (P in Eq. (6)) is equivalent to the absolute gradient of
a blurred image (at the location of the edge profile) normal-
ized by the difference between the two homogeneous colors
involved. i.e.,
|▽BI | = |(c1 − c2)|P (20)
From claim 3,5 we form Ip by assigning the absolute
value of (c1 − c2) at the centroid positions. Iep is formed
by collecting the slices (around the centroid of edge profile)
of absolute gradients along the direction of edge profiles. In
our experiments, we observed that taking slices only along
the direction of edge profile often leads to small errors in
the estimate of camera motion, mainly due to errors in the
estimate of edge profile orientation. As a solution, we have
chosen slices over a small angular window around the es-
timated edge profile orientation. This improves the robust-
ness of Eq. (19) by making it less sensitive to errors in
orientation estimates (as compared to Eq. (16)).
5For detailed discussions on claim 3, please refer to supplementary.
4451
(a) (b) (c) (d) (e) Xu et al. [37]
(f) EpAlone (g) Xu + Ep (h) kernels (EpAlone) (i) kernels (Xu + Ep)
(a) (e)
(f) (g)
(j) Patches from (a,e,f,g)Figure 3. Synthetic example on non-uniform deblurring: (a) Input motion blurred image, and (b) our prediction image. Edge profile image
generated by (c) single-direction mask, and (d) multi-direction mask. (e-g) Deblurred results. (h-i) Estimated kernels from proposed
methods.
An illustrative example on the generation of Ip and Iepis given in Fig. 3. We have used the centroid locations of
extracted edge profiles from the blurred image (Fig. 3(a)) to
form the prediction image (Fig. 3(b)), with the value at each
position being the absolute value of differences between the
colors of nearby homogeneous regions. The corresponding
edge profile images formed by using single as well as multi-
direction mask is shown in Figs. 3(c-d).
Remarks: Our idea of building the prediction image is sim-
ilar to the use of edge prediction units in existing deblurring
approaches. The difference lies in the fact that, we use the
idea of edge profiles for direct prediction of step edges, as
compared to the iterative prediction schemes (through filter-
ing techniques or image priors) used by other methods. Our
primary objective in proposing EpAlone is to reveal the po-
tential of the new constraint (that we advocate through Eq.
(19)) ‘alone’ to regularize the problem of CME.
5. Deblurring with edge profile constraint
Since the dimensionality of unknowns is large, Eq. (19)
needs adequate number of edge profiles spanning a wide
range of orientations to obtain an accurate estimate of MDF.
But it may not always be possible to fulfill this require-
ment. In this section, we show that Eq. (19) can still serve
as a valuable constraint to regularize existing methods for
CME. This is in spirit with recent works on blind deblur-
ring ([21, 22]) that have proposed new priors as an addition
to the image prior in [37], to achieve better performance.
Following this line of thought, we also propose to use Eq.
(19) as an additional constraint to the existing method in
[37] to uplift its performance. Although our constraint can
be used to improve the performance of any deblurring meth-
ods, we chose to use [37], since their L0 norm prior on latent
image gradients was found to yield the best results among
works in SV deblurring. We incorporate Eq. (19) into the
formulation of [37] and call this new approach as Xu + Ep.
Following are the modified equations used for estimation of
latent image and MDF in Xu + Ep.
f t+1 = argminf
{||M twf − g||2 + λf ||▽f ||0} (21)
wt+1 = argminw
{||M t+1
▽f w − ▽g||2+
λIp ||MIpw − ˜Iep||2 + λw||w||1}(22)
where t is the iteration number, ∗ refers to lexicographically
arranged form of ∗. M▽f and MIp are the warping matrices
built from gradient of latent image (▽f ) and Ip, and Mw is
the measurement matrix formed of w. The first and second
term in Eq. 22 act as data constraints whereas the last term
is the sparsity prior on the camera trajectory. While λIp con-
trols the influence of our proposed constraint on the entire
optimization, λw determines the degree of sparsity in w. In
Eq. 21, we have used an L0 norm prior ([37, 21, 22]) on im-
age gradients weighted by a scalar parameter λf for latent
image estimation. It is straightforward to see that by enforc-
ing our edge-profile constraint, we are indirectly motivating
the step edges present in the latent image to strongly influ-
ence the camera motion estimate. This improvement over
[37] delivers state-of-the-art performance as we shall show
later.
Since our additional constraint respects the filter-flow
framework formulation ([7, 32]), the optimization problems
in Eq. (21) and Eq. (22) can be solved entirely using filter-
flow. In Xu + Ep, we alternately minimize between latent
image and MDF estimation in a scale-space fashion. All
4452
Algorithm 1 CME for a single scale.
Input: Blurred image g at current scale, initial MDF esti-
mate w0 obtained from previous scale.
Output: Refined MDF estimate at current scale.
1: For t = 1 : nmax
2: Estimate f t by solving Eq. (21)
3: Estimate wt by solving Eq. (22)
4: End
5: Return wt as the refined estimate of MDF for current
scale.
major steps (for a single scale) for Xu + Ep is listed in Al-
gorithm 1.6 It should be noted that, the objective of the
AM is to obtain an accurate estimate of camera motion. To
obtain the final restored image, we perform a non-blind de-
blurring using the camera motion estimate returned by AM.
Remarks: Exploiting claims 1, 2, and 3 we have developed
a computationally efficient constraint leading to two elegant
CME methods, EpAlone and Xu + Ep. However, there are
important differences in their applicability. Unlike existing
approaches (including Xu + Ep), EpAlone circumvents the
need for intermediate latent image estimation. Therefore,
applications such as CRF estimation ([30]), normal recov-
ery ([25]), depth-aware deblurring ([24, 36]), rolling shutter
motion deblurring [28] etc. that either do not warrant AM or
need a good initial estimate of camera motion, can benefit
from EpAlone or its underlying formulations. For appli-
cations such as blind deblurring, HDR imaging etc. which
rely on AM and explicit latent image estimation, the edge
profile constraint we derived can be a valuable add-on as
discussed for the deblurring problem.
Figure 4. Quantitative evaluation on first 7 kernels from [13].
6. Experiments
Since visual evaluation of estimated camera motion is
difficult, we compare the performance of our proposed
methods based on the latent image estimates obtained from
the resulting camera motion. This in turn signifies the ac-
6Implementation details of our algorithm and the parameter settings
used in our experiments are provided in the supplementary.
curacy of CME. We use [14] to get a latent image esti-
mate using the MDF obtained from both EpAlone and Xu
+ Ep. We employ the filter-flow implementation of Lasso
algorithm [19] provided by [33, 32], for CME using both
EpAlone and Xu + Ep. Although our main focus was on
CME from non-uniformly blurred images, we do compar-
isons for SI cases too. For SI blur, we limit our qualita-
tive comparisons to two closely related approaches [2, 37],
and the current state-of-the-art work [22]. For SV blur, we
compare with [5, 34, 6, 7, 37, 22], and [26]. Qualitative
comparisons are done mainly on real image databases from
[2, 5, 34, 6], and [22]. For fairness, quantitative compar-
isons are done only for AM-based deblurring methods.
Quantitative evaluation: We have used 28 blurred images
formed from the 4 latent images and first 7 kernels in the
benchmark dataset in [7]. We excluded other kernels since
they represent very large blurs which can lead to failure of
edge profile extraction. As is evident from the quantitative
comparisons in Figure 4, Xu + Ep outperforms competing
methods ([1, 35, 27, 3, 15, 32, 7, 37]) and is comparable or
even better than state-of-the-art ([22]).
Synthetic example: Figure 3 shows a synthetic example
on non-uniform deblurring. This is an ideal example for
edge profile based deblurring, because of abundant avail-
ability of edge profiles spanning all orientations. This en-
ables EpAlone Fig. 3(d) to perform better than the compet-
ing methods. Fig. 3(e) is the result from Xu + Ep, which
gives the best result of all.
Real examples: We give few representative examples on
deblurring here (Fig. 5). More examples are provided in
the supplementary. For completeness and to demonstrate
the potential of EpAlone, we have included the results from
EpAlone for the application of blind deblurring. Note that
in most of the examples, EpAlone by itself deliver good
quality output, although it uses only edge profiles for CME.
Our proposed approach (Xu + Ep) is able to give results
with comparable or better quality to the state-of-the-art for
all examples. A visual comparison also reveals that Xu +
Ep is able to integrate the strength of [37] and EpAlone to
achieve the best performance. For the examples in Fig. 3
and Fig. 5, [37] (Fig. 3(e), Fig. 5(b)) leaves residual blur
in the estimated latent image, whereas EpAlone (Fig. 3(f),
Fig. 5(d)) produces ringing artifacts while trying to enforce
abrupt transitions across step edges. However, Xu + Ep
(Fig. 3(g), Fig. 5(e)) is able to integrate desirable features
from both methods to capture the motion accurately.
Remarks: EpAlone performs CME faster than both Xu
+ Ep and [22], while producing promising results. At the
same time our efficient and improvised form of edge con-
straint enables EpAlone to perform better than its SI coun-
terpart in [2]. The main advantage of Xu + Ep over the
state-of-the-art work ([22]) is the significant reduction in
computational complexity, but with comparable/improved
4453
(a) Motion blurred image (b) Xu et al. [37] (c) Gupta et al. [5]
(d) EpAlone (e) Xu + Ep
(a)
(b)
(c)
(d)
(e)(f) Patches from (a-e)
(g) Motion blurred image (h) Xu et al. [37] (i) Whyte et al. [34] (j) Pan et al. [22]
(k) EpAlone (l) Xu + Ep
(g)
(h)
(i)
(m) Patches from (g-i)
(j)
(k)
(l)
(n) Patches from (j-l)
Figure 5. Real example on SV deblurring using image from the datasets of Gupta et al. [5] and Pan et al. [22].
performance. Comparisons of running times are provided
in the supplementary. The main limitation of our new con-
straint is its inability to handle large blurs unlike [22].
7. Conclusions
In this paper, we investigated the relation between edgeprofiles present in a motion blurred image and the under-lying camera motion. We proposed a method (EpAlone)for direct CME from edge profiles to aid applications wheretypical AM frameworks are inapt. To highlight the impor-tance of our new edge constraint for applications that ac-tually favor AM, we incorporated it into an existing blinddeblurring framework and demonstrated improved perfor-
mance. Experiments reveal that our proposed approach (Xu+ Ep) yields state-of-the-art results for the blind deblurringproblem, with significant improvements in speed-up. Asfuture work, it will be interesting to use the results devel-oped in our paper (directly or indirectly) for other importantapplications such as CRF estimation, normal estimation,depth-aware deblurring, rolling shutter motion deblurringetc.
References
[1] S. Cho and S. Lee. Fast motion deblurring. In ACM Transac-
tions on Graphics (TOG), volume 28, page 145. ACM, 2009.
1, 7
4454
[2] T. S. Cho, S. Paris, B. K. Horn, and W. T. Freeman. Blur
kernel estimation using the radon transform. In Computer
Vision and Pattern Recognition (CVPR), 2011 IEEE Confer-
ence on, pages 241–248. IEEE, 2011. 1, 2, 3, 4, 7
[3] R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T.
Freeman. Removing camera shake from a single photograph.
In ACM Transactions on Graphics (TOG), volume 25, pages
787–794. ACM, 2006. 1, 7
[4] A. Goldstein and R. Fattal. Blur-kernel estimation from spec-
tral irregularities. In European Conference on Computer Vi-
sion, pages 622–635. Springer, 2012. 1
[5] A. Gupta, N. Joshi, C. L. Zitnick, M. Cohen, and B. Curless.
Single image deblurring using motion density functions. In
European Conference on Computer Vision, pages 171–184.
Springer, 2010. 1, 2, 3, 4, 7, 8
[6] S. Harmeling, H. Michael, and B. Scholkopf. Space-
variant single-image blind deconvolution for removing cam-
era shake. In Advances in Neural Information Processing
Systems, pages 829–837, 2010. 7
[7] M. Hirsch, C. J. Schuler, S. Harmeling, and B. Scholkopf.
Fast removal of non-uniform camera shake. In 2011 Inter-
national Conference on Computer Vision, pages 463–470.
IEEE, 2011. 1, 6, 7
[8] M. Hirsch, S. Sra, B. Scholkopf, and S. Harmeling. Efficient
filter flow for space-variant multiframe blind deconvolution.
In CVPR, volume 1, page 2, 2010. 5
[9] J. Jia. Single image motion deblurring using transparency.
In 2007 IEEE Conference on Computer Vision and Pattern
Recognition, pages 1–8. IEEE, 2007. 1
[10] N. Joshi, R. Szeliski, and D. J. Kriegman. Psf estimation
using sharp edge prediction. In Computer Vision and Pattern
Recognition, 2008. CVPR 2008. IEEE Conference on, pages
1–8. IEEE, 2008. 1
[11] N. Joshi, C. L. Zitnick, R. Szeliski, and D. J. Kriegman. Im-
age deblurring and denoising using color priors. In Computer
Vision and Pattern Recognition, 2009. CVPR 2009. IEEE
Conference on, pages 1550–1557. IEEE, 2009. 1
[12] P. Kakar, N. Sudha, and W. Ser. Exposing digital image forg-
eries by detecting discrepancies in motion blur. IEEE Trans-
actions on Multimedia, 13(3):443–452, 2011. 1
[13] R. Kohler, M. Hirsch, B. Mohler, B. Scholkopf, and
S. Harmeling. Recording and playback of camera
shake: Benchmarking blind deconvolution with a real-world
database. In European Conference on Computer Vision,
pages 27–40. Springer, 2012. 1, 4, 5, 7
[14] D. Krishnan and R. Fergus. Fast image deconvolution using
hyper-laplacian priors. In Advances in Neural Information
Processing Systems, pages 1033–1041, 2009. 7
[15] D. Krishnan, T. Tay, and R. Fergus. Blind deconvolution
using a normalized sparsity measure. In Computer Vision
and Pattern Recognition (CVPR), 2011 IEEE Conference on,
pages 233–240. IEEE, 2011. 1, 7
[16] A. Levin, Y. Weiss, F. Durand, and W. T. Freeman. Un-
derstanding and evaluating blind deconvolution algorithms.
In Computer Vision and Pattern Recognition, 2009. CVPR
2009. IEEE Conference on, pages 1964–1971. IEEE, 2009.
1, 2
[17] A. Levin, Y. Weiss, F. Durand, and W. T. Freeman. Effi-
cient marginal likelihood optimization in blind deconvolu-
tion. In Computer Vision and Pattern Recognition (CVPR),
2011 IEEE Conference on, pages 2657–2664. IEEE, 2011. 1
[18] H.-Y. Lin and C.-H. Chang. Depth recovery from motion
blurred images. In 18th International Conference on Pattern
Recognition (ICPR’06), volume 1, pages 135–138. IEEE,
2006. 1
[19] J. Liu, S. Ji, J. Ye, et al. Slep: Sparse learning with efficient
projections. Arizona State University, 6:491, 2009. 7
[20] T. Michaeli and M. Irani. Blind deblurring using internal
patch recurrence. In European Conference on Computer Vi-
sion, pages 783–798. Springer, 2014. 1
[21] J. Pan, Z. Hu, Z. Su, and M.-H. Yang. Deblurring text im-
ages via l0-regularized intensity and gradient prior. In Pro-
ceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 2901–2908, 2014. 1, 2, 6
[22] J. Pan, D. Sun, H. Pfister, and M.-H. Yang. Blind image
deblurring using dark channel prior. In The IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), June
2016. 1, 2, 6, 7, 8
[23] C. Paramanand and A. Rajagopalan. Shape from sharp and
motion-blurred image pair. International journal of com-
puter vision, 107(3):272–292, 2014. 2, 3
[24] C. Paramanand and A. N. Rajagopalan. Non-uniform mo-
tion deblurring for bilayer scenes. In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recog-
nition, pages 1115–1122, 2013. 1, 3, 7
[25] M. P. Rao, A. Rajagopalan, and G. Seetharaman. Inferring
plane orientation from a single motion blurred image. In
Pattern Recognition (ICPR), 2014 22nd International Con-
ference on, pages 2089–2094. IEEE, 2014. 7
[26] C. J. Schuler, M. Hirsch, S. Harmeling, and B. Scholkopf.
Learning to deblur. IEEE Transactions on Pattern Analysis
and Machine Intelligence, 38(7):1439–1451, 2016. 7
[27] Q. Shan, J. Jia, and A. Agarwala. High-quality motion
deblurring from a single image. In ACM Transactions on
Graphics (TOG), volume 27, page 73. ACM, 2008. 1, 7
[28] S. Su and W. Heidrich. Rolling shutter motion deblurring.
In 2015 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pages 1529–1537. IEEE, 2015. 1, 4, 7
[29] L. Sun, S. Cho, J. Wang, and J. Hays. Edge-based blur kernel
estimation using patch priors. In Computational Photogra-
phy (ICCP), 2013 IEEE International Conference on, pages
1–8. IEEE, 2013. 1
[30] Y.-W. Tai, X. Chen, S. Kim, S. J. Kim, F. Li, J. Yang, J. Yu,
Y. Matsushita, and M. S. Brown. Nonlinear camera response
functions and image deblurring: Theoretical analysis and
practice. IEEE transactions on pattern analysis and machine
intelligence, 35(10):2498–2512, 2013. 1, 2, 4, 7
[31] Y.-W. Tai, P. Tan, and M. S. Brown. Richardson-lucy de-
blurring for scenes under a projective motion path. IEEE
Transactions on Pattern Analysis and Machine Intelligence,
33(8):1603–1618, 2011. 1
[32] O. Whyte, J. Sivic, and A. Zisserman. Deblurring shaken and
partially saturated images. In Proceedings of the IEEE Work-
shop on Color and Photometry in Computer Vision, with
ICCV, 2011. 6, 7
4455
[33] O. Whyte, J. Sivic, and A. Zisserman. Deblurring shaken and
partially saturated images. International Journal of Com-
puter Vision, 2014. 7
[34] O. Whyte, J. Sivic, A. Zisserman, and J. Ponce. Non-uniform
deblurring for shaken images. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition,
2010. 1, 2, 4, 7, 8
[35] L. Xu and J. Jia. Two-phase kernel estimation for robust mo-
tion deblurring. In European conference on computer vision,
pages 157–170. Springer, 2010. 1, 7
[36] L. Xu and J. Jia. Depth-aware motion deblurring. In Compu-
tational Photography (ICCP), 2012 IEEE International Con-
ference on, pages 1–8. IEEE, 2012. 1, 7
[37] L. Xu, S. Zheng, and J. Jia. Unnatural l0 sparse represen-
tation for natural image deblurring. In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recogni-
tion, pages 1107–1114, 2013. 1, 2, 6, 7, 8
[38] Y. Zheng, S. Nobuhara, and Y. Sheikh. Structure from mo-
tion blur in low light. In Computer Vision and Pattern Recog-
nition (CVPR), 2011 IEEE Conference on, pages 2569–2576.
IEEE, 2011. 1
4456
Recommended