Upload
wenyi-wang
View
222
Download
0
Embed Size (px)
Citation preview
8/2/2019 Optic Flow in Harmony
1/21
Int J Comput Vis (2011) 93: 368388
DOI 10.1007/s11263-011-0422-6
Optic Flow in Harmony
Henning Zimmer
Andrs Bruhn
Joachim Weickert
Received: 30 August 2010 / Accepted: 13 January 2011 / Published online: 26 January 2011
Springer Science+Business Media, LLC 2011
Abstract Most variational optic flow approaches just con-
sist of three constituents: a data term, a smoothness termand a smoothness weight. In this paper, we present an ap-
proach that harmonises these three components. We start by
developing an advanced data term that is robust under out-
liers and varying illumination conditions. This is achieved
by using constraint normalisation, and an HSV colour rep-
resentation with higher order constancy assumptions and a
separate robust penalisation. Our novel anisotropic smooth-
ness is designed to work complementary to the data term.
To this end, it incorporates directional information from the
data constraints to enable a filling-in of information solely
in the direction where the data term gives no information,
yielding an optimal complementary smoothing behaviour.
This strategy is applied in the spatial as well as in the spatio-
temporal domain. Finally, we propose a simple method for
automatically determining the optimal smoothness weight.
This method bases on a novel concept that we call opti-
mal prediction principle (OPP). It states that the flow field
obtained with the optimal smoothness weight allows for the
best prediction of the next frames in the image sequence.
The benefits of our optic flow in harmony (OFH) approach
H. Zimmer ()
J. Weickert
Mathematical Image Analysis Group, Faculty of Mathematics and
Computer Science, Saarland University, Campus E1.1, 66041
Saarbrcken, Germany
e-mail: [email protected]
J. Weickert
e-mail: [email protected]
A. Bruhn
Vision and Image Processing Group, Cluster of Excellence
Multimodal Computing and Interaction, Saarland University,
Campus E1.1, 66041 Saarbrcken, Germany
e-mail: [email protected]
are demonstrated by an extensive experimental validation
and by a competitive performance at the widely used Mid-dlebury optic flow benchmark.
Keywords Optic flow Variational methods Anisotropicsmoothing Robust penalisation Motion tensor Parameter selection
1 Introduction
Despite almost three decades of research on variational optic
flow approaches, there have been hardly any investigations
on the compatibility of their three main components: the
data term, the smoothness term and the smoothness weight.
While the data term models constancy assumptions on im-
age features, the smoothness term penalises fluctuations in
the flow field, and the smoothness weight determines the
balance between the two terms. In this paper, we present the
optic flow in harmony (OFH) method, which harmonises the
three constituents by following two main ideas:
(i) Widely-used data terms such as the one resulting from
the linearised brightness constancy assumption only
constrain the flow in one direction. However, most
smoothness terms impose smoothness also in the data
constraint direction, leading to an undesirable interfer-
ence. A notable exception is the anisotropic smoothness
term that has been proposed by Nagel and Enkelmann
(1986) and modified by Schnrr (1993). At large image
gradients, this regulariser solely smoothes the flow field
along image edges. For a basic data term modelling the
brightness constancy assumption, this smoothing direc-
tion is orthogonal to the data constraint direction and
thus both terms complement each other in an optimal
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]8/2/2019 Optic Flow in Harmony
2/21
Int J Comput Vis (2011) 93: 368388 369
manner. Unfortunately, this promising concept of com-
plementarity between data and smoothness term has not
been further investigated. Our paper revives this con-
cept for state-of-the-art optic flow models by presenting
a novel complementary smoothness term in conjunction
with an advanced data term.
(ii) Having adjusted the smoothing behaviour to the im-
posed data constraints, it remains to determine the op-timal balance between the two terms for the image se-
quence under consideration. This comes down to select-
ing an appropriate smoothness weight, which is usually
considered a difficult task. We propose a method that
is easy to implement for all variational optic flow ap-
proaches and nevertheless gives surprisingly good re-
sults. It bases on the assumption that the flow estimate
obtained by an optimal smoothness weight allows for
the best possible prediction of the next frames in the
image sequence. This novel concept we name optimal
prediction principle (OPP).
1.1 Related Work
In the first ten years of research on optic flow, several basic
strategies have been considered, e.g. phase-based methods
(Fleet and Jepson 1990), local methods (Lucas and Kanade
1981; Bign et al. 1991) and energy-based methods (Horn
and Schunck1981). In recent years, the latter class of meth-
ods became increasingly popular, mainly due to their poten-
tial for giving highly accurate results. Within energy-based
methods, one can distinguish discrete approaches that min-
imise a discrete energy function and are often probabilisti-cally motivated, and variational approaches that minimise a
continuous energy functional.
Our variational approach naturally incorporates concepts
that have proven their benefits over the years. In the follow-
ing, we briefly review advances in the design of data and
smoothness terms that are influential for our work.
Data Terms To cope with outliers caused by noise or oc-
clusions, Black and Anandan (1996) replaced the quadratic
penalisation from Horn and Schunck (1981) by a robust sub-
quadratic penaliser.
To obtain robustness under additive illumination changes,Brox et al. (2004) combined the classical brightness con-
stancy assumption of Horn and Schunck (1981) with the
higher-order gradient constancy assumption (Tretiak and
Pastor 1984; Schnrr 1994). Bruhn and Weickert (2005) im-
proved this idea by a separate robust penalisation of bright-
ness and gradient constancy assumption. This gives advan-
tages if one of the two constraints produces an outlier. Re-
cently, Xu et al. (2010) went a step further and estimate
a binary map that locally selects between imposing either
brightness or gradient constancy. An alternative to higher-
order constancy assumptions can be to preprocess the im-
ages by a structure-texture decomposition (Wedel et al.
2008).
In addition to additive illumination changes, realistic sce-
narios also encompass a multiplicative part (van de Weijer
and Gevers 2004). For colour image sequences, this issue
can be tackled by normalising the colour channels (Gollandand Bruckstein 1997), or by using alternative colour spaces
with photometric invariances (Golland and Bruckstein 1997;
van de Weijer and Gevers 2004; Mileva et al. 2007). If one is
restricted to greyscale sequences, using log-derivatives (Mil-
eva et al. 2007) can be useful.
A further successful modification of the data term has
been reported by performing a constraint normalisation (Si-
moncelli et al. 1991; Lai and Vemuri 1998; Schoenemann
and Cremers 2006). It prevents an undesirable overweight-
ing of the data term at large image gradient locations.
Smoothness Terms First ideas go back to Horn and Schunck(1981) who useda homogeneous regulariser that does not re-
spect any flow discontinuities. Since different image objects
may move in different directions, it is, however, desirable to
also permit discontinuities.
This can for example be achieved by using image-driven
regularisers that take into account image discontinuities. Al-
varez et al. (1999) proposed an isotropic model with a scalar-
valued weight function that reduces the regularisation at im-
age edges. An anisotropic counterpart that also exploits the
directional information of image discontinuities was intro-
duced by Nagel and Enkelmann (1986). Their method regu-
larises the flow field along image edges but not across them.As noted by Xiao et al. (2006), this comes down to con-
volving the flow field with an oriented Gaussian where the
orientation is adapted to the image edges.
Note that for a basic data term modelling the brightness
constancy assumption, the image edge direction coincides
with the complementary direction orthogonal to the data
constraint direction. In such cases, the regularisers of Nagel
and Enkelmann (1986) and Schnrr (1993) can thus be con-
sidered as early complementary smoothness terms.
Of course, not every image edge will coincide with a
flow edge. Thus, image-driven strategies are prone to give
oversegmentation artefacts in textured image regions. To
avoid this, flow-driven regularisers have been proposed that
respect discontinuities of the evolving flow field and are
therefore not misled by image textures. In the isotropic set-
ting this comes down to the use of robust, subquadratic pe-
nalisers which are closely related to line processes (Blake
and Zisserman 1987). For energy-based optic flow meth-
ods, such a strategy was used by Shulman and Herv (1989)
and Schnrr (1994). An anisotropic extension was later pre-
sented by Weickert and Schnrr (2001a). The drawback of
8/2/2019 Optic Flow in Harmony
3/21
370 Int J Comput Vis (2011) 93: 368388
flow-driven regularisers lies in less well-localised flow edges
compared to image-driven approaches.
Concerning the individual problems of image- and flow-
driven strategies, the idea arises to combine the advantages
of both worlds. This goal was first achieved in the discrete
method of Sun et al. (2008). There, the authors developed
an anisotropic regulariser that uses directional flow deriva-
tives steered by image structures. This allows to adapt thesmoothing direction to the direction of image structures and
the smoothing strength to the flow contrast. We call such a
strategy image- and flow-driven regularisation. It combines
the benefits of image- and flow-driven methods, i.e. sharp
flow edges without oversegmentation problems.
The smoothness terms discussed so far only assume
smoothness of the flow field in the spatial domain. As im-
age sequences usually consist of more than two frames,
yielding more than one flow field, it makes sense to also
assume temporal smoothness of the flow fields. This leads
to spatio-temporal smoothness terms. In a discrete setting
they go back to Murray and Buxton (1987). For varia-tional approaches, an image-driven spatio-temporal smooth-
ness terms was proposed by Nagel (1990) and a flow-driven
counterpart was later presented by Weickert and Schnrr
(2001b). The latter smoothness term was later successfully
used in the methods of Brox et al. (2004) and Bruhn and
Weickert (2005).
Recently, non-local smoothing strategies (Yaroslavsky
1985) have been introduced to the optic flow community by
Sun et al. (2010) and Werlberger et al. (2010). In these ap-
proaches, it is assumed that the flow vector at a certain pixel
is similar to the vectors in a (possibly large) spatial neigh-
bourhood. Adapting ideas from Yoon and Kweon (2006),the similarity to the neighbours is weighted by a bilateral
weight that depends on the spatial as well as on the colour
value distance of the pixels. Due to the colour value dis-
tance, these strategies can be classified as image-driven ap-
proaches and are thus also prone to oversegmentation prob-
lems. However, comparing non-local strategies to the pre-
viously discussed smoothness terms is somewhat difficult:
Whereas non-local methods explicitly model similarity in a
certain neighbourhood, the previous smoothness terms oper-
ate on flow derivatives that only consider their direct neigh-
bours. Nevertheless, the latter strategies model a globally
smooth flow field as each pixel communicates with each
other pixel through its neighbours.
Automatic Parameter Selection It is well-known that an
appropriate choice of the smoothness weight is crucial for
obtaining favourable results. Nevertheless, there has been
remarkably little research on methods that automatically es-
timate the optimal smoothness weight or other model para-
meters.
Concerning an optimal selection of the smoothness
weight for variational optic flow approaches, Ng and Solo
(1997) proposed an error measure which can be estimated
from the image sequence and the flow estimate only. Using
this measure, a brute-force search for the smoothness weight
that gives the smallest error is performed. Computing the
proposed error measure is, however, computationally expen-
sive, especially for robust data terms. Ng and Solo (1997)
hence restricted their focus to the basic method of Horn
and Schunck (1981). In a Bayesian framework, a parame-ter selection approach that can also handle robust data terms
was presented by Krajsek and Mester (2007). This method
jointly estimates the flow and the model parameters where
the latter encompass the smoothness weight and also the
relative weights of different data terms. This method does
not require a brute-force search, but the minimisation of the
objective function is more complicated and only computa-
tionally feasible if certain approximations are performed.
1.2 Our Contributions
The OFH method is obtained in three steps. We first de-velop a robust and invariant data term. Then an anisotropic
image- and flow-driven smoothness term is designed that
works complementary to the data term. Finally we propose
a simple method for automatically determining the optimal
smoothness weight for the given image sequence.
Our data term combines the brightness and the gradient
constancy assumption, and performs a constraint normali-
sation. It further uses a Hue-Saturation-Value (HSV) colour
representation with a separate robustification of each chan-
nel. The latter is motivated by the fact that each channel in
the HSV space has a distinct level of photometric invariance
and information content. Hence, a separate robustificationallows to choose the most reliable channel at each position.
Our anisotropic complementary smoothness term takes
into account directional information from the constraints im-
posed in the data term. Across constraint edges, we per-
form a robust penalisation to reduce the smoothing in the
direction where the data term gives the most information.
Along constraint edges, where the data term gives no in-
formation, a strong filling-in by using a quadratic penalisa-
tion makes sense. This strategy not only allows for an op-
timal complementarity between data and smoothness term,
but also leads to a desirable image- and flow-driven behav-
iour. We further show that our regulariser can easily be ex-
tended to work in the spatio-temporal domain.
Our method for determining the optimal smoothness
weight bases on the proposed OPP concept. This results
in finding the optimal smoothness weight as the one cor-
responding to the flow field with the best prediction quality.
To judge the latter, we evaluate the data constraints between
the first and the third frame of the sequence. Assuming a
constant speed and linear trajectory of objects within the
considered frames, this can be realised by simply doubling
8/2/2019 Optic Flow in Harmony
4/21
Int J Comput Vis (2011) 93: 368388 371
the flow vectors. Due to its simplicity, our method is easy
to implement for all variational optic flow approaches, but
nevertheless produces surprisingly good results.
The present article extends our shorter conference paper
(Zimmer et al. 2009) by the following points: (i) A more
extensive derivation and discussion of the data term. (ii)
An explicit discussion on the adequate treatment of the hue
channel of the HSV colour space. (iii) A taxonomy of ex-isting smoothness terms within a novel general framework.
The latter allows to reformulate most existing as well as our
novel regulariser in a common notation. (iv) The extension
of our complementary regulariser to the spatio-temporal do-
main. (v) A simple method for automatically selecting the
smoothness weight. (vi) A deeper discussion of implemen-
tation issues. (vii) A more extensive experimental validation.
Organisation In Sect. 2 we present our variational optic
flow model with the robust data term and the complementary
smoothness term. The latter is then extended to the spatio-
temporal domain. Section 3 describes the method proposedfor determining the smoothness weight. After discussing
implementation issues in Sect. 4, we show experiments in
Sect. 5. The paper is finished with conclusions and an out-
look on future work in Sect. 6.
2 Variational Optic Flow
Let f (x) be a greyscale image sequence where x :=(x,y,t). Here, the vector (x,y) denotes the locationwithin a rectangular image domain
R
2, and t [
0, T]denotes time. We further assume that f is presmoothed by a
Gaussian convolution: Given an image sequence f0(x), we
obtain f (x) = (K f0)(x), where K is a spatial Gaussianof standard deviation and denotes the convolution oper-ator. This presmoothing step helps to reduce the influence of
noise and additionally makes the image sequence infinitely
many times differentiable, i.e. f C.The optic flow field w := (u,v, 1) describes the dis-
placement vector field between two frames at time t and
t+ 1. It is found by minimising a global energy functionalof the general form
E(u,v) =
M(u,v) + V (2u, 2v)
dx dy, (1)
where 2 := (x , y ) denotes the spatial gradient operator.The term M(u,v) denotes the data term, V (2u, 2v) thesmoothness term, and > 0 is a smoothness weight. Note
that the energy (1) refers to the spatial case where one com-
putes one flow field between two frames at time t and t+ 1.The more general spatio-temporal case that uses all frames
t [0, T] will be presented in Sect. 2.3.
According to the calculus of variations (Elsgolc 1962),
a minimiser (u,v) of the energy (1) necessarily has to fulfil
the associated Euler-Lagrange equations
uM
x
ux V+ y uy V = 0, (2)
vM
x
vx V
+ y
vy V
= 0 (3)
with homogeneous Neumann boundary conditions.
2.1 Data Term
Let us now derive our data term in a systematic way. The
classical starting point is the brightness constancy assump-
tion used by Horn and Schunck (1981). It states that im-
age intensities remain constant under their displacement,
i.e. f (x + w) = f (x). Assuming that the image sequenceis smooth (which can be guaranteed by the Gaussian pres-
moothing) and that the displacements are small, we can per-
form a first-order Taylor expansion that yields the linearised
optic flow constraint (OFC)
0 = fx u + fy v + ft = 3 f w, (4)
where 3 := (x , y , t) is the spatio-temporal gradientoperator and subscripts denote partial derivatives. With a
quadratic penalisation, the corresponding data term is given
by
M1(u,v) =3 f w2 = wJ0 w, (5)
with the tensor
J0 := 3f3 f. (6)
The single equation given by the OFC involves two un-
knowns u and v. It is thus not sufficient to compute a unique
solution, which is known as the aperture problem (Bertero
et al. 1988). Nevertheless, the OFC does allow to compute
the flow component orthogonal to image edges, the so-called
normal flow. For |2f| = 0 it is defined as
wn :=
un , 1
:=
ft|2f|
2 f|2f|
, 1
. (7)
Normalisation Our experiments will show that normalis-
ing the data term can be beneficial. Following Simoncelli
et al. (1991), Lai and Vemuri (1998), Schoenemann and Cre-
mers (2006) and using the abbreviation u := (u,v), werewrite the data term M1 as
M1(u,v) =2 f u + ft2
=|2f|
2 f u|2f|
+ ft|2f|
2
8/2/2019 Optic Flow in Harmony
5/21
372 Int J Comput Vis (2011) 93: 368388
Fig. 1 Geometric interpretation of the rewritten data term (8)
= |2f|2 2 f
|2f|
u + ft2f|2f|2
2
= |2f
|2
2 f
|2f|u un =:d
2
. (8)
The term d constitutes a projection of the difference between
the estimated flow u and the normal flow un in the direction
of the image gradient 2f. In a geometric interpretation,the term d describes the distance from u to the line l in the
uv-space that is given by
v = fxfy
u ftfy
. (9)
On this line, the flow u has to lie according to the OFC (4),
and the normal flow un is the smallest vector that lies on
this line. A sketch of this situation is given in Fig. 1. Our
geometric interpretation suggests that one should ideally pe-
nalise the distance d in a data term M2(u,v) = d2. The dataterm M1, however, weighs this distance by the squared spa-
tial image gradient, as M1(u,v) = |2f|2 d2, see (8). Thisresults in a stronger enforcement of the data constraint at
high gradient locations. This overweighting is undesirable as
large gradients can be caused by unreliable structures, such
as noise or occlusions.
As a remedy, we normalise the data term M1 by multiply-
ing it with a factor (Simoncelli et al. 1991; Lai and Vemuri1998)
0 :=1
|2f|2 + 2, (10)
where the regularisation parameter > 0 avoids division by
zero. Additionally, it reduces the influence of small gradients
which are significantly smaller than 2, e.g. noise in flat re-
gions. Nevertheless, the normalisation is not influenced for
large gradients. Thus, it may pay off to choose a larger value
of in the presence of noise. Incorporating the normalisa-
tion into M1 leads to the data term
M2(u,v) = wJ0 w, (11)
with the normalised tensor
J
0 :=
0J
0 =
03f3 f. (12)Gradient Constancy Assumption To render the data term
robust under additive illumination changes, it was proposed
to impose the gradient constancy assumption (Tretiak and
Pastor 1984; Schnrr 1994; Brox et al. 2004). In contrast
to the brightness constancy assumption, it states that im-
age gradients remain constant under their displacement, i.e.
2f (x + w) = 2f (x). A Taylor linearisation gives
3 fx w = 0 and 3 fy w = 0, (13)
respectively. Combining both brightness and gradient con-stancy assumption in a quadratic way gives the data term
M3(u,v) = wJ w, (14)
where the tensor J can be written in the motion tensor no-
tation of Bruhn et al. (2006) that allows to combine the two
constancy assumptions in a joint tensor
J := J0 + Jxy := J0 +
Jx + Jy
:= 3f3 f +
3fx 3 fx + 3fy 3 fy
, (15)
where the parameter > 0 steers the contribution of the gra-dient constancy assumption.
To normalise M3, we replace the motion tensor J by its
normalised counterpart
J := J0 + Jxy := J0 +
Jx + Jy
:= 0 J0 +
x Jx + y Jy
:= 03f3 f
+ x3fx 3 fx+ y3fy 3 fy, (16)
with two additional normalisation factors defined as
x :=1
|2fx|2 + 2and y :=
1
|2fy |2 + 2. (17)
The normalised data term M4 is given by
M4(u,v) = wJ w. (18)
Colour Image Sequences In a next step we extend our data
term to multi-channel sequences (f1(x), f2(x), f3(x)). If
8/2/2019 Optic Flow in Harmony
6/21
Int J Comput Vis (2011) 93: 368388 373
one uses the standard RGB colour space, the three channels
represent the red, green and blue channel, respectively. We
couple the three colour channels in the motion tensor
J c :=3
i=1J i :=
3i=1
J i0 + J ixy
:=3
i=1
J i0 +
J ix + J iy
:=3
i=1
i03fi 3 fi
+ ix3fix 3 fix+ iy3fiy 3 fiy, (19)
with normalisation factors i for each colour channel fi .
The corresponding data term reads as
M5(u,v) = wJc
w. (20)
Photometric Invariant Colour Spaces Realistic illumina-
tion models encompass a multiplicative influence (van de
Weijer and Gevers 2004), which cannot be captured by the
gradient constancy assumption that is only invariant under
additive illumination changes. This problem can be tackled
by using the Hue Saturation Value (HSV) colour space, as
proposed by Golland and Bruckstein (1997). The hue chan-
nel is invariant under global and local multiplicative illu-
mination changes, as well as under local additive changes.
The saturation channel is only invariant under global mul-
tiplicative illumination changes, and the value channel ex-hibits no invariances. Mileva et al. (2007) thus only used
the hue channel for optic flow computation as it exhibits the
most invariances. We will additionally use the saturation and
value channel, because they contain information that is not
encoded in the hue channel.
As an example, consider the HSV decomposition of the
Rubberwhale image shown in Fig. 2. As we can see, the
shadow at the left of the wheel (shown in the zoom) is not
present in the hue and the saturation channel, but appears
in the value channel. Nevertheless, especially the hue chan-
nel discards a lot of image information, as can be observed
for the striped cloth. This information is, on the other hand,available in the value channel.
One problem when using a HSV colour representation is
that the hue channel f1 describes an angle in a colour circle,
i.e. f1 [0, 360). The hue channel is hence not differen-tiable at the interface between 0 and 360. Our solution tothis problem is to consider the unit vector (cos f1, sin f1)
corresponding to the angle f1. This results in treating the
hue channel as two (coupled) channels, which are both dif-
ferentiable. The corresponding motion tensor for the bright-
Fig. 2 HSV decomposition on the example of the Rubberwhale im-
age from the Middlebury optic flow database (Baker et al. 2009,
http://vision.middlebury.edu/flow/data/). First row, from left to right:
(a) Colour image with zoom in shadow region left of the wheel. (b)
Hue channel, visualised with full saturation and value. Second row,
from left to right: (c) Saturation channel. (d) Value channel
ness constancy assumption consequently reads as
J 10 := 103 cos f1 3 cos f1 + 3 sin f1 3 sin f1,
(21)
where the normalisation factor is defined as
10 :=12 cos f12 + 2 sin f12 + 2 . (22)
The tensor J 1xy for the gradient constancy assumption is
adapted accordingly. Note that in the differentiable parts of
the hue channel, the motion tensor (21) is equivalent to ourearlier definition, as
3 cos f1 3 cos f1 + 3 sin f1 3 sin f1
= sin2 f13f1 3 f1+ cos2 f13f1 3 f1=3f13 f1, (23)
where denotes the gradient in the differentiable parts ofthe hue channel.
Robust Penalisers To provide robustness of the data term
against outliers caused by noise and occlusions, Black andAnandan (1996) proposed to refrain from a quadratic penali-
sation. Instead they use a subquadratic penalisation function
M(s2), where s2 denotes the quadratic data term. Using
such a robust penaliser within our data term yields
M6(u,v) = M
wJ c w
. (24)
Good results are reported by Brox et al. (2004) for the sub-
quadratic penaliser M(s2) :=
s2 + 2 using a small reg-
ularisation parameter > 0.
http://vision.middlebury.edu/flow/data/http://vision.middlebury.edu/flow/data/8/2/2019 Optic Flow in Harmony
7/21
374 Int J Comput Vis (2011) 93: 368388
Bruhn and Weickert (2005) later proposed a separate
penalisation of the brightness and the gradient constancy
assumption, which is advantageous if one assumption pro-
duces an outlier. Incorporating this strategy into our ap-
proach gives the data term
M7(u,v) = M
wJ c0 w
+ M
wJ cxy w
, (25)
where the separate motion tensors are defined as
J c0 :=3
i=1J i0 and J
cxy :=
3i=1
J ixy . (26)
We will go further by proposing a separate robustification of
each colour channel in the HSV space. This can be justified
by the distinct information content of each of the three chan-
nels, see Fig. 2, that drives the optic flow estimation in dif-
ferent ways. The separate robustification then downweights
the influence of less appropriate colour channels.
Final Data Term Incorporating our separate robustifica-tion idea into M7 brings us to our final data term
M(u,v) =3
i=1M
wJ i0 w
+ 3i=1
M
wJ ixy w
,
(27)
with the motion tensors J 1 adapted to the HSV colour space
as described before. Note that our final data term is (i) nor-
malised, (ii) combines the brightness and gradient constancy
assumption, and (iii) uses the HSV colour space with (iv) a
separate robustification of all colour channels.
The contributions of our data term (27) to the Euler-Lagrange equations (2) and (3) are given by
uM=3
i=1
M
wJ i0 w
J i01,1u + J i01,2v + J i01,3+
3
i=1
M
wJ ixy w
J ixy
1,1
u +
J ixy
1,2
v +
J ixy
1,3
, (28)
vM =3
i=1
M
wJ i0 w
J i01,2u + J i02,2v + J i02,3+
3
i=1
M
wJ ixy w
J ixy1,2u + J ixy2,2v + J ixy2,3
, (29)
where [J]m,n denotes the entry in row m and column n ofthe tensor J, and M(s
2) denotes the derivative of M(s2)
w.r.t. its argument. Analysing the terms (28)a n d (29),we see
that the separate robustification of the HSV channels makes
sense: If a specific channel violates the imposed constancy
assumption at a certain location, the corresponding argu-
ment of the decreasing function M will be large, yielding a
downweighting of this channel. Other channels that satisfythe constancy assumption then have a dominating influence
on the solution. This will be confirmed by a specific experi-
ment in Sect. 5.1.
2.2 Smoothness Term
Following the extensive taxonomy on optic flow regularisers
by Weickert and Schnrr (2001a), we sketch some existing
smoothness terms that led to our novel complementary regu-
lariser. We rewrite the regularisers in a novel framework that
unifies their notation and eases their comparison.
Preliminaries for the General Framework We first intro-
duce concepts that will be used in our general framework.
Anisotropic image-driven regularisers take into account
directional information from image structures. These infor-
mation can be obtained by considering the structure tensor
(Frstner and Glch 1987)
S := K 2f2 f=: 2
i=1i si s
i , (30)
with an integration scale > 0. The structure tensor is asymmetric, positive semidefinite 2 2 matrix that possessestwo orthonormal eigenvectors s1 and s2 with corresponding
eigenvalues 1 2 0. The vector s1 points across im-age structures, whereas the vector s2 points along them. In
the case for = 0, i.e. without considering neighbourhoodinformation, one obtains
s01 =2f|2f|
and s02 =2 f|2f|
, (31)
where
2 f
:=(
fy , fx )
denotes the vector orthogonalto 2f. For = 0 we also have that |2f|2 = tr S0, with trdenoting the trace operator.
Most regularisers impose smoothness by penalising the
magnitude of the flow gradients. As s1 and s2 constitute an
orthonormal basis, we can write
|2u|2 = u2x + u2y = u2s1 + u2s2 , (32)
using the directional derivatives usi := si 2u. A corre-sponding rewriting can also be performed for |2v|2.
8/2/2019 Optic Flow in Harmony
8/21
Int J Comput Vis (2011) 93: 368388 375
To analyse the smoothing behaviour of the regularisers,
we will consider the corresponding Euler-Lagrange equa-
tions that can be written in the form
uM div (D 2u) = 0, (33)vM div (D 2v) = 0, (34)
with a diffusion tensor D that steers the smoothing of the
flow components u and v. More specific, the eigenvectors
of D give the smoothing direction, and the corresponding
eigenvalues determine the magnitude of smoothing.
Homogeneous Regularisation First ideas for the smooth-
ness term go back to Horn and Schunck (1981) who used a
homogeneous regulariser. In our framework it reads as
VH(2u, 2v) :=|2u|2 + |2v|2
= u2
s1 + u2
s2 + v2
s1 + v2
s2 . (35)
The corresponding diffusion tensor is equal to the unit ma-
trix DH = I. The smoothing processes thus perform homo-geneous diffusion that blurs important flow edges.
Image-Driven Regularisation To obtain sharp flow edges,
image-driven methods (Nagel and Enkelmann 1986; Al-
varez et al. 1999) reduce the smoothing at image edges, in-
dicated by large values of |2f|2 = tr S0.An isotropic image-driven regulariser goes back to Al-
varez et al. (1999) who used
VII(2u, 2v) := g|2f|2 |2u|2 + |2v|2
= g
tr S0
u2s1 + u2s2
+ v2s1 + v2s2
, (36)
where g is a decreasing, strictly positive weight function.
The corresponding diffusion tensor DII = g(tr S0) I showsthat the weight function allows to decrease the smoothing in
accordance to the strength of image edges.
The anisotropic image-driven regulariser of Nagel and
Enkelmann (1986) prevents smoothing of the flow field
across image boundaries but encourages smoothing along
them. This is achieved by the regulariser
VAI(2u, 2v) := 2 u P(2f )2u + 2 v P2f2v,
(37)
where P(2f ) denotes a regularised projection matrix per-pendicular to the image gradient. It is defined as
P(2f ) :=1
|2f|2 + 222 f
2 f + 2I . (38)
with a regularisation parameter > 0. In our common
framework, this regulariser can be written as
VAI(2u, 2v) =2
tr S0 + 22
u2s0
1
+ v2s0
1
+ tr S0 +
2
tr S0
+22
u2s0
2
+ v2s0
2. (39)
The correctness of above rewriting can easily be verified and
is based on the observations that s01 and s02 are the eigenvec-
tors of P, and that the factors in front of (u2s0
1
+ v2s0
1
) and
(u2s0
2
+ v2s0
2
) are the corresponding eigenvalues. The diffusion
tensor for the regulariser of Nagel and Enkelmann (1986) is
identical to the projection matrix: DAI = P. Concerning itseigenvectors and eigenvalues, we observe that in the lim-
iting case for 0, where VAI u2s02
+ v2s0
2
, we obtain
a smoothing solely in s02-direction, i.e. along image edges.
In the definition of the normal flow (7) we have seen that
a data term that models the brightness constancy assump-
tion constraints the flow only orthogonal to image edges. In
the limiting case, the regulariser of Nagel and Enkelmann
can hence be interpreted as a first complementary smooth-
ness term that fills in information orthogonal to the data con-
straint direction. This complementarity has been emphasised
by Schnrr (1993), who contributed a theoretical analysis as
well as modifications of the original Nagel and Enkelmann
functional.
The drawback of image-driven strategies is that they are
prone to oversegmentation artefacts in textured image re-
gions where image edges do not necessarily correspond to
flow edges.
Flow-Driven Regularisation To remedy the oversegmen-
tation problem, it makes sense to adapt the smoothing
process to the flow edges instead of the image edges.
In the isotropic setting, Shulman and Herv (1989) and
Schnrr (1994) proposed to use subquadratic penaliser func-
tions for the smoothness term, i.e.
VIF(2u, 2v) := V|2u|2 + |2v|2
= V
u2s1 + u2s2
+ v2s1 + v2s2
, (40)
where the penaliser function V(s2) is preferably increas-
ing, differentiable and convex in s. The associated diffusiontensor is given by
DIF = V
u2s1 + u2s2
+ v2s1 + v2s2
I. (41)
The underlying diffusion processes perform nonlinear iso-
tropic diffusion, where the smoothing is reduced at the
boundaries of the evolving flow field via the decreasing dif-
fusivity V. If one uses the convex penaliser (Cohen 1993)
V(s2) :=
s2 + 2, (42)
8/2/2019 Optic Flow in Harmony
9/21
376 Int J Comput Vis (2011) 93: 368388
one ends up with regularised total variation (TV) regularisa-
tion (Rudin et al. 1992) with the diffusivity
V(s2) = 1
2
s2 + 2 1
2 |s| . (43)
Another possible choice is the non-convex Perona-Malik
regulariser (Lorentzian) (Black and Anandan 1996; Perona
and Malik1990) given by
V(s2) := 2 log
1 + s
2
2
, (44)
that results in Perona-Malik diffusion with the diffusivity
V(s2) = 1
1 + s22
, (45)
using a contrast parameter > 0.
We will not discuss the anisotropic flow-driven regu-
lariser of Weickert and Schnrr (2001a) as it does not fitin our framework and also has not been used in the design
of our complementary regulariser.
Despite the fact that flow-driven methods reduce the
oversegmentation problem caused by image textures, they
suffer from another drawback: The flow edges are not as
well localised as with image-driven strategies.
Image- and Flow-Driven Regularisation We have seen
that image-driven methods suffer from oversegmentation
artefacts, but give sharp flow edges. Flow-driven strategies
remedy the oversegmentation problem but give less pleasant
flow edges. It is thus desirable to combine the advantages ofboth strategies to obtain sharp flow edges without overseg-
mentation problems.
This aim was achieved by Sun et al. (2008) who pre-
sented an anisotropic image- and flow-driven smoothness
term in a discrete setting. It adapts the smoothing direc-
tion to image structures but steers the smoothing strength
in accordance to the flow contrast. In contrast to Nagel and
Enkelmann (1986) who considered 2 f to obtain direc-tional information of image structures, the regulariser from
Sun et al. (2008) analyses the eigenvectors si of the struc-
ture tensor S from (30) to obtain a more robust direction
estimation. A continuous version of this regulariser can bewritten as
VAIF(2u, 2v) := V
u2s1
+ Vv2s1+ V
u2s2
+ Vv2s2. (46)Here, we obtain two diffusion tensors, that for p {u, v}read as
DpAIF = V
p2s1
s1 s
1 + V
p2s2
s2 s
2 . (47)
We observe that these tensors allow to obtain the desired be-
haviour: The regularisation direction is adapted to the image
structure directions s1 and s2, whereas the magnitude of the
regularisation depends on the flow contrast encoded in ps1and ps2 . As a result, one obtains the same sharp flow edges
as image-driven methods but does not suffer from overseg-
mentation problems.
2.2.1 Our Novel Complementary Regulariser
In spite of its sophistication, the anisotropic image- and
flow-driven model from Sun et al. (2008) given in (46) still
suffers from a few shortcomings. We introduce three amend-
ments that we will discuss now.
Regularisation Tensor A first remark w.r.t. the model from
(46) is that the directional information from the structure
tensor S is not consistent with the imposed constraints of
our data term (27). It is more natural to take into account di-
rectional information provided by the motion tensor (19) and
to steer the anisotropic regularisation process w.r.t. con-
straint edges instead of image edges. To this end we pro-
pose to analyse the eigenvectors r1 and r2 of the regularisa-
tion tensor
R :=3
i=1K
i02fi 2 fi
+
ix2fix 2 fix
+ iy
2fiy 2 fiy
, (48)
which can be regarded as a generalisation of the structure
tensor (30). Note that the regularisation tensor differs from
the motion tensor Jc
from (19) by the facts that it (i) in-tegrates neighbourhood information via the Gaussian con-
volution, and (ii) uses the spatial gradient operator 2 in-stead of the spatio-temporal operator 3. The latter is dueto the spatial regularisation. In Sect. 2.3 we extend our reg-
ulariser to the spatio-temporal domain, yielding a regulari-
sation tensor that also uses the spatio-temporal gradient 3.Further note that a Gaussian convolution of the motion ten-
sor leads to a combined local-global (CLG) data term in the
spirit of Bruhn et al. (2005). Our experiments in Sect. 5.1
will analyse in which cases such a modification of our data
term can be useful.
Rotational Invariance The smoothness term VAIF from
(46) lacks the desirable property of rotational invariance, be-
cause the directional derivatives of u and v in the eigenvec-
tor directions are penalised separately. We propose to jointly
penalise the directional derivatives, yielding
VAIFR ,RI (2u, 2v) := V
u2r1 + v2r1
+ Vu2r2 + v2r2,(49)
where we use the eigenvectors ri of the regularisation tensor.
8/2/2019 Optic Flow in Harmony
10/21
Int J Comput Vis (2011) 93: 368388 377
Table 1 Comparison of
regularisation strategies. The
next to last column names the
tensor that is analysed to obtain
directional information for
anisotropic strategies, and the
last column indicates if the
corresponding regulariser is
rotationally invariant
Strategy Regulariser V Directional Rotationally
adaptation invariant
Homogeneous u2s1 + u2s2 + v2s1 + v2s2 (Horn and Schunck1981)
Isotropic image-driven g (tr S0)
u2s1 + u2s2 + v2s1 + v2s2
(Alvarez et al. 1999)
Anisotropic image-driven u2s02+ v2s0
2, for 0 S0
(Nagel and Enkelmann 1986)
Isotropic flow-driven V
u2s1 + u2s2 + v2s1 + v2s2
(Shulman and Herv 1989)
Anisotropic image and V
u2s1
+ V
v2s1
+ V
u2s2
+ V
v2s2
S
flow-driven (Sun et al. 2008)
Anisotropic complementary V
u2r1 + v2r1+ u2r2 + v2r2 R
image- and flow-driven
Single Robust Penalisation. The above regulariser (49)
performs a twofold robust penalisation in both eigenvec-tor directions. However, the data term mainly constraints
the flow in direction of the largest eigenvalue of the spa-
tial motion tensor, i.e. in r1-direction. We hence propose a
single robust penalisation in r1-direction. In the orthogonal
r2-direction, we opt for a quadratic penalisation to obtain
a strong filling-in effect of missing information. The bene-
fits of this design will be confirmed by our experiments in
Sect. 5.2. Incorporating the single robust penalisation finally
yields our complementary regulariser
VCR(2u, 2v) := V
u2r1 + v
2r1
+ u2r2 + v
2r2
, (50)
that complements the proposed robust data term from (27)in an optimal fashion. For the penaliser V, we propose to
use the Perona-Malik regulariser (44).
The corresponding joint diffusion tensor is given by
DCR = V
u2r1 + v2r1
r1r1 + r2r2 , (51)
with V given in (45). The derivation of this diffusion tensoris presented in Appendix.
Discussion To understand the advantages of the comple-
mentary regulariser compared to the anisotropic image- and
flow-driven regulariser (46), we compare our joint diffusion
tensor (51) to its counterparts (47), which reveals the fol-
lowing innovations: (i) The smoothing direction is adapted
to constraint edges instead of image edges, as the eigenvec-
tors of the regularisation tensor ri are used instead of the
eigenvectors of the structure tensor. (ii) We achieve rota-
tional invariance by coupling the two flow components in the
argument of V. (iii) We only reduce the smoothing acrossconstraint edges, i.e. in r1-direction. Along them, always a
strong diffusion with strength 1 is performed, resembling
edge-enhancing anisotropic diffusion (Weickert 1996).
Furthermore, when analysing our joint diffusion ten-
sor, the benefits of the underlying anisotropic image- andflow-driven regularisation become visible. The smoothing
strength across constraint edges is determined by the expres-
sion V(u2r1
+ v2r1 ). Here we can distinguish two scenarios:At a flow edge that corresponds to a constraint edge, the flow
gradients will be large and almost parallel to r1. Thus, the
argument of the decreasing function V will be large, yield-ing a reduced diffusion which preserves this important edge.
At deceiving texture edges in flat flow regions, however,
the flow gradients are small. This results in a small argument
for V, leading to almost homogeneous diffusion. Hence,we perform a pronounced smoothing in both directions that
avoids oversegmentation artefacts.Finally note that our complementary regulariser has the
same structure, even if other data terms are used. Only the
regularisation tensor R has to be adapted to the new data
term.
2.2.2 Summary
To conclude this section, Table 1 summarises the discussed
regularisers rewritten in our framework. It also compares
the way directional information is obtained for anisotropic
strategies, and it indicates if the regulariser is rotationally
invariant. Note that despite the fact these regularisers havebeen developed within almost three decades, our taxonomy
shows their structural similarities.
2.3 Extension to a Spatio-Temporal Smoothness Term
The smoothness terms we have discussed so far model the
assumption of a spatially smooth flow field. As image se-
quences in general encompass more than two frames, yield-
ing several flow fields, it makes sense to also assume a
8/2/2019 Optic Flow in Harmony
11/21
378 Int J Comput Vis (2011) 93: 368388
temporal smoothness of the flow fields, leading to spatio-
temporal regularisation strategies.
A spatio-temporal (ST) version of the general energy
functional (1) reads as
EST(u,v) =[0,T]
M(u,v)
+ VST(3u, 3v)dx dy dt. (52)Compared to the spatial energy (1) we note the additional
integration over the time domain and that the smoothness
term now depends on the spatio-temporal flow gradient.
To extend our complementary regulariser from (50) to the
spatio-temporal domain, we define the spatio-temporal reg-
ularisation tensor
R ST := K Jc. (53)
For = 0 it is identical to the motion tensor Jc from (19).The Gaussian convolution with K is now performed inthe spatio-temporal domain, which also holds for the pres-
moothing of the image sequence. The spatio-temporal regu-
larisation tensor is a 3 3 tensor that possesses three ortho-normal eigenvectors r1, r2 and r3. With their help, we define
the spatio-temporal complementary regulariser (ST-CR)
VSTCR (3u, 3v) := V
u2r1 + v2r1+ u2r2 + v2r2 + u2r3 + v2r3 .
(54)
The corresponding spatio-temporal diffusion tensor reads as
DSTCR=
Vu2r1 + v2r1 r1 r1 + r2 r2 + r3 r3 . (55)3 Automatic Selection of the Smoothness Weight
The last step missing for our OFH method is a strategy that
automatically determines the optimal smoothness parameter
for the image sequence under consideration. This is es-
pecially important in real world applications of optic flow
where no ground truth flow is known. Note that if the latter
would be the case, we could simply select the smoothness
weight that gives the flow field with the smallest deviation
from the ground truth.
3.1 A Novel Concept
We propose an error measure that allows to judge the quality
of a flow field without knowing the ground truth. This error
measure bases on a novel concept, the optimal prediction
principle (OPP). The OPP states that the flow field obtained
with an optimal smoothness weight allows for the best pre-
diction of the next frames in the image sequence. This makes
sense as a too small smoothness weight would lead to an
overfit to the first two frames and consequently result in a
bad prediction of further frames. For too large smoothness
weights, the flow fields would be too smooth and thus also
lead to a bad prediction.
Following the OPP, our error measure needs to judge the
quality of the prediction achieved with a given flow field.
To this end, we evaluate the imposed data constraints be-
tween the first and the third frame of the image sequence,resulting in an average data constancy error (ADCE) mea-
sure. To compute this measure, we assume that the mo-
tion of the scene objects is of more or less constant speed
and that it describes linear trajectories within the considered
three frames. Under these assumptions, we simply double
the flow vectors to evaluate the data constraints between first
and third frame. Following this strategy, we can define the
ADCE between frame 1 and 3 as
ADCE1,3(w)
:=1
| | 3
i=1M i0fi (x + 2w) fi (x)2
+
3i=1
M
ix
fix (x + 2w) fix (x)2
+ iy
fiy (x + 2w) fiy (x)2
dx dy, (56)
where w denotes the flow field obtained with a smoothness
weight . The integrand of above expression is (apart from
the doubled flow field) a variant of our final data term (27)
where no linearisation of the constancy assumptions have
been performed. To evaluate the images at the subpixel lo-
cations fi (x+ 2w) we use Coons patches based on bicubicinterpolation (Coons 1967).
3.2 Determining the Best Parameter
In general, the relation between and the ADCE is not
convex, which excludes the use of gradient descent-like ap-
proaches for finding the optimal value of w.r.t. our error
measure.
We propose a brute-force method similar to the one of
Ng and Solo (1997): We first compute the error measures for
a sufficiently large set of flow fields obtained with differ-
ent values. We then select the that gives the smallest er-
ror. To reduce the number of values to test, we propose to
start from a given, standard value 0, say. This value is then
incremented/decremented n times by multiplying/dividing
it with a stepping factor a > 1, yielding in total 2 n + 1tests. This strategy results in testing more values of that
are close to 0 and, more important, tests less very small
or very large values of that hardly give reasonable re-
sults.
8/2/2019 Optic Flow in Harmony
12/21
Int J Comput Vis (2011) 93: 368388 379
4 Implementation
The solution of the Euler-Lagrange equations for our method
comes down to solving a nonlinear system of equations. We
solve the system by a nonlinear multigrid scheme based on
a Gau-Seidel type solver with alternating line relaxation
(Bruhn et al. 2006).
4.1 Warping Strategy for Large Displacements
The derivation of the optic flow constraint (4) by means of
a linearisation is only valid under the assumption of small
displacements. If the temporal sampling of the image se-
quence is too coarse, this precondition will be violated and
a linearised approach fails. To overcome this problem, Brox
et al. (2004) proposed a coarse-to-fine multiscale warping
strategy. To obtain a coarse representation of the problem,
we downsample the input images by a factor [0.5, 1.0).Prior to downsampling, we apply a low-pass filter to the im-
ages by performing a Gaussian convolution with standard
deviation
2/(4). This prevents aliasing problems.
At each warping level, we split the flow field into an
already computed solution from coarser levels and an un-
known flow increment. As the increments are small, they
can computed by the presented linearised approach. At the
next finer level, the already computed solution serves as ini-
tialisation, which is achieved by performing a motion com-
pensation of the second frame by the current flow, known as
warping. For warping with subpixel precision we again use
Coons patches based on bicubic interpolation (Coons 1967).
Adapting the Smoothness Weight to the Warping Level The
influence of the data term usually becomes smaller at coarser
levels of our multiscale framework. This is due to the
smoothing properties of the downsampling that leads to
smaller values of the image gradients at coarse levels. Such
a behaviour is in fact desirable as the data term might not
be reliable at coarse levels. Our proposed data term normal-
isation leads, however, to image gradients that are approxi-
mately in the same range at each level. To recover the previ-
ous reduction of the data term at coarse levels, we propose
to adapt the smoothness weight to the warping level k.
This is achieved by setting (k) = /k which results inlarger values of and an emphasis of the smoothness term
at coarse levels.
4.2 Discretisation
We follow Bruhn et al. (2006) for the discretisation of the
Euler-Lagrange equations. The images and the flow fields
are sampled on a rectangular pixel grid with grid size h and
temporal step size .
Spatial image derivatives are approximated via central fi-
nite differences using the stencil 112h
(1, 8, 0, 8, 1), re-sulting in a fourth order approximation. The spatial flow
derivatives are discretised by second order approximations
with the stencil 12h
(1, 0, 1). For approximating temporalimage derivatives we use a two-point stencil (1, 1), result-ing in a temporal difference. Concerning the temporal flow
derivatives that occur in the spatio-temporal case, we use thestencil (1, 1)/. Here, it makes sense to adapt the value of to the given image sequence to allow for an appropriate
scaling of the temporal direction compared to the spatial di-
rections (Weickert and Schnrr 2001b).
When computing the motion tensor, occurring derivatives
are averaged from the two frames at time t and t + 1 toobtain a lower approximation error. For the regularisation
tensor, the derivatives are solely computed at the first frame
as we only want to consider directional information from the
reference image.
5 Experiments
In our experiments we show the benefits of the OFH ap-
proach. The first experiments are concerned with our robust
data term and the complementary smoothness term in the
spatial and the spatio-temporal domain. Then, we turn to the
automatic selection of the smoothness weight. After a small
experiment on the importance of anti-aliasing in the warping
scheme, we finish our experiments by presenting the perfor-
mance at the Middlebury optic flow benchmark (Baker et al.
2009, http://vision.middlebury.edu/flow/eval/).
As all considered sequences exhibit relatively large dis-placements, we use the multiscale warping approach de-
scribed in Sect. 4.1. The flow fields are visualised by a
colour code where hue encodes the flow direction and
brightness the magnitude, see Fig. 3 (d). If not said oth-
erwise, we use the following constant parameters: = 0.5, = 20.0, = 4.0, = 0.95, = 0.1, = 0.001, = 0.1.
5.1 Robust Data Term
Benefits of Normalisation and the HSV Colour Space We
proposed two main innovations in the data term: constraint
normalisation and using an HSV colour representation. In
our first experiment, we thus compare our method against
variants (i) without data term normalisation, and (ii) using
the RGB instead of the HSV colour space. For the latter
we only separately robustify the brightness and the gradi-
ent constancy assumption, as a separate robustification of the
RGB channels makes no sense. In Fig. 3 we show the results
for the Snail sequence that we have created. Note that it is a
rather challenging sequence due to severe shadows and large
displacements up to 25 pixels. When comparing the results
http://vision.middlebury.edu/flow/eval/http://vision.middlebury.edu/flow/eval/8/2/2019 Optic Flow in Harmony
13/21
380 Int J Comput Vis (2011) 93: 368388
Fig. 3 Results for our Snail sequence with different variants of our
method. First row, from left to right: (a) First frame. (b) Zoom in
marked region of first frame. (c) Same for second frame. Second row,
from left to right: (d) Colour code. (e) Flow field in marked region,
without normalisation ( = 5000.0). (f) Same for RGB colour space(
=300.0). Third row, from left to right: (g) Same for TV regulari-
sation ( = 50.0). (h) Same for image- and flow-driven regularisation(Sun et al. 2008) ( = 2000.0). (i) Same for our method ( = 2000.0)
to our result in Fig. 3 (i), the following drawbacks of the
modified versions become obvious: Without data term nor-
malisation (Fig. 3 (e)), unpleasant artefacts at image edges
arise, even when using a large smoothness weight . When
relying on the RGB colour space (Fig. 3 (f)), a phantom mo-
tion in the shadow region at the right border is estimated.
To further substantiate these observations, we compare
the discussed variants to our proposed method on some Mid-
dlebury training sequences. The results of our method are
shown together with the ground truth flow fields in Fig. 4.
To evaluate the quality of the flow fields compared to the
given ground truth, we use the average angular error (AAE)
8/2/2019 Optic Flow in Harmony
14/21
Int J Comput Vis (2011) 93: 368388 381
Fig. 4 Results for some Middlebury sequences with ground truth.
First column: Reference frame. Second column: Ground truth (white
pixels mark locations where no ground truth is given). Third col-
umn: Result with our method. From top to bottom: Rubberwhale
( = 3500.0, = 0.3, = 20.0, = 1.0, = 0.01 = AAE = 2.38,
AEE = 0.076), Dimetrodon ( = 8000.0, = 0.7, = 25.0, = 1.5, = 0.01 = AAE = 1.39, AEE = 0.073), Grove3 ( = 20.0, =0.5, = 0.2, = 1.0, = 0.1 = AAE = 4.87, AEE = 0.487),and Urban3 ( = 75.0, = 0.5, = 1.0, = 1.5, = 0.1 =AAE = 2.77, AEE = 0.298)
measure (Barron et al. 1994). To ease comparison with other
methods, we also give the alternative average endpoint er-
ror (AEE) measure (Baker et al. 2009). The resulting AAE
measures are listed in Table 2. Compared to our proposed
method, the results without normalisation are always signifi-
cantly worse. When using the RGB colour space, the results
are mostly inferior, apart from the Rubberwhale sequence.
A deeper discussion on the assets and drawbacks of the RGB
and the HSV colour space can be found in Sect. 5.5. Addi-
tionally, Table 2 also lists results obtained with greyscale im-
ages. These are, however, always worse compared to RGB
or HSV results.
Effect of the Separate Robust Penalisation This experi-
ment illustrates the desirable effect of our separate robust
penalisation of the HSV channels. Using the Rubberwhale
sequence from the Middlebury database, we show in Fig. 5
8/2/2019 Optic Flow in Harmony
15/21
382 Int J Comput Vis (2011) 93: 368388
Table 2 Comparing different variants of our method (using the AAE).
In the Sun et al. regulariser, we use the Charbonnier et al. (1994) pe-
naliser function as this gives better results here
Sequence Rubber- Dime- Grove3 Urban3
whale trodon
Proposed 2.38 1.39 4.87 2.77
No normalisation 2.89 1.45 5.72 4.88RGB colour space 2.34 1.44 5.14 2.97
Greyscale images 2.61 1.59 5.35 3.21
TV regulariser 2.54 1.39 5.09 3.27
Sun et al. regulariser 2.58 1.40 5.27 3.53
CLG 2.44 1.42 4.86 2.72
Fig. 5 Effect of our separate robust penalisation of the HSV channels.
First row, from left to right: (a) Zoom in first frame of the Rubberwhale
sequence. (b) Visualisation of the corresponding hue channel weight.Brighter pixels correspond to a larger weight. Second row, from left
to right: (c) Same for the saturation channel. (d) Same for the value
channel
the data term weights M(wJ i0 w) for the brightness con-
stancy assumption on the hue, the saturation and the value
channel (i = 1, . . . , 3). Here, brighter pixels correspond to alarger weight and we only show a zoom for better visibility.
As we can see, the weight of the value channel is reduced
in the shadow regions (left of the wheel, of the orange toy
and of the clam). This is desirable as the value channel is not
invariant under shadows, see Fig. 2.
A CLG Variant of Our Method Our next experiment is con-
cerned with a CLG variant of our data term where we, as for
the regularisation tensor, perform a Gaussian convolution
of the motion tensor entries. First, we compare our method
against a CLG variant for some Middlebury sequences, see
Table 2. We find that the CLG variant may only lead to small
improvements. On the other hand, it may also deteriorate
the results. We thus conclude that for the considered Mid-
Table 3 Comparison of our method to a CLG variant on noisy ver-
sions of the Yosemite sequence (using the AAE). We have added
Gaussian noise with zero mean and standard deviationn
n 0 10 20 40
CLG 1.75 4.01 5.82 8.14
Proposed 1.64 3.82 6.08 8.55
dlebury test sequences, this modification seems not too use-
ful.
However, a CLG variant of our method can actually be
useful in the presence of severe noise in the image sequence.
To prove this, we compare in Table 3 the performance of
our method to its CLG counterpart on noisy versions of
the Yosemite sequence. As it turns out, the CLG variant im-
proves the results at large noise scales, but deteriorates the
quality for low noise scenarios. This also explains the expe-
rienced behaviour on the Middlebury data sets, which hardly
suffer from noise.
5.2 Complementary Smoothness Term
Comparison with Other Regularisers. In Fig. 3, we com-
pare our method against two results obtained when using
another regulariser in conjunction with our robust data term:
(i) Using the popular TV regulariser; see (40) and (42). (ii)
Using the anisotropic image and flow-driven regulariser of
Sun et al. (2008), which built the basis of our complemen-
tary regulariser. Here, we use a rotationally invariant formu-
lation that can be obtained from (49) by replacing the eigen-vectors ri of the regularisation tensor by the eigenvectors siof the structure tensor. Comparing the obtained results to our
result in Fig. 3 (i), we see that TV regularisation (Fig. 3 (g)),
leads to blurred and badly localised flow edges. Using the
regulariser of Sun et al. (2008) (Fig. 3 (h)), unpleasant stair-
casing artefacts deteriorate the result.
As for the data term experiments, we also applied our
method with different regularisers to some Middlebury se-
quences. Considering the corresponding error measures in
Table 2, we see that the alternative regularisation strategies
mostly lead to higher errors. Only for the Dimetrodon se-
quence, the TV regulariser is en par with our result. Ourflow fields in Fig. 4 in general closely resemble the ground
truth. However, minor problems arising from incorporating
image information in the smoothness term are visible at a
few locations, e.g. at the wheel in the Rubberwhale result
and at the bottom of the Urban 3 result. Furthermore, some
tiny flow details are smoothed out, e.g. at the leafs in the
Grove 3 result. As recently shown by Xu et al. (2010), the
latter problem could be resolved by using a more sophisti-
cated warping strategy.
8/2/2019 Optic Flow in Harmony
16/21
Int J Comput Vis (2011) 93: 368388 383
Fig. 6 Results for the Marble sequence with our spatio-temporal
method. First row, from left to right: (a) Reference frame (frame 16).
(b) Ground truth (white pixels mark locations where no ground truth
is available). Second row, from left to right: (c) Result using 2 frames
(1617). (d) Same for 6 frames (1419)
Table 4 Smoothness weight and AAE measures for our spatio-
temporal method on the Marble sequence, see Fig. 6. All results used
the fixed parameters = 0.5, = 0.5, = 1.0, = 1.5, = 0.5.When using more than two frames, the convolutions withK and Kare performed in the spatio-temporal domain
Number of frames 2 4 6 8
(from to) (1617) (1518) (1419) (1320)
Smoothn. weight 75.0 50.0 50.0 50.0
AAE 4.85 2.63 1.86 2.04
Optic Flow in the Spatio-Temporal Domain Let us now
turn to the spatio-temporal extension of our complementary
smoothness term. As most Middlebury sequences consist
of 8 frames, a spatio-temporal method would in general be
applicable. However, the displacements between two subse-
quent frames are often rather large there, resulting in a vio-
lation of the assumption of a temporally smooth flow field.
Consequently, spatio-temporal methods do not improve the
results. In our experiments, we use the Marble sequence
(available at http://i21www.ira.uka.de/image_sequences/)
and the Yosemite sequence from the Middlebury datasets.
These sequences exhibit relatively small displacements and
our spatio-temporal method allows to obtain notably better
results, see Fig. 6 and Tables 4 and 5. Note that when using
more than two frames, a smaller smoothness weight has
to be chosen and that a too large temporal window may also
deteriorate the results again.
Table 5 Smoothness weight and AAE measures for our spatio-
temporal method on the Yosemite sequence from Middlebury. All re-
sults used the fixed parameters = 1.0, = 20.0, = 1.5, = 1.0, = 0.5. Here, better results could be obtained when disabling the tem-poral presmoothing
Number of frames 2 4 6 8
(from to) (1011) (912) (813) (714)
Smoothn. weight 2000.0 1000.0 1000.0 1000.0
AAE 1.65 1.16 1.05 1.01
Fig. 7 Automatic selection of the smoothness weight at the Grove2
sequence. From left to right: (a) Angular error (AAE) for 51 values of
, computed from 0 = 400.0 and a stepping factor a = 1.1. (b) Samefor the proposed data constancy error (ADCE1,3)
Table 6 Results (AAE) for some Middlebury sequences when (i) fix-
ing the smoothness weight ( = 400.0), (ii) estimating and (iii) withthe optimal value of
Sequence Fixed Estimated Optimal
Rubberwhale 3.43 3.00 3.00
Grove2 2.59 2.43 2.43
Grove3 5.50 5.62 5.50
Urban2 3.22 2.84 2.66
Urban3 3.44 3.37 3.35
Hydrangea 1.96 1.94 1.86
Yosemite 2.56 1.89 1.71
Marble 5.73 5.05 4.94
5.3 Automatic Selection of the Smoothness Weight
Performance of our Proposed Error Measure We firstshow that our proposed data constancy error between frame
1 and 3 (ADCE1,3) is a very good approximation of the pop-
ular angular error (AAE) measure. To this end, we com-
pare the two error measures for the Grove2 sequence in
Fig. 7. It becomes obvious that our proposed error measure
(Fig. 7 (b)) indeed exhibits a shape very close to the angular
error shown in Fig. 7 (a). As our error measure reflects the
quality of the prediction with the given flow field, our result
further substantiate the validity of the proposed OPP.
http://i21www.ira.uka.de/image_sequences/http://i21www.ira.uka.de/image_sequences/8/2/2019 Optic Flow in Harmony
17/21
384 Int J Comput Vis (2011) 93: 368388
Benefits of an Automatic Parameter Selection Next, we
show that our automatic parameter selection works well for
a large variety of different test sequences. In Table 6, we
summarise the AAE obtained when (i) setting to a fixed
value ( = 0 = 400.0), (ii) using our automatic parame-ter selection method, and (iii) selecting the (w.r.t. the AAE)
optimal value of under the tested proposals. It turns out
that estimating with our proposed method allows to im-prove the results compared to a fixed value of in almost
all cases. Just for the Grove 3 sequence, the fixed value of
accidentally coincides with the optimal value. Compared to
the results achieved with an optimal value of , our results
are on average 3% and at most 10% worse than the optimal
result.
We wish to note that a favourable performance of the pro-
posed parameter selection of course depends on the validity
of our assumptions (constant speed and linear trajectories
of objects). For all considered test sequences these assump-
tions were obviously valid. In order to show the limitations
of our method, we applied our approach again to the samesequences, but replaced the third frame by a copy of the sec-
ond one. This violates the assumption of a constant speed
and leads to a severe impairment of results (up to 30%).
5.4 Importance of Anti-Aliasing in the Warping Scheme
We proposed to presmooth the images prior to downsam-
pling in order to avoid aliasing problems. In most cases, the
resulting artefacts will not significantly deteriorate the flow
estimation, which can be attributed to the robust data term.
However, for the Urban sequence from the official Middle-
bury benchmark, anti-aliasing is crucial for obtaining rea-sonable results, see Fig. 8. As it turns out, the large displace-
ment of the building in the lower left corner can only be esti-
mated when using anti-aliasing. We explain this by the high
frequent stripe pattern on the facade of the building.
5.5 Comparison to State-of-the-Art Methods
To compare our method to the state-of-the-art in optic flow
estimation, we submitted our results to the popular Middle-
bury benchmark (http://vision.middlebury.edu/flow/eval/).
We found that for the provided benchmark sequences,
using a HSV colour representation is not as beneficial as
seen in our experiment from Fig. 3. As the Middlebury se-
quences hardly suffer from difficult illumination conditions,
we cannot profit from the photometric invariances of the
HSV colour space. On the other hand, some sequences even
pose problems in their HSV representation. As an exam-
ple, consider the results for the Teddy sequence in the first
row of Fig. 9. Here we see that the small white triangle be-
neath the chimney causes unpleasant artefacts in the flow
field. This results from the problem that greyscales do not
Fig. 8 Importance of anti-aliasing on the example of the Urban se-
quence. Top row, from left to right: (a) Frame 10. (b) Frame 11. Bot-
tom row, from left to right: (c) Our result without anti-aliasing. (d)
Same with anti-aliasing. Note that for this sequence, no ground truth is
publicly available
have a unique representation in the hue as well as the satura-
tion channel. Nevertheless, there are also sequences where
a HSV colour representation is beneficial. For the Mequon
sequence (second row of Fig. 9) a HSV colour representa-
tion removes artefacts in the shadows left of the toys. The
bottom line is, however, that for the whole set of benchmark
sequences, we obtain slightly better results when using the
RGB colour space. Thus, we use this variant of our method
for evaluation at the Middlebury benchmark.
For our submission we used in accordance to the guide-
lines a fixed set of parameters:
=0.5,
=20.0,
=4.0, = 0.95, = 0.1, = 0.1. The smoothness weight was automatically determined by our proposed method
with the settings n = 8, 0 = 400.0, a = 1.2. For theTeddy sequence with only two frames, we set = 0, asour parameter estimation method is not applicable in this
case. The resulting running time for the Urban sequence
(640 480 pixels) was 620s on a standard PC (3.2 GHzIntel Pentium 4). For the parameter selection we com-
puted 2 8 + 1 = 17 flow fields, corresponding to approx-imately 36 s per flow field. Following the trend of paral-
lel implementations on modern GPUs (Zach et al. 2007;
Grossauer and Thoman 2008; Sundaram et al. 2010; Werl-
berger et al. 2010) we recently came up with a GPU version
of our method that can compute flow fields of size 640 480pixels in less than one second (Gwosdek et al. 2010).
At the time of submission (August 2010), we achieve the
4th place w.r.t. the AAE and the AEE measure among 39
listed methods. Note that our previous Complementary Op-
tic Flow method (Zimmer et al. 2009) only ranks 6th for the
AAE and 9th for the AEE, which demonstrates the benefits
of the proposed novelties in this paper, like the automatic
parameter selection and the anti-aliasing.
http://vision.middlebury.edu/flow/eval/http://vision.middlebury.edu/flow/eval/8/2/2019 Optic Flow in Harmony
18/21
Int J Comput Vis (2011) 93: 368388 385
Fig. 9 Comparison of results obtained with HSV or RGB colour representation.First row, from left to right: (a) Frame 10 of the Teddy sequence.
(b) Frame 11. (c) Result when using the HSV colour space (AAE = 3.94). (d) Same for the RGB colour space (AAE = 2.64). Second row, fromleft to right: (e) Frame 10 of the Mequon sequence. (f) Frame 11. (g) Result when using the HSV colour space (AAE = 2.28). (h) Same for the
RGB colour space (AAE = 2.84)
Table 7 Estimated values of the smoothness weight , using our au-
tomatic parameter selection method
Sequence Smoothness weight
Army 277.8
Grove 277.8
Mequon 277.8
Schefflera 691.2
Urban 480.0
Wooden 995.3Yosemite 1433.3
In Table 7 we additionally summarise the estimated val-
ues of resulting from our automatic parameter selection
method. As desired, for sequences with small details in
the flow field (Army, Grove, Mequon) a small smoothness
weight is chosen. On the other hand, sequences like Wooden
and Yosemite with a rather smooth flow yield significantly
larger values for the smoothness weight.
6 Conclusions and Outlook
In this paper we have shown how to harmonise the three
main constituents of variational optic flow approaches: the
data term, the smoothness term and the smoothness weight.
This was achieved by two main ideas: (i) We developed a
smoothness term that achieves an optimal complementary
smoothing behaviour w.r.t. the imposed data constraints. (ii)
We presented a simple, yet well performing method for de-
termining the optimal smoothness weight for the given im-
age sequence. To this end, we came up with a novel para-
digm, the optimal prediction principle (OPP).
Our optic flow in harmony (OFH) method bases on an
advanced data term that combines and extended success-
ful concepts like normalisation, photometric invariant colour
representation, higher order constancy assumptions and ro-
bust penalisation. The anisotropic complementary smooth-
ness term incorporates directional information from the mo-
tion tensor occurring in the data term. The smoothing in data
constraint direction is reduced to avoid interference withthe data term, while a strong smoothing in the orthogonal
direction allows to fill-in missing information. This yields
an optimal complementary between both terms. Further-
more, our smoothness term unifies the benefits of image-
and flow-driven regularisers, resulting in sharp flow edges
without oversegmentation artefacts. The proposed parame-
ter selection method bases on the OPP that we introduced
in this paper. It states the flow field obtained with an opti-
mal smoothness weight allows for the best prediction of the
next frames in the image sequence. Under some assump-
tions, the quality of the prediction can be judged by evaluat-
ing the data constraints between first and third frame of thesequence and using the doubled flow vectors. Due to its sim-
plicity, our method can easily be used in all variational optic
flow approaches and additionally gives surprisingly good re-
sults.
The benefits of the OFH idea are demonstrated by our
extensive experimental validation and the competitive per-
formance at the Middlebury optic flow benchmark. Our pa-
per thus shows that a careful design of data and smoothness
term together with an automatic choice of the smoothness
8/2/2019 Optic Flow in Harmony
19/21
386 Int J Comput Vis (2011) 93: 368388
weight allows to outperform other well-engineered methods
that incorporate many more processing steps, e.g. segmen-
tation (Lei and Yang 2009), or the integration of an epipo-
lar geometry prior (Wedel et al. 2009). We thus hope that
our work will give rise to more harmonised approaches in
other fields where energy-based methods are used, e.g. im-
age registration.
Our current research is on the one hand concerned withinvestigating possibilities to small issues with the proposed
regularisation strategy; see our remarks in Sect. 5.2. Further-
more, we are interested in extensions of our novel parameter
selection approach. Here, we focus on higher efficiency and
ways to apply our method in cases where only two images
are available. The latter would also remedy the problems
with non-constant speed of objects that we have mentioned
in Sect. 5.3.
Acknowledgements The authors gratefully acknowledge partial
funding by the International Max-Planck Research School (IMPRS)
and the Cluster of Excellence Multimodal Computing and Interac-
tion within the Excellence Initiative of the German Federal Govern-
ment.
Appendix: Derivation of the Diffusion Tensor for the
Complementary Regulariser
Consider the complementary regulariser from (50):
V (2u, 2v) = V
u2r1 + v2r1+ u2r2 + v2r2 . (57)
Its contributions to the Euler-Lagrange equations are given
by
x
ux V+ y uy V, (58)
and
x
vx V+ y vy V, (59)
respectively. We exemplify the computation of the first ex-
pression (58). The second one then follows analogously. Let
us first define the abbreviations
ri:=
(ri1, ri2) and V(ri )
:=Vu2ri + v2ri , (60)
for i = 1, . . . , 2. With their help, we compute the expres-sions
ux V = 2
V(r1) ur1 r11 + ur2 r21
, (61)
uy V = 2
V(r1) ur1 r12 + ur2 r22
(62)
from (58). Using the fact that
x
ux V+ y uy V= div(ux V , uy V ), (63)
we obtain by plugging (61) and (62) into (58):
x
ux V+ y
uy V
= 2div
V(r1) ur1 r11 + ur2 r21 V(r1) ur1 r12 + ur2 r22
.
(64)
By multiplying out the expressions inside the divergence ex-pressions one ends up with
x
ux V+ y uy V
= 2div
( V
(r1)r211
+ r221)ux + ( V(r1)r11r12 + r21r22)uy(
V(r1)r11r12 + r21r22)ux + ( V(r1)r212 + r222)uy
.
(65)
We can write above equation in diffusion tensor notation as
x
ux V+ y uy V= 2div(D 2u), (66)
with the diffusion tensor
D :=
V
(r1) r211 + 1 r221 V(r1) r11r12 + 1 r21r22
V(r1) r11r12 + 1 r21r22 V(r1) r212 + 1 r222
.
(67)
We multiplied the second term of each sum by a factor of
1 to clarify that the eigenvalues of D are V(r1) and 1, re-spectively. The corresponding eigenvectors are r1 and r2,
respectively, which allows to rewrite the tensor D as
D =
r11 r21r12 r22
V(r1) r11 V(r1) r12
1
r21 1
r22
= (r1 | r2)
V(r1) 00 1
r1r2
. (68)
This shows that D is identical to DCR from (51), as it can be
written as
D = V(r1) r1 r1 + r2 r2 = V
u2r1 + v2r1
r1 r1 + r2 r2 .
(69)
References
Alvarez, L., Esclarn, J., Lefbure, M., & Snchez, J. (1999). A PDE
model for computing the optical flow. In Proc. XVI congreso de
ecuaciones diferenciales y aplicaciones (pp. 13491356). Las Pal-
mas de Gran Canaria, Spain.
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski,
R. (2009). A database and evaluation methodology for optical
flow. Tech. Rep. MSR-TR-2009-179, Microsoft Research, Red-
mond, WA.
Barron, J. L., Fleet, D. J., & Beauchemin, S. S. (1994). Performance
of optical flow techniques. International Journal of Computer Vi-
sion, 12(1), 4377.
8/2/2019 Optic Flow in Harmony
20/21
Int J Comput Vis (2011) 93: 368388 387
Bertero, M., Poggio, TA, & Torre, V. (1988). Ill-posed problems in
early vision. Proceedings of the IEEE, 76(8), 869889.
Bign, J., Granlund, G. H., & Wiklund, J. (1991). Multidimensional
orientation estimation with applications to texture analysis and
optical flow. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 13(8), 775790.
Black, M. J., & Anandan, P. (1996). The robust estimation of multiple
motions: parametric and piecewise smooth flow fields. Computer
Vision and Image Understanding, 63(1), 75104.
Blake, A., & Zisserman, A. (1987). Visual reconstruction. Cambridge:
MIT Press.
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accu-
racy optical flow estimation based on a theory for warping. In T.
Pajdla & J. Matas (Eds.), Computer visionECCV 2004, Part IV.
Lecture notes in computer science (Vol. 3024, pp. 2536). Berlin:
Springer.
Bruhn, A., & Weickert, J. (2005). Towards ultimate motion estima-
tion: combining highest accuracy with real-time performance. In
Proc. tenth international conference on computer vision (Vol. 1,
pp. 749755). Beijing: IEEE Computer Society Press.
Bruhn, A., Weickert, J., & Schnrr, C. (2005). Lucas/Kanade meets
Horn/Schunck: combining local and global optic flow methods.
International Journal of Computer Vision, 61(3), 211231.
Bruhn, A., Weickert, J., Kohlberger, T., & Schnrr, C. (2006).A multigrid platform for real-time motion computation with
discontinuity-preserving variational methods.International Jour-
nal of Computer Vision, 70(3), 257277.
Charbonnier, P., Blanc-Fraud, L., Aubert, G., & Barlaud, M. (1994).
Two deterministic half-quadratic regularization algorithms for
computed imaging. In Proc. IEEE international conference on
image processing (Vol. 2, pp. 168172). Austin: IEEE Computer
Society Press.
Cohen, I. (1993). Nonlinear variational method for optical flow compu-
tation. In Proc. eighth Scandinavian conference on image analysis
(Vol. 1, pp. 523530). Norway, Troms.
Coons, S. A. (1967). Surfaces for computer aided design of space
forms (Tech. Rep. MIT/LCS/TR-41). Massachusetts Institute of
Technology, Cambridge.
Elsgolc, L. E. (1962). Calculus of variations. London: Pergamon.Fleet, D. J., & Jepson, A. D. (1990). Computation of component image
velocity from local phase information. International Journal of
Computer Vision, 5(1), 77104.
Frstner, W., & Glch, E. (1987). A fast operator for detection and
precise location of distinct points, corners and centres of circu-
lar features. In Proc. ISPRS intercommission conference on fast
processing of photogrammetric data (pp. 281305). Interlaken,
Switzerland.
Golland, P., & Bruckstein, A. M. (1997). Motion from color.Computer
Vision and Image Understanding, 68(3), 346362.
Grossauer, H., & Thoman, P. (2008). GPU-based multigrid: real-time
performance in high resolution nonlinear image processing. In
A. Gasteratos, M. Vincze, & J. K. Tsotsos (Eds.), Lecture notes in
computer science: Vol. 5008. Computer vision systems (pp. 141
150). Berlin: Springer.
Gwosdek, P., Zimmer, H., Grewenig, S., Bruhn, A., & Weickert, J.
(2010). A highly efficient GPU implementation for variational
optic flow based on the Euler-Lagrange framework. In Proc.
2010 ECCV workshop on computer vision with GPUs, Heraklion,
Greece.
Horn, B., & Schunck, B. (1981). Determining optical flow.Artificial
Intelligence, 17, 185203.
Krajsek, K., & Mester, R. (2007). Bayesian model selection for opti-
cal flow estimation. In F. A. Hamprecht, C. Schnrr, & B. Jhne
(Eds.), Pattern recognition. Lecture notes in computer science
(pp. 142151). Berlin: Springer.
Lai, S. H., & Vemuri, B. C. (1998). Reliable and efficient computation
of optical flow. International Journal of Computer Vision, 29(2),
87105.
Lei, C., & Yang, Y. H. (2009). Optical flow estimation on coarse-to-
fine region-trees using discrete optimization. InProc. 2009 IEEE
international conference on computer vision. Kyoto: IEEE Com-
puter Society Press.
Lucas, B., & Kanade, T. (1981). An iterative image registration tech-