Omnidirectional Vision and Inertial Cluesfor Robot Navigation

Irem Stratmann Erik Solda

Autonomous Intelligent Systems Autonomous Intelligent SystemsFraunhofer Institute Fraunhofer Institute

Schloss Birlinghoven, D-53754, Germany Schloss Birlinghoven, D-53754, GermanyIrem.Stratmann@ais.fraunhofer.de Erik.Solda@ais.fraunhofer.de

Abstract

The features inherent in the visual motion fieldof a mobile robot indicate important clues about itsnavigation and environment. Combining these visualclues with additional inertial sensor information al-lows reliable detection of navigation direction for amobile robot and the independent motion which maybe present in the scene. The motion field, which is the2D projection of the 3D scene variations induced bythe camera-robot system, is estimated through opticalflow calculations. The singular points of the global op-tical flow field of omnidirectional image sequences in-dicate the translation direction of the robot as well asthe deviation from its planned path. It is also possibleto detect motion patterns of near obstacles or indepen-dently moving objects of the scene. In this paper, afterreviewing the image velocity measurement techniquesshortly, we introduce the analysis of the intrinsic fea-tures of the omnidirectional motion fields for motiondetection, giving some preliminary examples of thisanalysis.

1 Introduction

Robot navigation requires relevant motion sensingmechanisms, specially in the case of a dynamical inter-action with the environment. Tasks like object avoid-ance or path finding rely on the estimation of the egomotion and the motion of the environment. Expandedfield of view, provided by omnidirectional vision sen-sors, enables visual motion detection mechanisms sim-ilar to some biological species like insects. The role ofwide angle view and optical flow in insect navigationhas been researched in the past [6]. The quintessenceof this research is that bees and other insects rely onthe basic properties of an estimated visual motion fieldfor navigation. Some robotic applications have fol-lowed the insights gained on this topic [4, 9]. A simi-lar motivation has given many researchers an impulseto look for methods to gain preliminary navigational

information using catadioptric sensors. These sensorsare constructed using a curved mirror combined witha vision sensor. Different types of curves have beenapplied for the mirror design, e.g., parabolic, hyper-bolic. Special curves preserving spatial features of theprojection like range and angle have been introduced,too. (For a review on panoramic imaging techniquesand mirror design see [2]) The motion field, whichis captured using a wide angle panoramic sensor, in-troduces significant structural features, like vanishingand emerging points of velocity vectors, that indicatethe ego motion direction. Also, dense flow fields con-tain patterns of different motion regions, which mayindicate independently moving obstacles. Our aim isto combine this structural information with standardinertial information as provided by a gyroscope andexamine their usefulness in indoor navigation of a mo-bile camera-robot system. This paper is organized asfollows: section 2 discusses the inherent features ofomnidirectional motion fields achieved by the opticalflow estimation. Section 3 shortly reviews the opticalflow estimation techniques and introduces methods forego- and independent motion detection using the om-nidirectional optical flow fields and gyroscope. Thelast section (4) describes the preliminary experiments.

2 Inherent features of omnidirectionaloptical flow fields

In the spherical views of a 3D scene, the Focusof Expansion (FOE), the points on the motion fieldwhere the flow vectors seem to be emerging, and theFocus of Contraction (FOC), the points where the flowvectors are vanishing, are always in the field of viewand span an angle of 180, if the camera motion ispurely translatory. By pure rotational motion neitherFOE nor FOC are in the field of view. In real roboticapplications, though, there is mostly a translatorycomponent in the motion field. A rotational compo-nent in a translational motion field causes the relativepositions of FOE and FOC to vary (see Fig.1). Nelson

(a) (b) (c)

Figure 1: (a) FOE and FOC lines by pure transla-tional motion of the camera system. (b) relative posi-tions of both lines change as the camera system followsa curvilinear path with a rotation to the left side. (c)rotational movement to the right side.

and Aloimonos [8] have proposed to use the sphericalprojections of a scene to calculate the translationaland rotational components of the motion fields witha qualitative analysis of flow fields.

Similar features are also present in the flow field ofomnidirectional scenes captured by catadioptric cam-eras. Translational or curvilinear motion of the robot-camera system induces global flow fields with singularpoints on the translational direction. This fact en-ables determining the navigation direction, which canconstitute a prediction for an inertial measurementmodel. Additionally, the regions with significant highflow values in the dense motion field can be detectedby a simple pattern analysis. These values are mostlycaused by very near stable or independently movingobjects.

3 Optical Flow

Projected relative motion of one image pixel can berepresented as a vector V(vx, vy)and can be evaluatedfrom the analysis of the instantaneous changes in thebrightness values at this pixel point (x, y). For thecalculation of the optical flow field, one may assumethat intensity is conserved throughout the image andthe only reason for the brightness changes is the rel-ative motion. Given a brightness function I(x, y, t)at a pixel position (x, y) and time t, this brightnessconservation condition (BCC)([5]) can be formulatedas:

I(x, y, t) = I(x + vxt, y + vyt, 0) (1)

where vx und vy define the components of the mo-tion velocity in x and y directions respectively. As-suming that the image intensities of the scene pointsare preserved, the motion field can be estimated byusing the spatiotemporal differentiation of the imageintensity function I(x, y, t). The classical formulationof this method given by [5] (shown below), dependson two assumptions. The first one, conservation ofintensities, has already been discussed above. The

next assumption is the smoothness of the motion ingiven scene points, which means that the neighboringpoints on the image move with the same velocity. Thislast assumption assures the differentiability of the im-age intensity signal I(x, y, t). Differentiating I(x, y, t)with respect to t and assuming that this is zero, yieldsthe following equation:

dI

dt=

∂I

∂x

dx

dt+

∂I

∂y

dy

dt+

∂I

∂t= 0 (2)

Partial differences of I ( ∂I∂x and ∂I

∂y ) indicate the mo-tion components in x, y directions respectively. Theywill be represented with vx and vy.

From the above equation it is clear that only thenormal velocity component, that is the image velocityin the direction of the image gradient, can be solvedfrom the BCC alone (1). This is also known as theaperture problem. To solve this problem and calculatethe actual velocity vectors, one needs to consider addi-tional information about the motion, e.g. smoothness,occlusion or disocclusion etc. Depending on the as-sumptions made on the features of the global motion,many methods for the optical flow calculation havebeen proposed. (For a comprehensive review of themethods see [1]). A popular method that can be foundin many implementations is the method proposed byLucas and Kanade [7]. It is based on a Linear LeastSquares Estimate fit of the normal flow vectors to aconstant velocity model v over a small neighborhoodΩ. It requires the minimization of the function:

∑

x∈Ω

ω(x) [∇I(x) · v + It(x)]2 (3)

where ω(x) in Ω are set to give more importance tothe pixels in the center of the window than at its pe-riphery. ∇ indicate the gradient operator applied onto I and It represents the temporal derivative of theintensity function. This method seems to be more ap-propriate for the applications that have to cope withobject deformations.(For a qualitative comparison ofthe different methods see ([10]) ) The robustness of themethod results from the assumption of a local smooth-ness of the motion in a small area of the image ratherthan a global smoothness. It is also inexpensive inthe sense that only five convolutions over the spatialneighborhood Ω are needed to compute the terms in(3).

3.1 Determining the FOE and FOC PositionsSmooth changes of the omnidirectional scenes as

provided by the catadioptric sensors, cause harmon-ical distributions of the optical flow vectors in thedense flow field. The histogram of these flow fieldvectors captured in the angular direction, vary in sig-nificant patterns as the mobile visual system moves in

translational or curvilinear paths. The analysis of thesingular points of this histogram, which resembles asine-like distribution, allows the detection of the direc-tion of the camera-robot motion. The zero-crossingsof the distribution indicate the FOE and FOC. Thesepoints are shifted proportional to the motion direc-tion. The angular distance between two crossings(FOE and FOC) may vary between 180, where theFOE and FOC divide the flow field into two equalparts and indicate a pure translational motion, and0, where there is no distinguishable FOE and FOC,indicating purely rotational motion.

Measurement of the rotational motion should besupported by additional instruments. It can be mea-sured using an odometry instrument, but this ismostly imprecise due to wheel-slippage, or errors thatstem from a navigation on uneven floors. There-fore gyroscopes are widely used to measure rotationalspeed and also absolute angles by integrating sensorreadings over time. During the last years the phys-ical dimensions of the gyroscopes have reduced re-markably, and even more important, they have be-come very cheap compared to former Ring Laser Gy-roscopes (RLG) or Fiber Optical Gyroscopes (FOG).This was achieved by using cheap vibrating piezo ele-ments, which are subject to secondary vibration whenrotated and silicon microstructures consisting of a ringshaped vibrating element that changes its direction ofvibration during rotation. The directional variationsof the original motion can be sensed and the rate ofturn can be evaluated. With these cheap and smallsensors, it is now possible to equip our mobile robot toimprove and to ease the motion analysis with the om-nidirectional camera. Combination of the two differ-ent sensor information was realized using a Kalman-Model which can be outlined as in figure 2. In thismodel we combine the two sensory information, gyro-scope measurement and angular direction estimation,which is expected to be less precise. Local positionand independent motion measurement are not consid-ered to be part of the Kalman-Model yet.

3.2 Detection of independently moving sceneobjects

Independent motion causes distinct changes on theglobal flow pattern of a moving camera-robot system.While the background motion occupies the larger partof the global flow field, independent motion arise asregions of disturbances in this pattern. Static objectswhich are located in the near regions of the mobilerobot may also cause greater flow vectors which differ-entiate in the value but not in the direction of the egomotion. The analysis of distinct regions of the globalflow field enables the detection of such static and/orindependently moving objects. Both cases might beinteresting from the navigational point of view. If the

Figure 2: Flow diagram of the sensory system thatdetects ego- and independent motion on a mobilerobot. θ′: gyroscope measurement, θ′′:angular transla-tion direction evaluated by optical flow field analysis,θest is the estimated navigation direction

major goal is to determine the dynamical objects inthe scene, then it might be necessary to look at thechanges of the flow field over time. One efficient wayto do so is to warp the given image applying the pre-viously estimated flow field. If the general motion ofthe robot is of constant velocity, any object movingwith an independent speed and direction will producea difference between the warped image and the realfollowing image. The region where this warp error ismaximal, indicates independent object motion. Thisprocedure can be outlined as follows:

• calculate the flow field using the first two images

• generate a pseudo image by warping the secondimage using the estimated flow field

• calculate the difference between the third imageand the pseudo image and detect the regions ofmaximum difference

4 Experiments

The image sequences are acquired by means ofa catadioptric sensor with a specially curved mirrorsurface that preserves the linear angular relationshipof incoming rays and their projections on the imageplane (designed as proposed in [3]). It is mounted ona mobile robot that follows translational and curvi-linear paths. In the first case, it is expected that theFOE and FOC vectors have opposite directions, span-ning an angle of 180. As the rotational componentaffects the motion field, it is expected that this angledecreases on one side of the field. Purely rotationalmotion causes the FOC and FOE vectors to vanish.

The analysis of the clockwise (CW) and counter-clockwise (CCW) flow vectors, marked in the images,

(a)

(b)

(c)

Figure 3: (a) In the scene there is only translationalmotion. Red (+) indicate counterclockwise, green (o)indicate clockwise motion (b) only rotational motion(c) curvilinear motion.

render the regions with contradirectional flow fields.The proportions of these fields indicate the positionsof the FOE and FOC. Ideally, this should lead to aunique detection of both of the lines. In figure 3, wepresent the resulting images.

To enable a quantitative and controlled analysis, wehave generated synthetical images distorted with thegiven mirror properties simulating the omnidirectionalcamera at hand. The camera-robot system navigatesin a virtual corridor with a known direction and speed.For the sequence from which the image in figure (4)was taken, the camera moves to the angular directionof 270, beginning from the right horizontal directionand incremented counterclockwise. (The robot navi-gates in the direction of the 6:30 position of an analogclock and the beginning of the angular coordinate isat the 3:15 position) The angular distribution of theflow vectors on the omnidirectional field shows a sine-like pattern (Figure 4(b)). The angular direction ofthe minima of this distribution indicates the motiondirection. In the pure translatory case, these minimaare exactly 180 apart. Deviation from the pure trans-latory motion causes the phase of the sine function tovary. Quantitative analysis of the synthetical imagesequences showed that it is possible to detect the nav-igation direction of the camera-robot system with upto a maximal error of 5 image.

In the real scenes, more accurate results can beachieved by combining and evaluating additional iner-tial information from a gyroscope. The currently usedsensor (ADXR300, Analog Devices Inc.) can measureup to 300 degrees per second, which is sufficient formost applications in mobile robotics where vision isalso applicable. Since the integration from rates toabsolute angles is required, even small errors are ac-cumulated over time and this would lead to large de-viations if no recalibration takes place. This problemis bypassed by only integrating over a short period oftime between two pictures. With our current camerasetup, we’re able to process a frame every 4 secondswhich is sufficient to neglect the error.

4.1 Independent Motion

In Fig.5, two real images of an indoor navigationsequence is shown. Fig.5d, presents the results ofdistinct flow region detection. Note that in Fig.5dthe algorithm detects independent object movementas well as the static near objects. Because the signifi-cant regions on the global flow pattern are also causedby depth changes, differentiation between these twoobject categories cannot be done until applying thewarping analysis, that was outlined above (see section3.2). Warping the image introduces a temporal analy-sis which assumes that the robot ego motion is smoothand independently moving objects differ significantlyin their motion direction and speed. In Fig.5e, the

(a)

0 90 180 270 360−6

−4

−2

0

2

4

6

theta in degrees

r

Distribution of the flow field in angular resolution

(b)

Figure 4: Flow field calculated on synthetical imagesequence with known FOE and FOC positions. The di-rection of the two singular points span exactly a 180

angle. The black line on the image represents the es-timated navigation direction. The images also includethe artificial scene distortion due to the mirror sur-face.

detected silhouette of the person on the right, whois moving independently from the robot-camera sys-tem, is shown. The detection is quite stable if theindependently moving objects are in the near regionof the camera. Distant objects are reduced in theirsize due to the distortions of the catadioptric sensor.Their motion can be detected using a special analysiswhich considers the outermost regions of the omnidi-rectional images, where distant object movement canbe expected, seperately.

Currently it is only possible to detect the angu-lar direction of the independent motion. We will beconsidering additional measurement techniques to de-termine the depth measurement of the objects, whichwill enable determining the exact position of the ob-jects in the scene.

5 Summary and Future Work

Visual information provided by omnidirectionalcameras include features of significant importance fora robot navigating in a dynamical environment. Es-timation of the ego motion direction and detection ofthe objects in the environment can help avoid obsta-cles or plan the navigation path. Such an estimationcan be done by considering the inherent features of theglobal flow fields of omnidirectional image sequencesin combination with standard inertial sensors. Thispaper has introduced techniques for gaining naviga-tional information from visual motion fields of om-nidirectional image sequences. The global structureof those motion fields gives hints about the naviga-tion direction of a robot in translational or curvilinearmotion. The angular position of the singular points ofsuch a field, where the flow vectors seem to be emerg-ing and vanishing ( the Focus Of Expansion and theFocus Of Contraction) indicate the navigation direc-tion. These points are related to the translationalcomponent of the navigation. Refining this analysis,specially for the purely rotational case, has requiredthe use of a gyroscope.

The regions with significant and rapid changes inthe global flow field indicate near obstacles or indepen-dently moving objects in the omnidirectional scene.The motion pattern analysis detects such regions. In-dependently moving objects with different velocityand/or motion direction can be detected by followingthe changes in the motion field in time, since omni-directional scenes allow these objects to track for alonger time and in a larger field of view.

Future Work. Our future work aims to refine theanalysis for detecting independent motion patterns.Depth estimation and distant object detection are twoof the topics that will be considered. We are also con-cerned with the refinement of the navigation modeling

of an autonomous mobile robot using visual motionpatterns and additional sensory input.

AcknowledgmentsSpecial thanks to Ralph Breithaupt and Hartmut Sur-

mann for their help in providing the image sequences.

References

[1] J. L. Barron, D. J. Fleet, and S. S. Beauchemin.Performance of optical flow techniques, systemsand experiments. International Journal of Com-puter Vision, 12(1):43–77, 1994.

[2] R. Benosman and S. Bing Kang, editors.Panoramic Vision, Sensors, Theory, and Appli-cations. Springer, 2001.

[3] J. S. Chahl and M.V. Srinivasan. Reflective sur-faces for panoramic imaging. Applied Optics,36(31):8275–8285, 1997.

[4] N. Franceschini, J.M. Pichon, and C. Blanes.From insect vision to robot vision. PhilosophicalTransactions of the Royal Society London: Bio-logical Sciences (SeriesB), (337):283–294, 1992.

[5] B. K. P. Horn and B. G. Schunck. Determiningoptical flow. Artificial Intelligence, 17:185–203,1981.

[6] A. Horridge. Visual Navigation: From BiologicalSystems to Unmanned Ground Vehicles, chapterPattern and 3D Vision of Insects. Lawrence Erl-baum Associates,Inc., 1997.

[7] B. D. Lucas and T. Kanade. An iterative im-age registration technique with an application tostereo vision. In Proceedings of the Seventh Int.joint Conf. on AI, IJCAI.81, volume 2, pages674–679, 1981.

[8] R.C. Nelson and J. Aloimonos. Finding mo-tion parameters from spherical motion fields (orthe advantages of having eyes in the back ofyour head). Biological Cybernetics, (58):261–273,1988.

[9] J. Santos-Victor, G Sandini, F. Curotto, andS. Garibaldi. Divergent stereo for robot navi-gation: Learning from bees. In Proceedings ofICRA’93, 1993.

[10] Irem Stratmann. Omnidirectional imaging andoptical flow. In Proc. Workshop on Omnidirec-tional Vision 2002, pages 104–111.

(a) (b) (c)

0 90 180 270 360−4

−3

−2

−1

0

1

2

3

(f)(e)50 100 150 200 250

50

100

150

200

250

(d)50 100 150 200 250

50

100

150

200

250

Figure 5: (a) First image of a sequence taken from a camera moving through a corridor (b) Second image ofthe sequence, note that the person on the right moves independently to the left (c) Flow field superposed onto thesecond image, the line on the left indicates the estimated translation direction (d) Flow regions with significantchanges with respect to their background (e) Region with the maximum warping error (f) Distribution of the flowvectors in angular resolution and the (sine) fitting function