Towards a Platform Independent Real-Time Panoramic Vision Based LocalisationSystem

Nghia Ho, Ray JarvisMonash University, Australia

{nghia.ho, ray.jarvis}@eng.monash.edu.au

Abstract

This paper demonstrates a proof of concept for aplatform independent vision based localisation sys-tem that can be mounted on any mobile platformand operates in real-time. The system is able toglobally localise given a pre-scanned 3D model ofthe environment. This is a deliberate deviation fromthe SLAM (Simultaneous Localisation and Map-ping) approach because we are interested in globallocalisation, which assumes a map is readily avail-able. A panoramic mirror and a webcam is usedas the vision sensor. A particle filter is used forlocalisation. Typically, the particle filter requiresodometry feedback from the mobile platform toknow where to move the particles. This depen-dency is relocated to visual odometry from the cam-era, which provides rotation and translation feed-back, reinforcing the concept of a portable and plat-form independent localisation system. Currently,our visual odometry does not provide metric valuesfor translation, but instead derives it by assumingthe platform moves at a some average speed. Wesuggest some solutions to this problem in the dis-cussion. Our system is analogous to a GPS devicebut can operate in conditions where a GPS signalis not available and can provides greater accuracythan current consumer GPS.

1 IntroductionLocalisation is a fundamental task that is used in a wide rangeof applications. Robots need it to perform autonomous nav-igation. Augmented reality applications involving a head updisplay whereby a user navigates the real world with a virtualworld overlay requires localisation. Localisation has foundapplication in health care to assist people with visual impair-ment [Treuillet et al., 2007] and has been applied in a hos-pital environment to monitor and track people with disability[Savogin et al., 2006]. Over the past decades there have beenmany localisation solutions proposed, such as those utilising

passive beacon [Jarvis and Byrne, 1986], RFID [Hahnel etal., 2004], Wi-Fi [Siddiqi et al., 2003], GPS, tactile [Erick-son et al., 2008], laser [Fox et al., 2000], sonar [Leonard andDurrant-Whyte, 1992] and vision [Wolf et al., 2002]. Thechoice of solution largely depends on the application and anyconstraints it has. We are interested in localisation solutionsfor mobile platforms. Not all of the solutions mentioned canbe easily ported to any mobile platform without some con-siderable effort. The radio based systems tend to be portablein terms of size and weight, but equipment cost and deploy-ment can be prohibitive. Not only that, other issues have tobe addressed such as signal coverage. Laser, sonar and visionare popular sensors used in robotics. They do not require anymodification of the environment to function and can provideaccurate localisation. Laser systems tend be bulky and ex-pensive, while sonar has attenuation-limited range and is notsuitable outdoors. We find vision an attractive sensor becauseit is relatively inexpensive, small, light weight, passive in op-eration and can operate both indoors and outdoors.

In this paper, we propose a portable vision based locali-sation system that uses a panoramic mirror and a webcam.The system is able to perform global localisation given a pre-scanned 3D model of the environment using a Riegl LMSZ420i laser range scanner. The environment is scanned atmany locations to build a complete dense map. From thismap, we sample images at various poses, spaced uniformlyacross the map, by rendering what the panoramic mirrorwould see at that pose. We then extract features based onthe Haar wavelet for each sampled image to build a compactdatabase of signatures. We use the KLD-particle filter to lo-calise the platform.

We argue this approach, whilst not necessarily efficient,is perfectly reasonable because it is only in rare situationsthat the working environment is not well known, particularlywhen associated with built environments for which there areusually detailed plans. Building the 3D model is a once offtask, thus we can justify spending the time required to collectand process the 3D scans. We chose to use a panoramic mir-ror because of its large field of view. This has the advantageof capturing more information than a standard perspective

camera, which improves scene recognition. This also givesit some robustness to unwanted occlusion, like people, be-cause it only occupies a small percentage of the mirror’s fieldof view.

We have demonstrated our vision based localisation systempreviously in [Ho and Jarvis, 2007a]. What differentiates thesystem presented in this paper is the dependency on the mo-bile platform’s odometry has now been relocated to the cam-era for visual odometry. Rotation and translation informa-tion is calculating using optical flow. This makes the visionlocalisation system self contained and platform independent.This is particularly useful for platforms that have no readilyavailable means of obtaining odometry information to pass tothe particle filter, such as a person. In our experiments, wemounted the system on a walking frame designed for elderlypeople and tracked the person as they moved freely. A draw-back with our visual odometry implementation is the lack ofa metric information returned for translation. It only knowswhether the platform is moving forward or backward, but notthe exact amount it travels. For experiments, we assumed anaverage walking speed of 1 m/s.

Our vision system may not be as compact as a GPS re-ceiver but is a proof of concept of a portable vision basedlocalisation system. In fact, it is possible to reduce the sizeand weight of the vision system by using a smaller panoramicmirror if one wished to do so. In its current form we considerit a reasonable size for most mechanical platforms that allowsan option for mounting an additional small payload.

The localisation unit consists of a Quickcam Pro 9000 we-bcam, panoramic mirror with a profile described in [Chahl,1998]. The mirror has a vertical field of view of about 135degrees when the camera is placed about 10cm away fromthe tip of the mirror. The laptop used has an Intel Centrinorunning at 1.8 GHz. The vision system is mounted on a walk-ing frame as shown in Figure 1.

2 3D Model AcquisitionThe 3D model of the environment was created using a RieglLMS Z420i terrestrial laser scanner and a high resolution dig-ital camera. This scanner has an effective range between 2-800 metres, 360x80 degrees field of view. Colour informa-tion for each 3D point is obtained via a Nikon D100 cameramounted on top. The environment of interest is scanned atvarious locations to build a complete model. The dense pointclouds from collective scan are then registered together us-ing in-house software based on the Trimmed Iterative ClosestPoint (TrICP) algorithm [Chetverikov et al., 2002]. Each scanis registered to the next nearest scan with one scan chosenas the reference point. Our test environment was scanned at19 different locations. The registered point clouds containedover 134 million points occupying about 2.2GB of disk space.

The point cloud was post processed by representing asmuch of the environment using planes and texture, which aremore memory efficient in representation. The plane fitting

Figure 1: Walking frame with vision system.

was carried out using RANSAC [Fischler and Bolles, 1981]as described in [Ho and Jarvis, 2007b]. This reduced the3D model to about 630MB (~71% compression over the rawpoint cloud), 171MB allocated to texture and the rest beingpoints. Extracting planes also has the desirable property ofcreating a continuous surface. This is particularly importantwhen creating panoramic images, as missing pixel data areinterpolated by the plane and texture.

From this new model we sample poses every 0.5 metres(across a uniformly spaced grid) and 10 degrees of rotation tocreate a large appearance map of all possible poses. This wasdone by simulating the geometric properties of our panoramicmirror in OpenGL using GLSL vertex/fragment shader, as de-tailed in [Ho and Jarvis, 2007a]. This equated to 150,732possible poses, though we do not explicitly store all the pose

images, only the pose at 0 degree on disk. Figure 2 shows the3D model of the engineering area, where we do testing, andthe underlying plane structure. The test environment mea-sures about 50x50 metres.

Figure 2: 3D model of the engineering area. Top: planes only(5562 planes). Bottom: textured planes and points.

3 Feature extractionWe use a Haar wavelet feature extraction method based on Ja-cobs’ [Jacobs et al., 1995] use of wavelets for a content basedimage retrieval system. The wavelet signature proposed byJacobs uses the largest N coefficients, where as we use thefirst MxN block of low frequency coefficients. Algorithm 1shows the process of creating the wavelet signature.

The signature can be efficiently stored as an array of bits.Our signature is more memory efficient than the signatureproposed by Jacobs because the index of the coefficient donot have to be stored, only the signs, because the data is storedas a contiguous array. We only use the intensity channel (Y)because we found that although adding the colours channel

did speed up particle convergence it was susceptible to colourchanges and sometimes localised incorrectly initially. Whenit did work, there was no noticeable increase in accuracy thuswe saw no advantage over using the Y channel only when theparticle filter is in tracking mode. The database of 150,732signatures totals to about 21MB using the Y wavelet signa-ture.

A weighted sum of absolute difference is used to measurethe similarity between two signatures. The coefficients areweighted according to the resolution they belong to, since thewavelet encodes information at different resolutions. We usea naive Bayes probability model, whereby we assume the co-efficients at different resolutions to be independent. We foundthis simple model to produce better results than assuming in-terdependence amongst the coefficient levels. For each reso-lution group, a probability of a match is calculated using a lo-gistic/sigmoid function, where the sigmoid function is givenas

sigmoid (x) =1

1+ e−x

Following the naive Bayes model, the final score is themultiplication of all the probabilities at the different levels,given as

score = ∏i

sigmoid (bi +wibinsi)

Where band w are the bias and weights of the sigmoid func-tion, bins is the accumulated number of binary matches andthe index i is the resolution level. There are 5 resolution lev-els for our 64x16 wavelet signature, 4x1, 8x2, 16x4, 32x8and 64x16. The weights were trained using logistic regres-sion using a set of ground truth images. For more detail onour wavelet signature please refer to [Ho and Jarvis, 2008].

4 Visual odometry

Visual odometry is a well researched area and can be tracedback to the early works by [Moravec, 1980]. Many new ap-proaches to visual odometry have been proposed since. [Nis-ter et al., 2004; 2006] used stereo vision to track point fea-tures in real-time. [Corke et al., 2004] used a single omnidi-rectional mirror for a planetary rover. [Maimone et al., 2007]used a stereo vision visual odometry system on the famousMars rover. For this paper, we have chosen to simplify thevisual odometry on purpose to demonstrate a proof of con-cept by limiting the motions detected. An optical flow al-gorithm using OpenCV’s Lucas-Kanade tracker [Lucas andKanade, 1981] was used to determine motion. The panoramicis aligned such that 0 degrees is at x=0 and partitioned intotwo partitions as shown in Figure 3.

The displacement of the tracked points, along the x direc-tion, is accumulated for both partitions and the median taken.We’ll call the two accumulated variables LeftPart and Right-Part. The optical flow, based on the sign of the median ofLeftPart and RightPart, has the following characteristics

Algorithm 1 Haar wavelet signature.

1. Unwrap panoramic image(512x128).

2. Convert image to YIQ colourspace.

3. Perform a 2D Haar wavelettransform on chosen colourchannels.

4. Quantise coefficients to binary[0,1] (negative, positive values).

5. Keep the first 64x16 coefficients.

signature = [010100101 .... ]6. The signature are the quantisedcoefficients, stored as an array ofbinary values.

Figure 3: Partitioning of image for optical flow calculations.

LeftPart RightPart Motion− − left rotation+ + right rotation+ - forward- + backward

There is also the “no motion” state, not shown in the tablebut is detected. The amount of rotation performed is relatedby

∆θ =∆x

imagewidth×360

Translation is calculated by

∆d = AV G_SPEED×∆twhere ∆t is the time to process a single frame capture and

averages about 100 ms when the particle filter is in trackingmode, resulting in a frame rate of 10 fps.

5 LocalisationFor localisation, we used an adaptive particle filter usingKLD-sampling by Fox [Fox, 2001]. The idea behind theparticle filter [Dellaert et al., 1999a; 1999b] is, rather thanmaintain all possible belief about the robot’s state, the statespace being represented by a sample of particles. Each par-ticle is a single hypothesis of the robot’s state. Each particleis also assigned a weight that measures how well the hypoth-esis fits with actual sensor and odometry information. Theparticle filter has two main phases, the prediction and updatephase. The prediction phase occurs when the robot has justexecuted a motion control and updates the particles by ap-plying the motion model. The update phase occurs when the

robot receives new sensor information. The particles’ weightare updated based on the likelihood of the sensor reading atthat hypothesis. A new set of particles is then created by ran-domly resampling from the current set based on the particles’weight. Particles with low weight will tend to die off whilethose with high weight survive. This behaviour is analogousto the biological evolutionary concept of survival of the fittest.

The adaptive particle filter is essentially the particle filterwith the resampling stage modified to vary the number of par-ticles, depending on the distance between the true posteriorand sampled base approximation. If the particles have con-verged tightly around the true pose then less particles willbe used to keep track of the pose and vice-versa. The adap-tive particle filter was chosen primarily for the speed increaseover the vanilla particle filter. The number of particles is de-termined by

n =k−12ε

{1− 2

9(k−1)+

√2

9(k−1)z1−σ

}3

where ε is the Kullback-Leibler distance between twoprobability distributions, z1−σ is the upper 1−σ quantile ofthe standard normal distribution, and k is the number of sup-porting bins. We found the following parameter values towork well (obtained experimentally), ε = 0.1, z1−σ = 2.32,nmin = 1000 and a bin size of 0.5m× 0.5m× 10◦. nminis theminimum number of particles to use at all times. We also per-turbed the particles after resampling by adding uniform ran-dom noise to spread it around the true pose. This increases ro-bustness to localisation failure. Previously we used a uniformrandom noise of ±2m for the position and ±10◦for orienta-tion, but have recently used an adaptive version. We noticedthat the particle filter can momentarily lose track for ambigu-ous scenes despite the added random noise. Our solution isto use a variable uniform random noise for the position givenby

N = max(2,2.5σx)r = rand (−N,N)x′ = x+ r cos(θ)y′ = y+ r sin(θ)

where σx is the standard deviation in the x direction. Ourenvironment does not shown any bias in any particular direc-tion, σx and σy tend to tbe similar. The lower bound of 2in the max function ensures there are always particles spreadaround the true pose. This simple adaptive noise, combinedwith KLD-sampling, has the effect of adding more particlesaround a larger radius where the tracking starts to fail. Theradius and number of particles increases as the tracking getsworse until it has localised correctly. The worst case scenariowould be a complete global localisation from scratch with allthe particles uniformly distributed across the map.

The estimated pose is calculated by taking the weightedmean

X̄ = ∑i

Xiwi

where X is the pose vector [x,y,θ ]T and w the weight of theparticle.

6 Experimental ResultsA total of 4 experiments were conducted around the Engineer-ing School area. The walking frame was placed at a randomlocation and the user moved freely while being localised inreal-time. 100,000 particles are initially distributed uniformlyacross the map. The system localises every 0.5m of travel anduses dead reckoning in between. Localisation results for all4 experiments are shown in Figure 5 and 6. The red path isthe estimated path and the green is the actual path travelled.Experiment 1 has an area where it has trouble localising andloses track momentarily and is highlighted in Figure 5a atthe top. We noticed for this experiment it was a cloudy dayand ambient lighting in general was dim. This triggered theautomatic lights to turn on for that area, which changed theappearance of the scene. A similar effect was experienced inexperiment 4 except the ambient lighting was slightly brighterbut the automatics still turned on. The ambient lighting waswell lit for experiments 2 and 3. Experiments 2 and 3 tend toproduce smoother localisation than the other two. This sug-gests the wavelet feature is sensitive to local lighting changes.

A comparison of the images at the area in question for all4 experiments and the computer generated (CG) image fromthe 3D map is shown in Figure 4. Experiment 1 visually looksthe most similar to the CG image, yet has the worst localisa-tion results. Experiments 2 and 3 have significant saturation,about 35% of the image’s area, on the left portion. Despitethis, they produce good localisation results even when theoriginal CG image from the 3D map is not saturated for thatportion. Experiment 4 has a combination of local lightingchanges and some saturation. The quality of the localisationis somewhere between that of experiment 1 and experiment2 and 3. This supports our findings in [Ho and Jarvis, 2008],where we found the wavelet signature is robust to mild oc-clusion using random noise (up to 40-50%). If we think ofrandom noise as removing useful information from the scenethen this is analogous to image saturation.

Figure shows a sequence where the particle filter starts tohave trouble localising and reacts by increasing the search ra-dius and number of particles. The reverse occurs when theparticle filter becomes confident of its localisation. This sce-nario occurs when the particle filter encounters a scene withambiguous matches.

7 DiscussionThe vision based localisation is capable of operating in real-time at 10fps on an Intel Centrino 1.8GHz, which is adequatefor an average speed of 1m/s. A breakdown of the processingtime for a single frame is given in Table 1. The time for up-dating the particle filter is given when it is in tracking mode,where 1000 particles are used. The initial global localisationof 100,000 particles takes about 2 seconds.

capturing image 10msunwrapping image 10ms

optical flow 20msupdating particle filter 60ms

Table 1: Breakdown of processing time for a single frame.

The localisation accuracy on average is within 60cm of thetrue position and 5 degrees for orientation. The accuracy ofvision based localisation is limited, amongst other factors, bythe minimum displacement required to resolve a change inscene. One can imagine placing the system in the middleof a football field. For the scene to change significantly tolocalise might require a few metres of displacement. Someparts of our test environment have localisation errors within30cm, such as narrow pathway where only a couple centime-tres of displacement is required to resolve a scene change.

The test environment is semi-outdoors and subjected tovarying lighting conditions. This causes the panoramic cam-era to experience saturation when there is a high contrast be-tween a bright and dark area. Saturation can be seen as eras-ing information from the scene and degrades localisation. Sofar we have not experienced extreme saturation because of theroof. A small part of the environment has changed since welast scanned, such as the introduction of a vending machine,new billboard posters and a whiteboard. These new objects,not part of the original scans, have not affected localisationnoticeably because they occupy only a small percentage ofthe mirror’s field of view.

We also noticed the system is quite robust to the motionblur exhibited by the camera. The wavelet signature is notaffected by motion blur because the signature itself ends upblurring the image anyway, by averaging the image at differ-ent resolution levels. Though, we were rather surprise thatthey optical flow still worked with motion blur present.

A weakness of the proposed system is the visual odometry.There is nothing preventing the optical flow from trackingdynamic objects, such as people. If there are more dynamictracked points than static then rotation information will beincorrect, though we have yet to test this. As mentioned pre-viously, the visual odometry does not return exact translationdistances but relies on assuming an average velocity. This as-sumption makes the system specific for a particular platform,where as generality is preferred. We have already mentionedsome past solutions using stereo vision and omnidirectionalmirrors. A more straight forward solution would be to add asecond camera pointing towards the ground and use opticalflow, much like how an optical mouse works. If the camerais at a known distance from the ground then metric informa-tion can be calculated. This assumes the distance between thecamera and the ground remains constant, which is reasonablefor a planar motion platform. This solution has the advantageof being immune to tracking unwanted dynamic objects.

Our approach of using a 3D model to generate pose images

allows the possibility to extend localisation to higher degreesof freedom. We have successfully experimented with local-isation in 3D space by considering height. A lighting tripodwas used to control the height of the camera. Adding theextra height dimension increased the database to a bit over900,000 pose signatures. Results for these experiments willbe published in the near future.

8 ConclusionWe have demonstrated a real-time vision based localisationthat is capable of localising a general mobile platform atwalking speed. The system is proof of concept that it can beapplied to any planar motion platform, with extensions to 3Dspace being possible provided odometry is available. The lo-calisation accuracy is better than consumer GPS and can op-erate indoors, where a GPS signal might not be available. Atrue visual odometry solution still remains the missing pieceto make the system fully platform independent. Although wehave not reached this objective, we have shown that with asimple assumption, such as an average walking speed, it isgood enough for a real world application.

References[Chahl, 1998] Javaan Singh Chahl. Panoramic surveillance

system. U.S. Patent 5,790,181, 1998.

[Chetverikov et al., 2002] D. Chetverikov, D. Svirko,D. Stepanov, and P. Krsek. The trimmed iterative closestpoint algorithm. In Proc. International Conf. on PatternRecognition, Quebec, Canada. IEEE Comp. Soc., 2002.

[Corke et al., 2004] Peter Ian Corke, Dennis Strelow, andSanjiv Singh. Omnidirectional visual odometry for a plan-etary rover. In Proceedings of IROS 2004, 2004.

[Dellaert et al., 1999a] Frank Dellaert, Wolfram Burgard,Dieter Fox, and Sebastian Thrun. Using the condensationalgorithm for robust, vision-based mobile robot localiza-tion. IEEE Computer Society Conference on ComputerVision and Pattern Recognition ( CVPR’99 ), June 1999.

[Dellaert et al., 1999b] Frank Dellaert, Dieter Fox, WolframBurgard, and Sebastian Thrun. Monte carlo localizationfor mobile robots. IEEE International Conference onRobotics and Automation (ICRA99), 2:1322–1328 vol.2,1999.

[Erickson et al., 2008] L.H. Erickson, J. Knuth, J.M.O’Kane, and S.M. LaValle. Probabilistic localizationwith a blind robot. Robotics and Automation, 2008.ICRA 2008. IEEE International Conference on, pages1821–1827, May 2008.

[Fischler and Bolles, 1981] Martin A. Fischler and Robert C.Bolles. Random sample consensus: a paradigm for modelfitting with applications to image analysis and automatedcartography. Commun. ACM, 24(6):381–395, 1981.

[Fox et al., 2000] D. Fox, S. Thrun, F. Dellaert, and W. Bur-gard. Particle filters for mobile robot localization. InA. Doucet, N. de Freitas, and N. Gordon, editors, Sequen-tial Monte Carlo Methods in Practice. Springer Verlag,New York, 2000. To appear.

[Fox, 2001] Dieter Fox. Kld-sampling: Adaptive particle fil-ters. In Advances in Neural Information Processing Sys-tems 14. MIT Press, 2001.

[Hahnel et al., 2004] D. Hahnel, W. Burgard, D. Fox,K. Fishkin, and M. Philipose. Mapping and localizationwith rfid technology. Robotics and Automation, 2004. Pro-ceedings. ICRA ’04. 2004 IEEE International Conferenceon, 1:1015–1020 Vol.1, April-1 May 2004.

[Ho and Jarvis, 2007a] N. Ho and R. Jarvis. Global locali-sation in real and cyberworlds using vision. AustralasianConference on Robotics and Automation, 2007.

[Ho and Jarvis, 2007b] Nghia Ho and R. A. Jarvis. Largescale 3d environmental modelling for stereoscopic walk-through visualisation. 3DTV Conference, 2007.

[Ho and Jarvis, 2008] Nghia Ho and R. A. Jarvis. Visionbased global localisation using a 3d environmental modelcreated by a laser range scanner. IEEE/RSJ InternationalConference on Intelligent Robots and Systems, 2008.

[Jacobs et al., 1995] Charles E. Jacobs, Adam Finkelstein,and David H. Salesin. Fast multiresolution imagequerying. Computer Graphics, 29(Annual ConferenceSeries):277–286, 1995.

[Jarvis and Byrne, 1986] Raymond Jarvis and Julian Byrne.Robot navigation : Touching, seeing and knowing. 1stAustralian Artificial Intelligence Congress, Melbourne,page Section e. (12 pages), 18-20 Nov 1986.

[Leonard and Durrant-Whyte, 1992] John J. Leonard andHugh F. Durrant-Whyte. Directed Sonar Sensing for Mo-bile Robot Navigation. Kluwer Academic Publishers, Nor-well, MA, USA, 1992.

[Lucas and Kanade, 1981] Bruce D. Lucas and TakeoKanade. An iterative image registration technique withan application to stereo vision (darpa). In Proceedings ofthe 1981 DARPA Image Understanding Workshop, pages121–130, April 1981.

[Maimone et al., 2007] Mark Maimone, Yang Cheng, andLarry Matthies. Two years of visual odometry on themars exploration rovers: Field reports. J. Field Robot.,24(3):169–186, 2007.

[Moravec, 1980] Hans Moravec. Obstacle avoidance andnavigation in the real world by a seeing robot rover.In tech. report CMU-RI-TR-80-03, Robotics Institute,Carnegie Mellon University & doctoral dissertation, Stan-ford University. September 1980. Available as Stanford

AIM-340, CS-80-813 and republished as a Carnegie Mel-lon University Robotics Institue Technical Report to in-crease availability.

[Nister et al., 2004] David Nister, Oleg Naroditsky, andJames Bergen. Visual odometry. Computer Vision andPattern Recognition, IEEE Computer Society Conferenceon, 1:652–659, 2004.

[Nister et al., 2006] David Nister, Oleg Naroditsky, andJames Bergen. Visual odometry for ground vehicle ap-plications. Journal of Field Robotics, 23:2006, 2006.

[Savogin et al., 2006] I. Savogin, M. Scaioni, and C. Fasoli.Portable monitoring and localization systems for disabledpatients. In IEVM06, 2006.

[Siddiqi et al., 2003] Sajid Siddiqi, Gaurav S. Sukhatme,and Andrew Howard. Experiments in monte-carlo local-ization using wifi signal strength. In Proceedings of the In-ternational Conference on Advanced Robotics, Coimbra,Portugal, Jul 2003.

[Treuillet et al., 2007] Sylvie Treuillet, Eric Royer, ThierryChateau, Michel Dhome, and Jean marc Lavest. Bodymounted vision system for visually impaired outdoor andindoor wayfinding assistance. Conference & Workshop onAssistive Technologies for People with Vision & HearingImpairments Assistive Technology for All Age, 2007.

[Wolf et al., 2002] Jürgen Wolf, Wolfram Burgard, and HansBurkhardt. Robust vision-based localization for mobilerobots using an image retrieval system based on invariantfeatures. In IEEE International Conference on Roboticsand Automation, 2002.

(a) image generated from the 3D map

(b) Experiment 1

(c) Experiment 2

(d) Experiment 3

(e) Experiment 4

Figure 4: Comparison of the images at the area where track-ing was momentarily lost in experiment 1 for all the other 3experiments and the CG image from the 3D map.

Figure 7: Example of handling potential localisation failure.

(a) Experiment 1

(b) Experiment 2

Figure 5: Localisation results for experiments 1 and 2. The red path is the estimated path and the green is the actual pathtravelled. Top: Experiment 1 - started at 2:47 pm, path is approximately 255 metres long. Bottom: Experiment 2 - started at12:47 pm, path is approximately 300 metres long.

Figure 6: Localisation results for experiments 3 and 4. The red path is the estimated path and the green is the actual pathtravelled. Top: Experiment 3 - started at 1:00 pm, path is approximately 400 metres long. Bottom: Experiment 4 - started at3:16 pm, path is approximately 400 metres long.