Predicting the Change – A Step Towards Life-LongOperation in Everyday Environments

Niko Sunderhauf, Peer Neubert, Peter ProtzelDepartment of Electrical Engineering and Information Technology

Chemnitz University of Technology, Germany{niko.suenderhauf, peer.neubert, peter.protzel}@etit.tu-chemnitz.de

Abstract—Changing environments pose a serious problem tocurrent robotic systems aiming at long term operation. Whileplace recognition systems perform reasonably well in static orlow-dynamic environments, severe appearance changes that occurbetween day and night, between different seasons or differentlocal weather conditions remain a challenge. In this paper wepropose to learn to predict the changes in an environment. Ourkey insight is that the occurring scene changes are in partsystematic, repeatable and therefore predictable. The goal of ourwork is to support existing approaches to place recognition bylearning how the visual appearance of an environment changesover time and by using this learned knowledge to predict its ap-pearance under different environmental conditions. We describethe general novel idea of scene change prediction and a proof ofconcept implementation based on vocabularies of superpixels. Wecan show that the proposed approach improves the performanceof SeqSLAM and BRIEF-Gist for place recognition on a large-scale dataset that traverses an environment under extremelydifferent conditions in winter and summer.

I. INTRODUCTION

Long term operation in changing environments is one ofthe major challenges in robotics today. Robots operatingautonomously over the course of days, weeks, and monthsare faced with significant changes in the appearance of anenvironment: A single place can look extremely differentdepending on the current season, weather conditions or thetime of day. Since state of the art algorithms for autonomousnavigation are often based on vision and rely on the system’scapability to recognize known places, such changes in theappearance pose a severe challenge for any robotic systemaiming at autonomous long term operation.

The problem has recently been addressed by few authors,but so far no congruent solution has been proposed. Milfordand Wyeth [3] proposed to increase the place recognitionrobustness by matching sequences of images instead of singleimages and achieved impressive results on two across-seasonsdatasets. Exploring into a different direction, Churchill andNewman [2] proposed to accept that a single place can havea variety of appearances. Their conclusion was that insteadof attempting to match different appearances across seasonsor severe weather changes, different experiences should beremembered for each place, where each experience coversexactly one appearance. Both suggested approaches can be un-derstood as the extreme ends of a spectrum of approaches thatspans between interpreting changes as individual experiencesof a single place on one hand and increasing the robustness of

State of the Art Approach

Learned Dictionary

Prediction

Proposed Approach With Scene Change Prediction

Matching

Matching

Training Dataset

Fig. 1. State of the art approaches to place recognition attempt to directlymatch two scenes, even if they have been observed under extremely differentenvironmental conditions. This is prone to error and leads to bad recognitionresults. Instead, we propose to predict how the query scene (the winterimage) would appear under the same environmental conditions as the databaseimages (summer). This prediction process uses a dictionary that exploits thesystematic nature of the seasonal changes and is learned from training data.

the matching against appearance changes on the other hand.Our work presented in the following is orthogonal to thisspectrum.

What current approaches to place recognition (and environ-mental perception in general) lack, is the ability to reasonabout the occurring changes in the environment. Most ap-proaches try to merely cope with them by developing change-invariant descriptors or matching methods. Potentially morepromising is to develop a system that can learn to predictcertain systematic changes (e.g. day-night cycles, weather andseasonal effects, re-occurring patterns in environments whererobots interact with humans) and to infer further informationfrom these changes. Doing so without being forced to explic-itly know about the semantics of objects in the environmentis in the focus of our research and the topic of this paper.

Fig. 1 illustrates the core idea of the paper and how itcompares to the current state of the art place recognitionalgorithms. Suppose a robot re-visits a place under extremely

different environmental conditions. For example, an environ-ment was first experienced in summer and is later re-visitedin winter time. Most certainly, the visual appearance hasundergone extreme changes. Despite that, state of the artapproaches would attempt to match the currently seen winterimage against the stored summer images.

Instead, we propose to predict or hallucinate how thecurrent scene would appear under the same environmentalconditions as the stored past representations, before attemptingto match against the database. That is, when we attempt tomatch against a database of summer images but are in wintertime now, we predict how the currently observed winter scenewould appear in summer time or vice versa.

The result of this prediction process depends on the ac-tual place recognition algorithm that is applied. When usingapproaches like SeqSLAM [3] or BRIEF-Gist [4], the resultwould be a synthesized image as illustrated in Fig. 1. Thisimage preserves the structure of the original scene but is closein visual appearance to the corresponding original summerscene. When using place recognition based on a bag of wordsapproach (e.g. FAB-MAP), the result of the prediction processwould be a translated bag of words.

In any case, the proposed prediction can be understood astranslating the image from a winter vocabulary into a summervocabulary or from winter language into summer language. Asis the case with translations of speech or written text, somedetails will be lost in the process, but the overall idea, i.e. thegist of the scene will be preserved. Sticking to the analogy,the error rate of a translator will drop with experience. Thesame can be expected of our proposed system: It is dependenton training data, and the more and the better training data isgets, the better can it learn to predict how a scene changesover time or even across seasons.

To the best of our knowledge, the idea of predicting extremescene changes across seasons to aid place recognition is noveland has not been proposed before.

II. LEARNING TO PREDICT SCENE CHANGES ACROSSSEASONS

How can the severe changes in appearance a landscape un-dergoes between winter and summer be learned and predicted?The underlying idea of our approach is that the appearancechange of the whole image is the result of the appearancechange of its parts. If we had an idea of the behavior of eachpart, we could predict the whole image. However, instead oftrying to recover semantic information about the image partsand model their behavior explicitly, we make the assumptionthat similarly appearing parts change their appearance in asimilar way. While this is for sure not always true, it seems tohold for many practical situations. This idea can be extendedto groups of parts, incorporating their mutual relationships.

To predict how the appearance of a scene changes betweendifferent conditions (e.g. summer and winter), we proposeto first conduct a learning phase on training data. This datacomprises scenes observed under both summer and winterconditions. In the subsequent prediction phase, the change in

Condition 1(e.g. Winter)

Condition 2(e.g. Summer)

Superpixel

DescriptorsDescriptors

Superpixel

Vocabulary for Condition 2(e.g. Summer)

Learned Dictionary(e.g. Winter - Summer)

Vocabulary for Condition 1(e.g. Winter)

Known Correspondences

Fig. 2. Learning a dictionary between images under different environmentalconditions (e.g. winter and summer). The images are first segmented into su-perpixels and a descriptor is calculated for each superpixel. These descriptorsare then clustered to obtain a vocabulary of visual words for each condition. Ina final step, a dictionary that translates between both vocabularies is learned.This can be done due to the known pixel-accurate correspondences betweenthe input images.

appearance of a new scene can be predicted using the resultsof the training phase. In the following, we explain our currentproof of concept implementation of the proposed scene changeprediction approach.

A. Learning Vocabularies and a Dictionary

During the training phase we have to learn a vocabulary foreach viewing condition and a dictionary to translate betweenthem. In a scenario with two viewing conditions (e.g. summerand winter), the input to the training are images of the samescenes under both viewing conditions and known associationsbetween pixels corresponding to the same world point. Obvi-ously the best case would be perfectly aligned pairs of images,e.g. captured by stationary webcams. Which approach to visualvocabulary learning is the most promising for the proposedscene change prediction has to be evaluated in future work.

Fig. 2 illustrates the training phase. In our current proof ofconcept implementation, each image is segmented into SLICsuperpixels [1]. For each superpixel a descriptor that containsa color histogram in LAB color space (each channel with 10bins), an U-SURF descriptor (128 byte) to capture textureinformation and the y-coordinate to encode spatial informationis computed. The set of descriptors for each viewing conditionis clustered to a vocabulary using hierarchical k-means. Eachcluster center becomes a word in this visual vocabulary. Thedescriptors and the average appearance of each word (the wordpatch) are stored for later synthesizing of new images. For ourexperiments, we learned 10.000 words for each vocabulary.

Given the learned vocabularies, we can proceed to learn

Query Image(e.g. Winter)

DescriptorsSuperpixel

Winter Vocabulary

Translate with Dictionary(e.g. Winter - Summer)

Predicted Image(e.g. Summer)

Word Representationof Query Image

Fig. 3. Predicting the appearance of a query image under differentenvironmental conditions: How would the current winter scene appear insummer? The query image is first segmented into superpixels and a descriptoris calculated for each of these segments. With this descriptor each superpixelcan be classified as one of the visual words from the vocabulary. This wordimage representation can then be translated into the vocabulary of the targetscene (e.g. summer) through the dictionary learned during the training phase.The result of the process is a synthesized image that predicts the appearanceof the winter query image in summer time.

a dictionary that can translate between visual words fromtwo environmental “languages” or conditions as illustrated inthe lower part of Fig. 2. This dictionary captures the transi-tions of the visual words when the environmental conditionschange. The dictionary can either capture the complete discreteprobability distribution of these transitions or only store thetransition that occurs most often.

B. Predicting Image Appearances Across Seasons

Fig. 3 illustrates how we can use the learned vocabulariesand the dictionary to predict the appearance of a query imageacross different environmental conditions.

The query image is segmented into superpixels and a de-scriptor for each superpixel is computed. Using this descriptor,a word from the vocabulary corresponding to the currentenvironmental conditions (e.g. winter) is assigned to eachsuperpixel. The learned dictionary between the query condi-tions and the target conditions (e.g. winter-summer) is used totranslate these words into words of the target vocabulary.

If the vocabularies also contain word patches, i.e. anexpected appearance of each word, we can synthesize thepredicted image based on the word associations from thedictionary and the spatial support given by the superpixelsegmentation.

III. EXPERIMENTS AND RESULTS

After the previous section explained how scene changeprediction across seasons can be performed, we are going todescribe the conducted experiments and their results.

A. The Nordland Dataset

To test our proposed approach of scene change prediction,we required a dataset where a camera traverses the same placesunder very different environmental conditions but under asimilar viewing perspective: The TV documentary “Norlands-banen – Minutt for Minutt” by the Norwegian Broadcasting

Corporation NRK provides video footage of a 728 km longtrain ride that has been filmed from the perspective of thetrain driver four times in spring, summer, fall, and winter.The full-HD recordings have been time-synchronized such thatan arbitrary frame from one video corresponds to the sameframe of any of the other three videos etc. Therefore, frame-accurate ground truth information, e.g. corresponding scenes,are available. Furthermore, since the cameras were mountedexactly in the same spot in the driver’s cabin, the four videosare almost perfectly aligned and thus allow easy learningof visual word transitions between the four seasons. Thevideos are available online at http://nrkbeta.no/2013/01/15/nordlandsbanen-minute-by-minute-season-by-season/ under aCreative Commons licence (CC-BY).

For our experiments described in the following we extracted30 minutes from the spring and the winter videos, startingapproximately at 2 hours into the drive. From the four avail-able videos, the spring video best resembled typical summerweather conditions. To form the training dataset, we extractedapproximately 900 frames from the first 8 minutes of this 30minutes subset. This training dataset was used to learn thevisual vocabulary for summer and winter and the dictionaryto translate between both seasons. The remaining 22 minutesof the video subset served as the test dataset to evaluate theperformance of the proposed scene change prediction.

B. Extending and Improving BRIEF-Gist and SeqSLAM

SeqSLAM [3] and BRIEF-Gist [4] are two establishedapproaches to appearance-based place recognition. BRIEF-Gist is a holistic descriptor that encodes the visual appearanceof a whole image in a short bit string. It supports placerecognition by applying the Hamming distance between twodescriptors in order to find the single global best matchingquery image. In contrast, SeqSLAM performs place recogni-tion by matching whole sequences of images and has beenshown to perform well despite severe appearance changes[3, 5]. We use OpenSeqSLAM [5] to perform the experiments.

Combining both approaches with our scene change predic-tion is particularly easy, since the change prediction algorithmcan be executed as a preprocessing step before SeqSLAMor BRIEF-Gist start with their own processing. Since we at-tempted to match summer against winter images, we predictedthe visual appearance of each summer scene in winter and fedthe predicted winter images together with the original realwinter images into BRIEF-Gist and SeqSLAM.

Fig. 4 compares precision-recall curves achieved by bothalgorithms with and without our proposed scene changeprediction. The apparent result is that both BRIEF-Gist andSeqSLAM can immediately benefit from the change predic-tion. For SeqSLAM we plot the results for several values ofthe ds parameter that controls the minimal required lengthof the matched image sequences in seconds. We can seethat SeqSLAM’s performance increases with larger ds, asexpected.

We can conclude that although SeqSLAM alone reachesgood matching results, it can be significantly improved by first

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1OpenSeqSLAM on Nordland Dataset

Pre

cisi

on

Recall

Summer − Winter, ds = 3

Predicted Winter − Winter

Summer − Winter, ds = 5

Predicted Winter − Winter

Summer − Winter, ds = 10

Predicted Winter − Winter

Summer − Winter, ds = 20

Predicted Winter − Winter

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

BRIEF−Gist on Nordland Dataset

Pre

cis

ion

Recall

ISO−F

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

Summer − Winter

Predicted Winter − Winter

S − S Words

Fig. 4. Precision recall plots obtained by place recognition across seasonswith SeqSLAM [3] (left) and BRIEF-Gist [4] (right). The plots compare theperformance of the stand-alone algorithms with the boosted performance whenthe appearance of the winter images is predicted before place recognition isattempted. It is apparent that our proposed approach can significantly improvethe performance of both algorithms. For comparison, the blue curve in theright plot shows the performance of BRIEF-Gist when matching summerimages directly with summer word images, i.e. performing place recognitionunder constant environmental conditions.

predicting the appearance of the query scene under the viewingconditions of the stored database scenes. Also BRIEF-Gistcan benefit from the proposed appearance change prediction,although its performance is in general much worse thanSeqSLAM’s.

For comparison we also evaluated the performance of FAB-MAP (using openFAB-MAP) on the dataset. As expected,directly matching winter against summer images was notsuccessful. The best measured recall was 0.025 at 0.08 preci-sion, presumably because FAB-MAP fails to detect commonfeatures in the images from both seasons.

IV. DISCUSSION AND CONCLUSIONS

Our paper described the novel concept of learning to predictsystematic changes in the appearance of environments. We ex-plained our implementation based on superpixel vocabulariesand demonstrated how two approaches to place recognition,BRIEF-Gist and SeqSLAM, can benefit from the scene changeprediction step.

We can synthesize an actual image during this prediction.This simplifies the qualitative evaluation by visually compar-ing the predicted with the real images and further allows touse existing place recognition algorithms for quantitative eval-uation. However, the proposed idea of scene change predictioncan in general be performed on different levels of abstraction:It could also be applied directly on holistic descriptors likeBRIEF-Gist, on visual words like the ones used by FAB-MAPor on the downsampled and patch-normalized thumbnail im-ages used by SeqSLAM. Furthermore, the learned dictionarycan be as simple as a one-to-one association or capture afull distribution over possible translations for a specific word.In future work this distribution could also be conditioned onthe state of neighboring segments, and other local and globalimage features and thereby incorporate mutual influences andsemantic knowledge. This could be interpreted as introducinga grammar in addition to the vocabularies and dictionaries.How such extended statistics can be learned from training data

efficiently is an interesting direction for future work.If the dictionary does not exploit such higher level knowl-

edge (as in the superpixel implementation introduced here) thequality of the prediction is limited. In particular, when solelyrelying on local appearance of image segments for prediction,the choice of the training data is crucial. It is especiallyimportant that the training set is from the same domain asthe desired application, since image modalities that were notwell-covered by the training data can not be correctly modelledand predicted. Exploring the requirements for the trainingdataset and how the learned vocabularies and dictionary canbest generalize between different environments will be part ofour future research.

In its current form, our algorithm requires perfectly alignedimages in the training phase. This requirement is hard to fulfilland limits the available training datasets. We will explore waysto ease this requirement in future work, e.g. by anchoring thetraining images on stable features. Another key limitation ofthe system in its current form is that it requires different vo-cabularies for discrete sets of environmental conditions. Whileit is of course possible to create and manage a larger numberof such vocabularies and the respective mutual dictionaries, aunified approach that learns and maintains a single vocabularythat captures all conditions would be more desirable. Asalready mentioned, the Nordland dataset provides somewhatoptimal conditions (apart from the season-induced appearancechanges) for place recognitions, since the camera observesthe scene from almost exactly the same viewpoint in all fourseasons and the variability of the scenes in terms of semanticcategories is rather low. These conditions would usually notbe met and we therefore prepare to evaluate the proposedapproach in a more general setting using data from vehicles inurban environments and training data that has been collectedfrom stationary webcams over the course of several months.

REFERENCES

[1] Radhakrishna Achanta, Appu Shaji, Kevin Smith, AurelienLucchi, Pascal Fua, and Sabine Susstrunk. SLIC Super-pixels Compared to State-of-the-Art Superpixel Methods.volume 34, 2012.

[2] Winston Churchill and Paul M. Newman. Practice makesperfect? Managing and leveraging visual experiences forlifelong navigation. In Proc. of Intl. Conf. on Roboticsand Automation (ICRA), 2012.

[3] Michael Milford and Gordon F. Wyeth. SeqSLAM: Visualroute-based navigation for sunny summer days and stormywinter nights. In Proc. of Intl. Conf. on Robotics andAutomation (ICRA), 2012.

[4] Niko Sunderhauf and Peter Protzel. BRIEF-Gist – Closingthe Loop by Simple Means. In Proc. of IEEE Intl. Conf.on Intelligent Robots and Systems (IROS), 2011.

[5] Niko Sunderhauf, Peer Neubert, and Peter Protzel. AreWe There Yet? Challenging SeqSLAM on a 3000 kmJourney Across All Four Seasons. In Proc. of Workshopon Long-Term Autonomy, IEEE International Conferenceon Robotics and Automation (ICRA), 2013.