Vision-Based Place Recognition and Cognitive Mapping Hanspeter A. Mallot Faculty of Biology and Werner-Reichardt-Center for Integrative Neuroscience University

Vision-Based Place Recognition and Cognitive Mapping

Hanspeter A. Mallot

Faculty of Biology and

Werner-Reichardt-Center for Integrative Neuroscience

University of Tübingen

Fakultät für Biologie, University of Tübingen

Spatial knowledge and problem solving: The "Means-Ends-Field"

E.C. Tolman, Purposive Behavior in Animals and Men, The Century Comp. 1932, p 177

Nodes Places, goal,

intermediate goals Sensory cues

characterizing places States

Edges Actions "Means-Ends-Expectations" Schemata <s,r,s'>

Edward C. Tolman (1886-

1959)

Place Recognition


Landmarks and Places

Localized cues approach

Landmarks are object features that have a position (coordinates) in the environment.

Snapshot approach

Landmarks are image features that characterize the place from which they are viewed. They need not correspond to objects in the world (view axes!).

"Four lakes view " of the river Rhein close to Boppard.


Snapshot Homing Accuracy

xy

sensor 1

senso

r 2

sens

or 3

Image manifold I(u,v;x,y,

Local image variance

2222 ,liv

y

I

x

Ι

y

Ι

x

Ιyx

Catchment area: homing possible

Area of uncertainty: no further approach possible

Size of both areas proportional to local image variance


Circular Color Texture

)0),((),(

)sincos(sin

)(

121

p

p

TIpI

pp

T

x

),( 21 pp

y

2,1,0

3cos1

2

1)(

i

icfi

Sinusoidal intensity modulation per color channel

Image manifold


Snapshot-based homing in humans

Subject with HMD in 5.2 x 6 m tracked walking arena

Circular room with homogeneous color gradient

Task: Subject at position 1 View scene at

position 2 Walk to position 2 View scene at

position 3 …

Dependent measure: trajecory, homing error


Sample Trajectories and Viewing Directions

a-d: subjects head towards goal from the beginning

right: heading directions top: small room, subject quickly finds goal

directions bottom: large room, subjects looks around

and then starts moving towards the right direction.

General performance is good. Residual homing error is well below chance level.

Gillner S, Weiß A, Mallot HA, Cognition


Place recognition models

i i

ii

g

pcgcpgS

2

2

)(

||)()(||

2

1exp:),(

• g: goal

• p: current place

• c: place code (vector of local position information)

• S(g,p): comparison function (radial basis function)

• confusion area }),(|{)( pgSpgN


Image Comparison Model: c(p) = I(ψ;p)

tangential var

radial variance

• Circular confusion areas

• Size decreases with l.i.v.

• Weak dependence on eccentricity


Closest Wall Segment Model: c(p) = (1-|p|,I(arg(p))

tangential var

radial variance

• Shape of confusion areas depends on l.i.v.

• Size decreases with contrast

• Size depends on eccentricity


Boundary Vector Model:

tangential var

radial variance

• Confusion areas do not depend on contrast

• Shape depends on eccentricity


Dependence on Color Modulation Gillner S, Weiß A, Mallot HA, submitted

Color modulation 10%

Color modulation 100%

homing in 6 subjects,

4 repetitions

prediction from

squared image

difference algorithm


Dependence on Color Modulation Gillner S, Weiß A, Mallot HA, submitted

Overall signifi-cant effect of color modulation

Per point, effect is significant only for the three peripheral points

Threshold effect for higher modulations

Model Prediction

Experimental Data


Dependence on Room Size Gillner S, Weiß A, Mallot HA, submitted

Human visual homing in featureless environment depends on contrast and room size, i.e. on local image variation, l.i.v.

S room (diameter 4.5 m)

XL room (diameter 27 m)

Model Prediction

Experimen-tal Data

Metric Embedding of Place Graphs


path integration metric embedding

Metric Embedding of Place Graphs

place recognition by panoramic view comparison view graph for topological navigation egomotion estimates from odometry or optical flow metric embedding of view graph by modified MDS

actual positio

n

represented

position


vector product:match triangle

area

MDS and Metric Embedding

x34x12

x23

Measurement D

x3

x1

x2

Embedding X Optimization Q

x4

ikj

jkijjkji xxxxxx 2()()(

),( DXQ

dot product: match leg projection

ikj

jkijjkji xxxxxx 2()()(

Hübner & Mallot, Autonomous Robots 2007


residual error route planning on subgraph

Embedded Place Graph



Path integration error corrected by visual homing

Replanning after getting lost at an obstacle

Route following



Metric Knowledge in Human Longterm Memory

Foo P, Warren WH, Duchon A, Tarr MJ. J Exp. Psychol: LMC 31:195-215,2005Little evidence for metric embedding of locally learnt segments.


Metric Knowledge in Human Longterm Memory

Foo P, Warren WH, Duchon A, Tarr MJ. J Exp. Psychol: LMC 31:195-215,2005Little evidence for metric embedding of locally learnt segments.

Ishikawa T, Montello D, Cognitive Psychology 52:93-129 (2006)No improvement of metric performance over sessions.

Distance estimates


Scaleable model of spatial memory: The view-graph approach

Place recognition

Topological navi-gation: Routes (chains) and maps (graphs)

Local metric information

Metric embedding?

Regions and route planning


Cognitive Neuroscience Lab

Rodent behaviour Johannes Thiele Alexandar Jovalekic Okuary Osechas Phillip Schwedhelm Berna Ertas Martin Seitz

Human behaviour Gregor Hardiess Dagmar Schoch Wolfgang Röhrig Stephan Storch Geraldine Hopf

Robots and Models Chunrong Yuan Kai Basten Fabian Recktenwald Stefan Blazcek Deniz Bahadir Isabelle Schwab Hansjürgen Dahmen

Staff Heinz Bendele Martina Schmöe-Selich Annemarie Kehrer

EU ESTPerAct

Documents

Vision-Based Place Recognition and Cognitive Mapping Hanspeter A. Mallot Faculty of Biology and Werner-Reichardt-Center for Integrative Neuroscience University