Thrust IIA: Environmental State Estimation and Mapping

Thrust IIA: Environmental State Estimation and Mapping

Dieter Fox (Lead)

Nicholas Roy

MURI 8Kickoff Meeting 2007

Task Objective: Human-Centered Maps

Observation: Automatic map-building (SLAM) is solved

sufficiently well

Goal: Describe environments by higher-level concepts:

Places (room, hallway, street, walkway, parking lot, …)

Objects (tree, person, building, car, wall, …)

Key challenges:

Estimating concept types is mostly a discrete problem

Complex features and relationships

MURI 8 Kickoff Meeting 2007

University of Washington

Existing Technology

Human-centered mapping requires integration of high-dimensional, continuous

features from multi-modal sensor data reasoning about spatial and temporal

relationships

Conditional Random Fields provide extremely flexible probabilistic framework for learning and inference

MURI 8 Kickoff Meeting 2007

University of Washington

Conditional Random Fields

Discriminative, undirected graphical model

Introduced for labeling sequence data to overcome weaknesses of

Hidden Markov Models [Lafferty-McCallum-Pereira: ICML-01]

Applied successfully to

Natural language processing [McCallum-Li: CoNLL-03], [Roth-Yih: ICML-05]

Computer vision [Kumar-Hebert: NIPS-04], [Quattoni-Collins-Darrel: NIPS-05]

Robotics [Limketkai-Liao-Fox: IJCAI-05], [Douillard-Fox-Ramos: IROS-07]

MURI 8 Kickoff Meeting 2007

University of Washington

Conditional Random Fields

Directly models conditional probability p(x|z)

(instead of modeling p(z|x) and p(x), and using Bayes rule to infer p(x|z)).

No independence assumption on observations needed!

k 1xHidden states x

Observations z

MURI 8 Kickoff Meeting 2007

University of Washington

Online Object Recognition

MURI 8 Kickoff Meeting 2007 [Douillard-Fox-Ramos: IROS-07, ISRR-07]

From Laser Scans to CRFs

MURI 8 Kickoff Meeting 2007

Object type of laser beam 1

Shape and appearance

Object type of laser beam 2

Object type of laser beam 3

Object type of laser beam 4

Object type of laser beam n

Shape and appearance

…

Temporal Integration

MURI 8 Kickoff Meeting 2007

…………

k-2 k-1 k k+1

Taking past and future scans into account can improve labeling accuracy.

Match consecutive laser scans using ICP.

Associated laser points are connected in CRF.

Can perform online filtering or offline smoothing via BP.

Example Trace: Car vs. Others

MURI 8 Kickoff Meeting 2007

Trained on 90 labeled scans

Inference via filtering in CRF

7 Class Example Labeling

MURI 8 Kickoff Meeting 2007

Proposed Technical Advances

Integrate recognition results into maps Improve results by leveraging web training

data and high level object detectorss Add object types suited for target scenario Improve CRF training

MURI 8 Kickoff Meeting 2007

University of Washington

Situation Awareness via Wearable Sensors

Microphone Camera

Light sensors

2 GB SD card

Indicator LEDs

Records 4 hours of audio, images (1/sec), GPS, and sensor data (accelerometer, barometric pressure, light intensity, gyroscope, magnetometer)

Soldier Activity Recognition

Automatic generation of mission summaries Motion type (linger, walk, run, drive, …) Environment (inside, outside building) Events (conversations, marked via keyword)

Technical challenges High-dimensional, continuous observations /

features Different data rates: (1 Hz - 256 Hz) Getting labeled training data Different persons / environments

MURI 8 Kickoff Meeting 2007

Data Visualization / Summarization

MURI 8 Kickoff Meeting 2007

GPS traces

Image sequence(currently in car)

Timeline of soldier activities

Milestones

Goals: Real time wearable interface on cell phone Data sharing among soldiers and robots Real time display on remote laptop

MURI 8 Kickoff Meeting 2007

Milestones

Year 1: Real time data sharing between wearable sensor

platforms Integration of object recognition into mapping

Year 2: Real time data sharing between soldiers, robots,

and remote laptop Detection of specific soldier states / activities

(moving, incapacitated, ...)