Interactive Discovery and Semantic Labeling of

Patterns in Spatial Data

Thomas Funkhouser, Adam Finkelstein,David Blei, and Christiane Fellbaum

Princeton University

FODAVA Review MeetingDecember 2009

Motivation

Lots of 3D data is available with spatial patterns that reveal semantic information

Archeology

Climate Simulation

Molecular Biology

Example: Lidar Scans of Cities

Ottawa

Goal

Discover spatial patterns in 3D data to assist semantic segmentation and labeling

Related Work

Specific Objects• [Chen and Chen 07]

Buildings, trees, etc.

Markov Random Fields• [Anguelov et al. 05]

Point labeling

Segment, label• [Secord and Zakhor 07]• [Carlberg et al. 09]• [Golovinskiy et al. 09]

Current Approach

Supervised learning:• Training data

Locate Segment Describe Build classifier

• New data Locate Segment Describe Apply classifier Training

Area

Current Approach

Supervised learning:• Problems

Training data only usefulif matches new data

Trainer prescribessemantic classes

Trainer must labelenough training datato cover all possiblenew data

TrainingArea

Problems I

Raw data may be difficult to segment automatically into semantic objects

Problems II

Local spatial patterns may not be descriptive enough to assign semantic labels

What is this?

Problems III

Spatial patterns/features for objects of same typemay be different in different data sets

Problems IV

Semantic objects of interest may be different for different users

• What areas of city have too few street lights?• What is spacing between fire hydrants?• Where should trees be planted? • Where could a terrorist could hide a bomb?• Where do people park?

Another Possible Approach

Active learning:• Off-line

Locate Segment Describe

• On-line System

builds classifierby requesting user labels

Another Possible Approach

Active learning:• Problems

Computer drivestraining process – tries to learn user’ssemantic model

Usually classes are pre-specified

Discrete sequenceof visual recognition tasks – if jump from example to example

Our Approach

User-driven learning:• Off-line

Locate Segment Describe

• On-line User interactively

adjusts segments and labels on data

System builds clusters, classifiers, and provides visual feedback

Our Approach

User-driven learning:• Advantages:

System can guideuser towards most usefulinput with visualization

User drives process – can focus on whathe/she cares about

User can create/removeclasses during labelingprocess

Continuous visualrecognition, easiersince camera is controlled by user

Main Challenge

User-driven learning:• Integrate off-line analysis with unsupervised learning

with interactively updated probabilistic inference modelwhile providing interactive visual feedback

User

Class labels

First Steps

Outline

Introduction

User-driven learning

Specific research issues• Segmentation• Shape description• Pattern discovery• Visualization

Wrap up

Outline

Introduction

User-driven learning

Specific research issuesSegmentation• Shape description• Pattern discovery• Visualization

Wrap up

Segmentation

Problem: • Clustering points into semantic objects

Segmentation

Current approach: • Hierarchical clustering to find candidate clusters• Min-cut separation of foreground from clutter

SegmentationProximity graphInput data

Segmentation

Current approach: • Hierarchical clustering to find candidate clusters• Min-cut separation of foreground from clutter

Segmentation

Current results:

Segmentation

Current results:

Segmentation

Future challenges:1) Provide interactive, adaptive segmentation tools2) Integrate segmentation with recognition3) Integrate segmentation with inference

Outline

Introduction

User-driven learning

Specific research issues• SegmentationShape description• Pattern discovery• Visualization

Wrap up

Shape Description

Problem: • Describe cluster of points by a feature vector that

discriminates its semantic class

Shape Description

Current approach: • Shape features (volume, eccentricity, …)• Shape descriptors (spin images, shape contexts, …)• Contextual cues (distances to other objects)

Spin Image

Shape Description

Current results:

Training Area Testing Area

Shape Description

Current results:

Shape Description

Proposed approach: • Data-adaptive dictionaries -- adaptable filters designed to

discriminate user selected object types

Blue = positive, Red = negative Feature vector

. . .

-0.9 0.8 1.0 0.8

Adaptable filters

QueryShape

Shape Description

Future challenges: • Computational representation for adaptable descriptors• Efficient adaptation of descriptors as user labels examples• Interactive user guidance in refinement of descriptors

Outline

Introduction

User-driven learning

Specific research issues• Segmentation• Shape descriptionPattern discovery• Visualization

Wrap up

Pattern Discovery

Goal:• Recognize spatial patterns and use them to

segment and label clusters of points

Pattern Discovery

Current approach:• Learn probabilistic representation of symmetries

and use it to predict labels

Input data Marked lampposts Symmetry transform(probabilistic model of translational symmetry)

Pattern Discovery

Current results:• Adding probabilistic symmetry as a feature

helps recognition (by a little)

Symmetry

Pattern Discovery

Future challenges:1) Better representations for symmetries

and spatial relationships2) Integrate symmetries and spatial relationships

into probabilistic inference model3) Interactive specification and visualization

of symmetries and spatial patterns

Symmetry transform(probabilistic model of translational symmetry)

Outline

Introduction

User-driven learning

Specific research issues• Segmentation• Shape description• Pattern discoveryVisualization

Wrap up

Visualization

Goals:• Provide interactive displays to help user understand …

Input data Specified segments and labels Inferred segments and labels Value of further input Computational models

Visualization

Proposed approach:• Multiple views

3D space Feature space Symmetry space

Symmetry Space3D Space

Feature Space

Visualization

Future challenges:• Provide methods to …

Integrate multiple views Represent uncertainty Guide user input Reduce clutter

Other Applications

Molecular Biology

Paleontology Climate

Archeology

Wrap Up

Goal: • Segment and label patterns in 3D data

Approach: user-driven learning• User interactively guides segmentation and labeling• System learns model and provides visual feedback

Research challenges: user-driven …• Segmentation• Shape description• Pattern discovery• Inference• Visualization

Acknowledgments

Students:• Former: Alex Golovinskiy• Current: Aleksey Boyko, Vladimir Kim

Other funding sources:• Former: AFRL, URGENT• Current: NSF, Google