Sensing and Sensibility: a quest to visual intelligence

Silvio Savarese

May 17th, 2016

New Frontiers in Computing

Sensing is the future

4

Sensing is the future

5

Sensing is the future

6

Sensing is the future

Sensing is the future

Everything is a sensor…

Everything is a sensor…

Everything is a sensor…

Modern vision sensors

night

Kinect

thermal

w/ gravity

Sensing is not the hard problem

Intelligent understanding of the sensing data is the challenge!

What does it mean “intelligent understanding of the sensing data”?

car

car

Car

Street

Building

image labels

Image-to-labels

Turk & Pentland, 91Poggio et al., 93Belhumeur et al., 97LeCun et al. 98Amit and Geman, 99Shi & Malik, 00Viola & Jones, 00Felzenszwalb & Huttenlocher 00Belongie & Malik, 02Ullman et al. 02

Argawal & Roth, 02Ramanan & Forsyth, 03Weber et al., 00Vidal-Naquet & Ullman 02Fergus et al., 03Torralba et al., 03Vogel & Schiele, 03Barnard et al., 03Fei-Fei et al., 04Kumar & Hebert ‘04

He et al. 06Gould et al. 08Maire et al. 08Felzenszwalb et al., 08Kohli et al. 09L.-J. Li et al. 09Ladicky et al. 10,11Gonfaus et al. 10Farhadi et al., 09 Lampert et al., 09

But…

• It is just one ingredient of a much more complex problem.

14

15

Chocolate bar

Tree

Road

Sky

Wheels

16

Sky

Chocolate bar

Tree

Road

Wheels

18

Car-right

Car-right

Car-left

Car- 3/4 right

toy car

Building facade

Road

Car-3/4 right

Road

19

Building facade

20

21

22

chasing

Intelligent understanding of sensory data is not just a labeling problem!

It implies recognizing objects, their physical properties and their relationship with the environment within which the objects live.

24

Biederman, Mezzanotte and Rabinowitz, 1982

25

Humans perceive the world in 3D

V1

where pathway(dorsal stream)

what pathway(ventral stream)

26

Humans perceive the world in 3D

V1Pre-frontal

cortex

27

Humans perceive the world in 3D

where pathway(dorsal stream)

what pathway(ventral stream)

My group’s research

Space understanding

• 3D shape recovery • 3D scene reconstruction • Camera localization• Pose estimation

Recognition

• Object detection• Texture classification• Target tracking• Activity recognition

28

• Objects

• Space

• Activities

From sensing intelligence• Images

• Videos

• RGB-Depth

From sensing to the 3D objects

CHAIR

BED

TABLE

Xiang & Savarese, 2012-2014

CAR

30

Car Person Tree Sky

Street Building Else

From sensing to the 3D scenes

…Bao & Savarese, 2011-2013

31

Car Person Tree Sky

Street Building Else

… Bao

& S

avar

ese,

20

11

From sensing to the 3D scenesBao & Savarese, 2011-2013

Interactions between:- Objects-space

- Object-object- Object-scene class

From sensing to the 3D scenes

32

Choi, Chao, Pantofaru, Savarese, CVPR 13

From 3D point clouds to 3D dynamic scenes

33

Held, Thrun & Savarese, RSS 2014

• Interactions among humans

34

queuing

talking

From sensing to activities

• Interaction human-3D space

Choi et al., VSWS 09Choi et al., CVPR 11Choi & Savarese, ECCV 2012

Queuing

Crossing streetChoi, Savarese, 2012-2014

From sensing to activities

Sensors

Objects

Sensing and sensibility paradigm

3D physical environment

Applications

37

Social Robotics

Mobile vision

Large scale information management

Safe driving

Visual intelligence and large scale information management

Golparvar-Fard, Pena-Mora, Savarese , 2008-2012

James R. Croes Medal, October 2013 (from the American Society of Civil of Engineers)

Automatic coordination of construction progress can lead to huge savings (10 billions USD/year) in construction business!

[Census Bureau, www.census.gov, 2007]

39

Opportunity to modernize age-old process in a profound way

freeing up critical human resources

12/02/2006; 1:13:00 PM (As-built)

Our revolution

- Thousands of images- Computer vision

Our revolution

Images are cheap!

42

Our revolution

Our revolution

12/02/2006; 1:13:00 PM (As-built)

Our revolution

45

Behind ScheduleOn ScheduleAhead of Schedule

12/02/2006; 1:13:00 PM (As-planned)

12/02/2006; 1:13:00 PM (As-planned)12/02/2006; 1:13:00 PM (As-built)

46

Behind ScheduleOn ScheduleAhead of Schedule

12/02/2006; 1:13:00 PM (As-planned)

12/02/2006; 1:13:00 PM (As-planned)12/02/2006; 1:13:00 PM (As-built)

Large impact in the civil engineering community

47

• James R. Croes Medal, October 2013 (from the American Society of Civil of Engineers)• Best paper award from journal of CEM, 2011 • Best paper award at AEC/FM 2010• Best paper award at Construction Research Congress 2009

• Better tools for visualization• Automatic communication of performance deviations• Reduction in delivery time• Safety management• Potential to identify unsafe locations/components

Applications

48

Social Robotics

Mobile vision

Large scale information management

Safe driving

The Jackrabbot

49An Autonomous Perambulating Robot for On-campus Deliveries

The Jackrabbot project

Social affinity

Social affinity

The Jackrabbot project

The Jackrabbot project

Thank you!

TOYOTA

53

54