MIT6870_ORSU_lecture6: Scenes and objects

8/3/2019 MIT6870_ORSU_lecture6: Scenes and objects

http://slidepdf.com/reader/full/mit6870orsulecture6-scenes-and-objects 1/104

Lecture 6Scenes and objects

6.870 Object Recognition and Scene Understandinghttp://people.csail.mit.edu/torralba/courses/6.870/6.870.recognition.htm



Class business

Next Wednesday«



Week 2: Objects without scenes

Week 5: Scenes without objects

Week 6: Scenes and objects



Why is detection hard?

x

y

1,000,000 images/dayPlus, we want to do this for ~ 1000 objects

10,000 patches/object/image

time



Standard approach to scene analysis

1) Object representation based on intrinsic features:

Local

featuresno car

Classifier

p( car | VL )

2) Detection strategy:

Sky

Mountain

Buildings

cars

3) The scene representation



Is local information enough?



With hundreds of categories

roadtablechair

keyboardtablecar

road

If we have 1000 categories (detectors), and each detector produces 1 fa every 10

images, we will have 100 false alarms per image« pretty much garbage«



Is local information even enough?



Is local information even enough?

Distance

Information

Local featuresContextual features



We know there is a keyboard present in this scene even if we cannot see it clearly.

We know there is no keyboard present in this scene

« even if there is one indeed.

The system does not care about the

scene, but we do«



The multiple personalities of a blob



The multiple personalities of a blob









Look-Alikes by Joan Steiner









Why is context important?

Changes the interpretation of an object (or its function)

Context defines what an unexpected event is



The influence of an object extends beyond its physical boundaries



The context challenge

How far can you go withoutusing an object detector?



21

What are the hidden objects?



What are the hidden objects?

Chance ~ 1/30000





The importance of context

Cognitive psychology ± Palmer 1975

± Biederman 1981

± «

Computer vision ± Noton and Stark (1971)

± Hanson and Riseman (1978)

± Barrow & Tenenbaum (1978) ± Ohta, kanade, Skai (1978)

± Haralick (1983)

± Strat and Fischler (1991)

± Bobick and Pinhanez (1995)

± Campbell et al (1997)



Biederman 1972

Arrow appeared before or after picture.

Selected object from 4 pictures.







Biederman 1972

Better accuracy with normal scene and

with pre-cue.

C

oherence of surroundings affected objectperception.

But, jumbled pictures had unnatural edge

artifacts.



Palmer 1975

Scene preceded object to identify.

Better identification when preceded by a

semantically consistent scene.

Objects seen for 20, 40, 60 or 120 ms.



Palmer

Scenes shown ahead of time for 2 s.

More accurate recognition of consistent

objects than inconsistent objects. Similar looking objects were misnamed,

showing a bias effect.



Loftus & Mackworth

Inconsistent objects

fixated earlier and

longer.

Suggested additional

processing of objects

out of context. Similar results found

by Friedman (1979).



De Graef et al. 1990

Prior results due to memory task?

Measured eye movements during non-

object search task.





Object Detection

Biederman et al. 1982, relational violations





Biederman 1982

Pictures shown for 150

ms.

Objects in appropriate

context were detected

more accurately than

objects in aninappropriate context.

Scene consistency

affects object detection.



Objects and Scenes

Biederman¶s violations (1981):



Support

[Golconde Rene Magritte]



Interposition

[Blank Check Rene Magritte]





Position, Probability

[P ersonal Values Rene Magritte]



Object Consistencies

Biederman et al (1982), DeGraef(1990).



Object Consistencies

Examples of inconsistencies

Biederman et al (1982), DeGraef(1990).



Contextual cueing

Chun & Jiang, 1998



Object priming

Torralba, Sinha, Oliva, VSS 2001

Increasing contextual information



Object priming


bj i i



?Car, pedestrian, mailbox, «

Object priming

p(object | scene)


Obj i i



Object priming




Ex amples of consistent scenes (a), inconsistent scenes (b), and isolated objects and

backgrounds (c); from Davenport & P otter, 2004



But do we really need context?



Hollingworth & Henderson

Concerns with object detection studies

± Object label could bias results.

± Location cue selectively helpful for consistent

objects.

Controlled for false alarm biases with post-

cue and 2AFC.

Failed to find consistency effects.



Hollingworth & Henderson

Post-cue

2AFC with object

labels

± Both consistent or

inconsistent.

2AFC with tokendiscrimination.

± E.g. sports car or

sedan.











CONDOR systemStrat and Fischler (1991)

Guzman (S EE ), 1968

Noton and Stark 1971

Hansen & Riseman (VI S ION S ), 1978

Barrow & Tenenbaum 1978

Brooks ( ACRONYM ), 1979

Marr, 1982

Ohta & Kanade, 1978

Yakimovsky & Feldman, 1973

A A f S U d di



An Age of Scene Understanding

Guzman (S EE ), 1968 Noton and Stark 1971

Hansen & Riseman(VI S ION S ), 1978

Barrow & Tenenbaum 1978

Brooks ( ACRONYM ), 1979

Marr, 1982

Ohta & Kanade, 1978

Yakimovsky & Feldman, 1973

[Ohta & Kanade 1978]

C



Current approaches

1) Scene to object dependencies

2) Object to object dependencies

L l f



Levels of context

Context in low-level vision

Part-based models

Objects relations Long-range connections

Weak constraints

Multimodal

Fix graph structurescan be useful

approximations

C t h



Current approaches


2) Object to object dependencies



b t this co occ rrence has a hidden



« but this co-occurrence has a hidden

common ³cause´: the scene

streetsoffices

It is easier to first recognize the scene, then predict object presence, than

running local object classifiers

Th l d t t f



The layered structure of scenes

In a display with multiple targets present, the location of one target constraints the µy¶

coordinate of the remaining targets, but not the µx¶ coordinate.

Assuming a human observer standing on the ground



D t ti f ith t f d t t



Detecting faces without a face detector

Torralba & Sinha, 01; Torralba, 03

Context based vision system for place



Context-based vision system for place

and object recognition

Hidden states = location (63 values) Observations = vG

t (80 dimensions)

Transition matrix encodes topology of

environment Observation model is a mixture of

Gaussians centered on prototypes (100

views per place)

Office 610 Corridor 6b Corridor 6c Office 617

We use 17 annotated sequences for training

Torralba, Murphy, Freeman and Rubin. ICCV 2003

O bil i



Our mobile rig

Torralba, Murphy, Freeman, Rubin. 2003









An integrated model of Scenes



An integrated model of Scenes,

Objects, and Parts

Ncar

S

g

Scene

Scene

gist

features

0

0

1

1

5

5

N

P(Ncar | S = street)

P(Ncar | S = park)

0 5 10 1 50

0 .0 5

0. 1

0 .1 5

0. 2

0 5 1 0 1 50

0. 2

0. 4

0. 6

0. 8

N



Global to local



Global to local

Use global context to predict objects but there is nomodeling of spatial relationships between objects.

Op1,c1

vp1,c1

OpN,c1

vpN,c1. . .

Op1,c2

vp1,c2

OpN,c2

vpN,c2. . .

Class 1 Class 2

E1 E2

S

c2maxVc1

maxV

X1X2

vg

Keyboards

Murphy, Torralba & Freeman (03)

3d Scene Context



3d Scene Context

Image World

Hoiem, Efros, Hebert ICCV 2005

3d Scene Context



3d Scene Context

meters

m e t e

r s

Ped

Ped

Car

Hoiem, Efros, Hebert ICCV 2005

3D City Modeling using Cognitive Loops



3D City Modeling using Cognitive Loops

N.C

ornelis, B. Leibe, K.C

ornelis, L.V

an Gool.CV

PR'06

Current approaches



Current approaches


2)O

bject to object dependencies

Where should I put the silverware?



Where should I put the silverware?

Sampling from the labels








Cf. Hoiem et al; Hays, Efros. Siggraph 2007

Contextual object relationships



Contextual object relationshipsCarbonetto, de Freitas & Barnard (2004) Kumar, Hebert (2005)

Torralba Murphy Freeman (2004)

Fink & Perona (2003)E. Sudderth et al (2005)

Object-Object Relationships



Fink & Perona (NIPS 03)Use output of boosting from other objects at previous

iterations as input into boosting for this iteration


Pixel labeling using MRFs



Pixel labeling using MRFs

Enforce consistency between neighboring

labels, and between labels and pixels

Carbonetto, de Freitas & Barnard, ECCV¶04

Beyond nearest-neighbor grids



Beyond nearest-neighbor grids

Most MRF/C

RF models assume nearest-neighbor graph topology

This cannot capture long-distance

correlations

Dynamically structured trees



Dynamically structured trees

Each node pick its parents(Storkey& Williams, PAMI¶03)

2D SCFGs(Pollak, Siskind, Harper & Bouman IC ASSP¶03)




Object Object Relationships

Use latent variables to induce long distance correlations

between labels in a Conditional Random Field (CRF)

He, Zemel & Carreira-Perpinan (04)





[Kumar Hebert 2005]

Hierarchical Sharing and Context



Hierarchical Sharing and Context

Scenes share objects

Objects share parts

Parts share features

E. Sudderth, A. Torralba, W. T. Freeman, and A. Wilsky.

3d Scene Context



3d Scene Context

Image Support Vertical Sky

V-Left V-Center V-Right V-Porous V-Solid

[Hoiem, Efros, Hebert ICCV 2005]

Object

Surface?

Support?

Detecting difficult objects




Office Maybethere is

a mouse

Start recognizing the scene

Torralba, Murphy, Freeman. NIPS 2004.







Detect first simple objects (reliable detectors) that provide strong

contextual constraints to the target (screen -> keyboard -> mouse)

Torralba, Murphy, Freeman. NIPS 2004.

BRF for screen/keyboard/mouse




Iteration





Iteration





Iteration





Iteration





Iteration

BRF for car detection: topology



BRF for car detection: topology

BRF for car detection: results



BRF for car detection: results

A ³car´ out of context is less of a car



A car out of context is less of a car

Car Building Road

b F G b F G b F G

From image

From detectors

Thresholded beliefs



Contextor no context

Documents

MIT6870_ORSU_lecture6: Scenes and objects