Transfer Learning of Object Classes: From Cartoons to Photographs

NIPS WorkshopInductive Transfer: 10 Years Later

Geremy HeitzGal Elidan

Daphne Koller

December 9th, 2005

Localization vs. Recognition

Traditional question:

“Is there an object of type X in this image?”

Airplane? NO

Human? YES

Dog? YESOur question:

“Where in this image is the object of type X?”

MAN

DOG

The man is walking the dog

Outline

Landmark-based shape model Localization as inference Transfer learning from cartoon

drawings Results

Shape Model

Set of landmarks Piecewise-linear contour between

neighbors Features of individual landmarks Features of pairs of landmarks

tail

nose

Outline

Landmark-based shape model Localization as inference Transfer learning from cartoon

drawings Results

“Registering” the Model to an Image

Requires assigning each landmark to a pixel location

??

Localization

Are local cues enough?

Need to jointly consider all cues (features)

“Correct” pixel is often not the best match!Markov Random Field

• Potentials = Functions of local and global features

Registration = Most Likely Assignment

Lnose Ltail

LunderLcockpit

Outline

Landmark-based shape model Localization as inference Transfer learning from cartoon

drawings Results

Bootstrap from simple instanceswhere outlining is easy = cartoons / drawings

Learning ChallengeHand Label Hidden Variables

Costly, and time-consuming

Where to start?Local optima problem

no confusing background

outline (shape) is easily recovered using snake

??

???

Learning from Cartoon Drawings

Registration

Shape Learning

Shape and Appearance Learning

+

Phase I: Learning from Cartoons

Extract high resolution contour using snake Create shape-based model from training

contours Pairwise merging of models Selection of landmarks

Registration PyramidFinal Shape

Model

Training Set Selection

high score

low score

Phase II: Learning from Images

Correspond initial model to training images Select best correspondences as training instances Learn final shape- and appearance-based model

Cartoon PhaseModel

Natural ImageModel

Transfer

Outline

Landmark-based shape model Localization as inference Transfer learning from cartoon

drawings Results

Localization Results

0.84 0.75 0.84 0.72 0.18

0.81 0.81 0.66 0.77 0.40

sampletrainingcartoons

sampleregistration

Transfer of Object Shape

Transfer of shape speeds up learning

Benefit of shape

transfer

0 2 4 6 8 10

0

0.1

0.2

0.3

0.4

0.5

0.6

# images in phase II

Ave

rag

e o

verl

ap

transfer

no transfer

Learning Appearance

No Appearance

FG/BG Appearance

0 2 4 6 8 100.46

0.48

0.5

0.52

0.54

0.56

0.58

0.6

0.62

0.64

Ave

rag

e o

verl

ap

Shape template

shape + appearance

# images in phase II

Training Instance Selection

AUTO PICKED

0 2 4 6 8 10 120.3

0.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

AUTO

PICKED

HAND

Ave

rag

e o

verl

ap

# images in phase II

Summary and Future Work Flexible probabilistic shape model Effective registration to images Transfer

Shape from cartoons Appearance from real images

Develop a better appearance model Investigate self-training issues Transfer from one class to another

Thanks!

Cartoon vs. Hand Segmentation

0 1 2 3 4 5

Number of Training Instances

0.1

0.3

0.5

0.7

0.9

Mea

n O

verla

p S

core

Learned from Drawings

Hand Constructed

Human Inter-Observer

cartoon handsegmented

Learning shape from cartoons is competitive with hand segmentation!

Landmark Features Shape Template

Patch Appearance (Foreground/Background)

Location

Prediction

0 0.2 0.4 0.6 0.8 1

False positive rate

0

0.2

0.4

0.6

0.8

1

Tru

e p

osi

tive

rat

e object recognition

car side 86%

cougar 86%

airplane 86%

buddha 84%

bass 76%

rooster 73%

Comparable to constellation w/ 5 instances (Fei Fei et. Al)

Leading (discriminative) methods require many instances