Computer Science Department

Learning on the Fly:Rapid Adaptation to the Image

Erik Learned-Millerwith Vidit Jain, Gary Huang,

Laura Sevilla Lara, Manju Narayana, Ben Mears

2Learning on the Fly

“Traditional” machine learning Learning happens from large data sets

• With labels: supervised learning• Without labels: unsupervised learning• Mixed labels: semi-supervised learning,

transfer learning,learning from one (labeled) example,self-taught learning,domain adaptation

3Learning on the Fly

Learning on the Fly Given:

• A learning machine trained with traditional methods• a single test image (no labels)

Learn from the test image!

4Learning on the Fly

Learning on the Fly Given:

• A learning machine trained with traditional methods• a single test image (no labels)

Learn from the test image!• Domain adaptation where the “domain” is the new

image• No covariate shift assumption.• No new labels

5Learning on the Fly

An Example in Computer Vision Parsing Images of Architectural Scenes

Berg, Grabler, and Malik ICCV 2007.• Detect easy or “canonical” stuff.• Use easily detected stuff to bootstrap models of harder

stuff.

6Learning on the Fly

Claim This is so easy and routine for humans that it’s

hard to realize we’re doing it.• Another example…

7Learning on the Fly

Learning on the fly…

8Learning on the Fly

Learning on the fly…

9Learning on the Fly

Learning on the fly…

10Learning on the Fly

What about traditional methods… Hidden Markov Model for text recognition:

• Appearance model for characters• Language model for labels• Use Viterbi to do joint inference

11Learning on the Fly

What about traditional methods… Hidden Markov Model for text recognition:

• Appearance model for characters• Language model for labels• Use Viterbi to do joint inference

DOESN’T WORK!

Prob( |Label=A) cannot be well estimated, fouling up the whole process.

12Learning on the Fly

Lessons We must assess when our models are broken,

and use other methods to proceed….• Current methods of inference assume probabilities are

correct!• “In vision, probabilities are often junk.”• Related to similarity becoming meaningless beyond

a certain distance.

13Learning on the Fly

2 Examples Face detection (CVPR 2011) OCR (CVPR 2010)

14Learning on the Fly

Preview of results: Finding false negatives

Viola-Jones Learning on the Fly

15Learning on the Fly

Eliminating false positives

Viola-Jones Learning on the Fly

16Learning on the Fly

Eliminating false positives

Viola-Jones Learning on the Fly

17Learning on the Fly

Run a pre-existing detector...

18Learning on the Fly

Run a pre-existing detector...

Key

Face

Non-face

Close to boundary

19Learning on the Fly

Gaussian Process Regression

negative positive

learn smooth mappingfrom appearance to score

apply mapping to borderline patches

20Learning on the Fly

Major Performance Gains

21Learning on the Fly

Comments No need to retrain original detector

• It wouldn’t change anyway! No need to access original training data Still runs in real-time GP regression is done for every new image.

22Learning on the Fly

Noisy Document

We fine herefore tlinearly rolatcd to thewhen this is calculated equilibriurn. In short,on the null-hypothesis:

Initial Transcription

23Learning on the Fly

Premise We would like to fine confident words

to build a document-specific model,but it is difficult to estimate Prob(error).

However, we can bound Prob(error). Now, select words with

• Prob(error)<epsilon.

24Learning on the Fly

“Clean Sets”

25Learning on the Fly

Document specific OCR Extract clean sets (error bounded sets) Build document-specific models from clean set

characters Reclassify other characters in document

• 30% error reduction on 56 documents.

26Learning on the Fly

Summary Many applications of learning on the fly. Adaptation and bootstrapping new models is

more common in human learning than is generally believed.

Starting to answer the question: “How can we do domain adaptation from a single image?”