30
Interactively Discovery of Attributes Vocabulary Devi Parikh and Kristen Grauman

Interactively Discovery of Attributes Vocabulary

  • Upload
    cadee

  • View
    28

  • Download
    0

Embed Size (px)

DESCRIPTION

Interactively Discovery of Attributes Vocabulary. Devi Parikh and Kristen Grauman. Traditional Recognition. Dog. Chimpanzee. Tiger. ???. Attributes-based Recognition. Furry White. Black Big. Stripped Yellow. Stripped Black White Big. Dog. Chimpanzee. Tiger. Applications. - PowerPoint PPT Presentation

Citation preview

Page 1: Interactively Discovery of Attributes Vocabulary

Interactively Discovery of Attributes Vocabulary

Devi Parikh and Kristen Grauman

Page 2: Interactively Discovery of Attributes Vocabulary

Traditional Recognition

Dog Chimpanzee Tiger ???

Page 3: Interactively Discovery of Attributes Vocabulary

Attributes-based Recognition

FurryWhite

BlackBig

StrippedYellow

StrippedBlackWhite

BigTigerChimpanzeeDog

Page 4: Interactively Discovery of Attributes Vocabulary

Applications

Zebra

A Zebra is…WhiteBlack

Stripped

Zero-shot learning

Image description

StrippedBlackWhite

Big

Attributes provide a mode of

communication between humans and

machines!

Page 5: Interactively Discovery of Attributes Vocabulary

Attributes

Attributes are most useful if they are• Discriminative• Nameable

Approaches Discriminative Nameable

Page 6: Interactively Discovery of Attributes Vocabulary

Attributes

Attributes are most useful if they are• Discriminative• Nameable

Approaches Discriminative NameableHand-

generatedMaybe not Yes

Page 7: Interactively Discovery of Attributes Vocabulary

Attributes

Attributes are most useful if they are• Discriminative• Nameable

Approaches Discriminative NameableHand-

generatedMaybe not Yes

Mining the web Maybe not Yes

Page 8: Interactively Discovery of Attributes Vocabulary

Attributes

Attributes are most useful if they are• Discriminative• Nameable

Approaches Discriminative NameableHand-

generatedMaybe not Yes

Mining the web Maybe not YesAutomatic splits Yes Maybe

not

Page 9: Interactively Discovery of Attributes Vocabulary

Attributes

Attributes are most useful if they are• Discriminative• Nameable

Approaches Discriminative NameableHand-

generatedMaybe not Yes

Mining the web Maybe not YesAutomatic splits Yes Maybe

notProposed Yes Yes

Page 10: Interactively Discovery of Attributes Vocabulary

Interactive System1. Name: Fluffy2. Name: x3. Name: Metal…

How do we show the user a candidate-attribute?How do we ensure proposals are discriminative?

How do we ensure proposals are nameable?

Page 11: Interactively Discovery of Attributes Vocabulary

Attribute Visualization

Page 12: Interactively Discovery of Attributes Vocabulary

Attribute Visualization

Page 13: Interactively Discovery of Attributes Vocabulary

Ensure Discriminability

Normalized cuts

Max Margin Clustering

Page 14: Interactively Discovery of Attributes Vocabulary

Ensure Nameability1. Name: Fluffy2. Name: x3. Name: Metal…

Page 15: Interactively Discovery of Attributes Vocabulary

Ensure Nameability1. Name: Fluffy2. Name: x3. Name: Metal…

Mixture of Probabilistic PCA

Page 16: Interactively Discovery of Attributes Vocabulary

Interactive System

Page 17: Interactively Discovery of Attributes Vocabulary

Evaluation

• Outdoor Scenes • Animals with Attributes• Public Figures Face

• Gist and Color features (LDA)

Page 18: Interactively Discovery of Attributes Vocabulary

Interactive System

Page 19: Interactively Discovery of Attributes Vocabulary

Evaluation

• Annotate all candidates off-line

“Black”

… ~25000 responses

Page 20: Interactively Discovery of Attributes Vocabulary

Evaluation

• Annotate all candidates off-line

“Spotted”

… ~25000 responses

Page 21: Interactively Discovery of Attributes Vocabulary

Evaluation

• Annotate all candidates off-line

Unnameable

… ~25000 responses

Page 22: Interactively Discovery of Attributes Vocabulary

Evaluation

• Annotate all candidates off-line

“Green”

… ~25000 responses

Page 23: Interactively Discovery of Attributes Vocabulary

Evaluation

• Annotate all candidates off-line

“Congested”

… ~25000 responses

Page 24: Interactively Discovery of Attributes Vocabulary

Evaluation

• Annotate all candidates off-line

“Smiling”

… ~25000 responses

Page 25: Interactively Discovery of Attributes Vocabulary

Results

Our active approach discovers more discriminative splits than baselines

Structure exists in nameability space allowing for prediction

Page 26: Interactively Discovery of Attributes Vocabulary

Results

Comparing to discriminative-only baseline

Page 27: Interactively Discovery of Attributes Vocabulary

Results

Comparing to descriptive-only baseline

Page 28: Interactively Discovery of Attributes Vocabulary

ResultsAutomatically generated descriptions

Page 29: Interactively Discovery of Attributes Vocabulary

Summary

• Machines need to understand us– Attributes need to be detectable & discriminative

• We need to understand machines– Attributes need to be nameable

• Interactive system for discovering attributes

Page 30: Interactively Discovery of Attributes Vocabulary

Thank you.