Adding Unlabeled Samples to Categories by Learned Attributes

Jonghyun ChoiMohammad Rastegari

Ali FarhadiLarry S. Davis

PPT Modified By Elliot Crowley

Constructing Training Set is Expensive

• Human labeling is expensive• Hard to find training samples for some

categories– Heavy-tailed distribution of training set

• Solutions– Semi-supervised learning– Active learning

Assume that unlabeled data are drawn from the same

distribution of labeled data

Require humans in the loop

Adding Unlabeled Data Without .modeling a distributionhumans in the Loop

Number of training samplesIn SUN 09 Dataset

Salakhutdinov, Torralba, Tenenbaum (CVPR’11)

Long Tail Problem

• Training sets can have lots of objects in one pose

• f

• But not another.

Expanding Training Set by Attribute• Given small number of labeled training samples• Expanding visual boundary of a category to build a

better visual classifier

Given initial training set

Large UnlabeledImage Pool

Find bylow-level feature

Find byattributes

Similar to specific sample images

Dotted

whitishColor

Animal-like shape

Added back

From web

Why Attribute?• Low-level features are– Specific to shape, color, pose of given seeds– Expanding the visual boundary to the known region

• Add similar examples to the current set

• Attributes are– More general mid-level description than low-level

description (e.g., dotted)– Able to find visually unseen examples but

maintaining traits of the current set

Why Attributes Again?

• Not necessarily needed.

• Training set should have at least one example of each pose.

• Remainder are found by Exemplar method. (The strong point of this paper)

How to Add Samples by Attributes?

• Given candidate attributes– Pre-learned by an auxiliary data

• By learning– A discriminative set of attributes

• Find samples confident in combinations of attributes

Formulation• A joint optimization for– Learning classifier in visual feature space (wca)– Learning classifier in attribute space (wcv)– With finding the samples (I)

• Non-convex– Mixed integer program: NP-complete problem– Solution: Block coordinate-descent

Learning a classifier on visual feature space

Learning a classifier on attribute spacewith a selection criterion

Mutual ExclusionNot convex

discrete continuous

Formulation Detail

Classifier on visual feature space

Classifier on attribute space

Mutual Exclusion

Max-margin Classifieron visual feature space

Max-margin Classifieron attribute spacew/ Top-Lambda Selector

Mutual Exclusion

AttributeMapper

How to Build the Attribute Mapper• Candidate attribute set• Labeling attribute by human is expensive• Automatic attribute space generation by [1]– Learn in offline with any labeled set available on

the web• Attributes essentially reduce

feature space into binarydecision boundaries.

[1] Mohammad Rastegari, Ali Farhadi, David Forsyth, “Attribute Discovery via Predictable Discriminative Binary Codes”, ECCV 2012

Overview Diagram

Initial Labeled-Samples

Build Attribute Space

Project

Find Useful Attributes

Unlabeled Samples

Project

Choose Confident Examples To Add

Auxiliary data

• Exemplar: specific traits of a sample

Two Types of Attributes• Categorical: common traits of a category

Selected by Categorical Attributes

Initial Labeled Training Examples

Selected by Exemplar Attributes

Dotted

Animal-like shape

…

Exemplar Attributes

• Exemplar-SVM [1]– Captures example-specific

information• But requires lots of

negative samples to build

• Our approach– Difference on retrieval set

obtained by leave-one-out classifier and full-set classifier

[1] Tomasz Malisiewicz, Abhinav Gupta, Alexei A. Efros, “Ensemble of Exemplar-SVMs for Object Detection and Beyond”, ICCV 2011

Use allExcept each

Scor

e hi

gh

Given Training Samples

Modify the top-lambda-selector term in the formulation

Experimental Results• Dataset– A subset of ImageNet (ILSVRC 2010)– Target set: 11 categories (initial training #: 10/cat.,

testing: 500/cat.)• Both distinctive and fine-grained categories

– 6 vegetables (mashed potato, orange, lemon, green onion, acorn, coffee bean), 5 dog breeds (Golden Retriever, Yorkshire Terrier, Greyhound, Dalmatian, Miniature Poodle)

– Unlabeled set: randomly chosen samples from entire 1,000 categories in ILSVRC

– Auxiliary set: exclusive categories from target set

Download available soon in http://umiacs.umd.edu/~jhchoi/addingbyattr

Comparison with Other Approaches

• Red: using low-level features only

• Blue: our approach– Categorical attribute

only (Cat.)– Exemplar+categorical

attributes (E+C) Decrease!

Increase

Purity of Added Sample• Purity: # added samples of same category

total # added samples

Not very pure but still improves mAP!

Saturation of Performance

• Increase in mAP for method quickly saturates based on initial training set size.

• Most useful for small training sets.

Exemplar Attributes• Compare our exemplar attribute learning with

exemplar-SVM with large negative set

Summary• A new way of expanding a training set by attributes– A joint optimization formulation– Solve by block-coordinate descent

• Using both categorical and exemplar attributes to find examples

• Future work– Constraining expansion path

• Attribute may mislead as we added more samples due to low purity

Additional Slides

Varying number of added samples and accuracy