Style-Finder: Fine-­‐Grained  Clothing  Style  Detec5on  and  Retrieval  

Wei  Di*,  Catherine  Wah+,  Anurag  Bhardwaj*,    Robinson  Piramuthu*,    Neel  Sundaresan*  

6/30/13

* eBay Research Labs, Computer Vision Group +  Department of Computer Science and Engineering, UCSD

research labs

research labs

Outline

2

q  Motivation

q  Human annotation of attributes

q  Fine-grained classification

q  Search by attribute/style

q  Text generation

research labs

Motivation - Mobile

3

Vs

Retrieve Clothing with similar Style

Old Navy IVORY LONG PEA COAT WOOL NWT

Women Double-Breasted Long Slim Wool Trench Jacket Coat Outwear

New Navy Blue Double Breasted Long Peacoat Jacket Pea Coat

L.K. Bennett Ami Burgundy Coat

research labs

Coats & Jackets in various Shapes and Forms

Motivation - Style

4

Color Pattern Style

Aeropostale wool peacoat

research labs

Flow: Data Collection, Classification /retrieval /Annotation

Define style elements, Crowd Sourcing

Detected Style:

- Search: Visually similar rank - Annotation: Describe style

Style Modeling, Classification

Local + Global

+

SVM-based style element

detector

5

AMT

…

research labs

Ambiguity (class level)

Challenges in Data

High variability • Shape • How it’s worn/ photographed

Label noise/incomplete

Less detailed descriptions or incorrect labeling

Is this a (a) Puffer jacket (b) Vest (c) Both?

6

research labs

Data collection: Coats/Jackets Dataset

q  Harvested images from 18+ online clothing retailers §  Used retailer category labels and/or search terms

q  Data §  Well-aligned, clean images §  2k images, 12 sub-category:

blazer, cape, jean, military, motorcycle, parka, peacoat, poncho, puffer, raincoat, trench, vest.

7

research labs

Round V-shape

Simple Collar

Turtle V-Shirt

Folded Collar

Notched Shawl Peak Round Shirt

Human Annotation of Attributes

Has fur Denim Leather Shiny Wool

Material

Zip Button Open Has belt

Fastener

8

research labs

Human Annotation of Attributes

9

Attribute type Attributes Material Fur, Denim, Leather/is leather-like, Shiny, Wool/is woolen

or felt-like Fastener Zip, Button, Open, Has belt Fastener style Symmetrical (single-breasted), Asymmetrical (single-breasted),

Asymmetrical (double-breasted) Length Short, Medium, Long Cut Fitted, Loose Pocket Chest pocket, Side pocket Collar V-neck collar, Round collar, Turtle neck, V-neck shirt collar,

Round shirt collar, Notched collar, Shawl collar, Peak collar

-  5 annotations per image -  Majority agreement on the presence or absence of the

attribute -  27 binary attributes

research labs

Statistics of Crowd Sourcing

10

Average Attributes per Cloth Image

Co-occurrence of Style Attributes

research labs

Fine Grained Classification : Kernel Mapping q  Visual features

-  SIFT, HOG, GIST, etc -  Exclude color features

q  Kernelized bag-of-words features

–  Kernel trick deal with linear non-separable data –  Homogeneous Kernel Mapping –  Low dimensional kernel approximations

11

Linear vs non-linear kernels 4

Linear SVM

✔ fast✘ restrictive

Non-linear SVM

✘ much slower✔powerful

>=< xwx ,)(F

∑=

=N

iiiKF

1),()( xxx α

xwxxxxx Tl

kki

Tl

kiif +=+=+= ∑∑

==

βαβαβ11

,)(

∑=

=N

iiiKF

1),()( xxx α

( ))(ˆ),(ˆ),( iiK xxxx ψψ≈

Approximated feature map

)(ˆ,)( xwx ψ=F )(ˆ1

xw ψα∑=

=N

ii

xxxxxxkʹ′+

ʹ′=ʹ′2),(

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=Φ

)log6.0sin(6.0)log6.0cos(6.0

8.0)(

xxxx

Homogeneous kernel map Closed form, simple, small

X2 Kernel Excellent for bag-of-words…

Binary classifier

research labs

Fine Grained Classification : Classifier

Offline: learn binary SVM for each style type

Has style-X Not Has

Style-X

q  Visual features (SIFT, HOG, etc)

q  Kernelized bag-of-words features

q  Binary classifier for each attribute

- Explicit kernel features

-  Max-margin Linear SVM

-  Fast

12

research labs

Style Recognition/ Retrieval

Input image

At test time:

Has/Is Style-X?

YES

Ranking by attribute similarity

User ask questions: coats with collars like this

q  Visual features (SIFT, HOG, etc) q  Kernelized bag-of-words features

–  Homogeneous Kernel Mapping -  Binary classifier -  Linear SVM

q  Retrieval & Annotation -  Recognition -  Rank by visual similarity

13

research labs

Result Evaluation q  Evaluated by

–  Classification accuracy –  Precision-recall

14

research labs

Is-shiny

Wool/ wool-like

Query Retrieval Example

Has-fur

15

research labs

Has-button

Has-zip

Has-belt

Query Retrieval Example

16

research labs

Automatic Style Element Attribute Annotation

Image Automatic Generated Descriptions

(a) Shiny, short

(b) Has-fur, leather/is leather-like

(c) Open, loose, round collar

(d) Button, short, chest pocket, v-neck shirt collar

(a) (b) (c) (d)

17

research labs

Automatic Style Element Attribute Annotation

Image Automatic Generated Descriptions

(a) Button, has-belt, asymmetrical and double-breasted, long, slim, v-neck shirt collar, round shirt collar

(b) Wool/is woolen or felt-like, zip, loose, round collar

(c) Has-fur, v-neck-collar

(d) Symmetrical and single-breasted, shawl collar

(a) (b) (c) (d)

18

research labs

Future Work

q  Joint learning of Attribute (ex. collar) + Category (ex: coats)

q  Multi-attribute query retrieval

q  Automatic style element localization

q  Personalization based on style

19

research labs 20

Thanks!