Unsupervised Feature Learning by Deep Sparse Coding

3/1/20 1

Yunlong He, Georgia TechKoray Kavukcuoglu, DeepMind TechnologiesYun Wang, Princeton UniversityArthur Szlam, The City College of New YorkYanjun Qi, University of Virginia

Two lines of Previous Research• Bag of Visual words

• Deep learning

• Motivation: Can we combine the power of two methods?

3/1/20 2

Image Dense CodePyramid Pooling ClassifierSparse Code

Input Image Features

Encoder

DecoderFeatures

Encoder

Decoder

Class label

Encoder

Decoder…

Repeat[Hinton & Salakhutdinov 06, 09]

Layer 1 Layer n [LeCun et al. 98]

[Lazebnik et al. 06][Yang et al. 09]

SIFT Descriptors

Background: Bag of Visual words

3/1/20 3

Image

Dense Code

Pyramid Pooling

Classifier

Sparse Code

Background: Bag of Visual words

3/1/20 4

Image

SIFT Descriptors

Image

Dense Code

Background: Bag of Visual words

3/1/20 5

Example:900 Patches

Image

Dense Code

Background: Bag of Visual words

3/1/20 6

SIFT filter

SIFT Descriptorsfor every patch

Image

Dense Code

Background: Bag of Visual words

3/1/20 7

Dense Code

Sparse Code

Background: Bag of Visual words

3/1/20 8

Pyramid Pooling

Sparse Code

Level 1 Level 0

21 pooling regions

Concatenate Sparse Codes of 21 pooling regions

Background: Bag of Visual Words

Relatively shallow

3/1/20 9

Image

Dense Code

Pyramid Pooling

Classifier

Sparse CodeCan we learn multiple layers of sparse representations?

Deep Sparse Coding

Relatively shallow

3/1/20 10

Image

Dense Code

Pyramid Pooling

Classifier

Sparse Code

Image

Dense Code

Pyramid PoolingSparse Code

Dense Code

Sparse Code

Dense Code

Sparse Code

Pyramid Pooling

Pyramid Pooling

Classifier

Unsupervised Deep Architecture!

Deep Sparse Coding

Relatively shallow

3/1/20 11

Image

Dense Code

Pyramid Pooling

Classifier

Sparse Code

Image

Dense Code

Pyramid PoolingSparse Code

Dense Code

Sparse Code

Dense Code

Sparse Code

Pyramid Pooling

Pyramid Pooling

Classifier

Unsupervised Deep Architecture!

Deep Sparse Coding

Relatively shallow

3/1/20 12

Image

Dense Code

Pyramid Pooling

Classifier

Sparse Code

Image

Dense Code

Pyramid PoolingSparse Code

Dense Code

Sparse Code

Dense Code

Sparse Code

Pyramid Pooling

Pyramid Pooling

Classifier

Unsupervised Deep Architecture!

Dimension expansion

Idea:• Separation (for classification)• Representation (bag of visual words)

Method:• Sparse coding

3/1/20 13

Dense Code

Sparse Code

Dimension reduction

Idea:• Invariance• Compositionality

Method:• Locally spatial pooling + Low dimensional embedding• Both unsupervised

3/1/20 14

Sparse Code

Dense Code

Locally spatial pooling

3/1/20 15

Covers larger area

Low dimensional embedding

3/1/20 16

embedding

Unstable Stable

Low dimensional embedding

3/1/20 17

embedding

Unstable Stable

Idea: embedding with the help of spatial information

Low dimensional embedding

3/1/20 18

embedding

Dimensionality Reduction by Learning an Invariant Mapping (DR. LIM)

Low dimensional embedding

3/1/20 19

embedding

Dimensionality Reduction by Learning an Invariant Mapping (DR. LIM)

Deep Sparse Coding

Relatively shallow

3/1/20 20

Image

Dense Code

Pyramid Pooling

Classifier

Sparse Code

Image

Dense Code

Pyramid PoolingSparse Code

Dense Code

Sparse Code

Dense Code

Sparse Code

Pyramid Pooling

Pyramid Pooling

Classifier

Unsupervised Deep Architecture!

3/1/20 21

Conclusion

§Combine sparse coding with deep learning§Combine dimension expansion and dimension reduction§A connecting function is learned by embedding method§ Combining multiple layers of sparse code achieves state-of-the-art performance on image classification tasks