Face Detection Face Alignment Face Parsingpages.cs.wisc.edu/~dyer/cs534-spring11/slides/... · Face detection, face alignment, and face image parsing Brandon M. Smith Guest Lecturer,

12/6/2013

1

Face detection, face alignment, and face image parsing

Brandon M. Smith

Guest Lecturer, CS 534

Monday, October 21, 2013

12/6/2013 1 CS 534: Computation Photography

Lecture overview

• Brief introduction to

local features

• Face detection

• Face alignment and

landmark localization

• Face image parsing

12/6/2013 CS 534: Computation Photography 2

http://www.ima.umn.edu/2008-2009/MM8.5-14.09/activities/Wohlberg-Brendt/sift.png

http://www.noio.nl/assets/2011-03-01-stitching-smiles/violajones.png

http://www.mathworks.com/matlabcentral/fx_files/32704/4/icaam.jpg

http://homes.cs.washington.edu/~neeraj/projects/face-parts/images/teaser.png

Local features: broad goal

• What are local features trying to capture?

The local appearance in a region of the image


David G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, 60, 2 (2004)

Local features: motivation What are their uses?

o Matching


http://docs.opencv.org/_images/Featur_FlannMatcher_Result.jpg

12/6/2013

2


o Matching

o Image indexing and retrieval


Shen et al., CVPR 2012


o Matching


o Aligning images, e.g., for panorama stitching


http://www.leet.it/home/lale/files/Garda-pano.jpg


o Matching



o Video stabilization



o Matching




o 3D reconstruction


http://www.nsf.gov/news/special_reports/science_nation/images/virtualrealitymaps/duomopisa500.jpg

http://youtu.be/3TlCGh5Pc90?t=1m57s

12/6/2013

3


o Matching




o 3D reconstruction

o Object recognition, including face recognition


http://doi.ieeecomputersociety.org/cms/Computer.org/dl/trans/tp/2007/11/figures/i192714.gif

Local features: types

Types of features and feature descriptors o Image intensity or gradient patches

o Shift Invariance Feature Transform (SIFT) – very popular!

o DAISY

o SURF

o Many more…


Face detection: goal

Automatically detect the presence and

location of faces in images.


Shen et al., Detecting and Aligning Faces by Image Retrieval, CVPR 2013

Face detection: motivation

• Automatic camera focus


http://cdn.conversations.nokia.com.s3.amazonaws.com/wp-content/uploads/2013/09/Nokia-Pro-Camera-auto-focus_half-press.jpg http://cdn.conversations.nokia.com.s3.amazonaws.com/wp-content/uploads/2013/09/Nokia-Pro-Camera-auto-focus_half-press.jpg

http://cdn.conversations.nokia.com.s3.amazonaws.com/wp-content/uploads/2013/09/Nokia-Pro-Camera-auto-focus_half-press.jpg




























12/6/2013

4



• Easier photo tagging


http://sphotos-d.ak.fbcdn.net/hphotos-ak-ash3/163475_10150118904661729_7246884_n.jpg



• Easier photo tagging

• First step in any face recognition algorithm


http://images.fastcompany.com/upload/camo1.jpg

Face detection: challenges

• Large face shape and appearance variation

• Scale and rotation (yaw, roll, pitch) variation

• Background clutter

• Occlusions

• Image noise

• Efficiency

• False positives


Face detection: Viola-Jones*

• Paul Viola and Michael Jones, Robust Real-time

Face Detection, International Journal of Computer

Vision (IJCV), 2004.

o Feature type?

o Which features are important?

o Decide: face or not a face


* Next few slides are based on a presentation by Kostantina Pall & Alfredo Kalaitzis, available at http://www1.cs.columbia.edu/~belhumeur/courses/biometrics/2010/violajones.ppt











12/6/2013

5

Face detection: Viola-Jones

Feature type?

• Useful domain knowledge: o The eye region is darker than the forehead or the

upper cheeks

o The nose bridge region is brighter than the eyes

o The mouth is darker than the chin

• Encoding o Location and size: eyes, nose bridge, mouth, etc.

o Value: darker vs. brighter



Feature type?

• Rectangle features o Value = ∑(pixels in black)

- ∑(pixels in white)

o Three types: 2,3,4 rectangles

o Very fast: integral image




* From http://www.cs.ubc.ca/~lowe/425/slides/13-ViolaJones.pdf

Which features are important? • Tens of thousands of features to choose from

• AdaBoost (Singer and Schapire, 1997)

o Given a set of weak classifiers: ℎ𝑡 𝑥 ∈ {−1,1}

o Iteratively combine classifiers to form a strong

classifier:

𝐻 𝑥 = 1 𝑖𝑓 𝛼𝑡ℎ𝑡(𝑥)

𝑡> 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑

0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒


Final decision: face or not a face

• Cascade of classifiers 1. Two-feature classifier: >99% recall, >60% precision

2. Five-feature classifier

3. 10-feature classifier

…

10. 200-feature classifier


12/6/2013

6



http://vimeo.com/12774628#

Face detection: recent approaches Xiangxin Zhu and Deva Ramanan, Face Detection,

Pose Estimation, and Landmark Localization in the

Wild, CVPR 2012.


Face detection: recent approaches

Shen et al., Detecting and Aligning Faces by Image

Retrieval, CVPR 2013.


Face detection: recent approaches

Shen et al., Detecting and Aligning Faces by Image

Retrieval, CVPR 2013.


http://vimeo.com/12774628

12/6/2013

7

Face alignment and landmark

localization: goal

Goal of face alignment: automatically align a face

(usually non-rigidly) to a canonical reference

Goal of face landmark localization: automatically

locate face landmarks of interests


http://www.mathworks.com/matlabcentral/fx_files/32704/4/icaam.jpg

http://homes.cs.washington.edu/~neeraj/projects/face-parts/images/teaser.png


localization: motivation

• Preprocess for:

o Face recognition

o Portrait editing wizards

o Face image retrieval

o …

• Face tracking

• Expression recognition

• Facial pose recognition


http://static3.businessinsider.com/image/52127e2169bedd4d60000012-752-564/realeyes-facial-recognition.png

http://mission0ps.com/wp-content/uploads/2013/04/10-special-effects.jpg


localization: challenges

• Pose

• Expression

• Identity variation

• Occlusions

• Image noise



localization: approaches

Parametric appearance models o Cootes, Edwards, and Taylor, Active Appearance Models, ECCV 1998





















12/6/2013

8



Parametric appearance models o Cootes, Edwards, and Taylor, Active Appearance Models, ECCV 1998




Part-based deformable models o Saragih et al., Face Alignment through Subspace Constrained Mean-

Shifts, ICCV 2009




Part-based deformable models o Saragih et al., Face Alignment through Subspace Constrained Mean-

Shifts, ICCV 2009




Supervised descent o Xiong and De la Torre, Supervised Descent Method and its Applications to

Face Alignment, CVPR 2013


12/6/2013

9



Exemplar-based/non-parametric methods o Shen et al., Detecting and Aligning Faces by Image Retrieval, CVPR 2013.


Face image parsing

Smith, Zhang, Brandt, Lin, and Yang, Exemplar-Based

Face Parsing, CVPR 2013.


Face image parsing: goal

Given an input face image, automatically segment

the face into its constituent parts.


Face image parsing: motivation

• Like face alignment, can be used as a preprocess

for face recognition, automated portrait editing,

etc.

• Encodes ambiguity

• Generalizes to hair, teeth, ears etc. across datasets


12/6/2013

10

Face image parsing: our approach


2K exemplar images 11 landmarks

~150 SIFT features

Exemplar labels

Database




~150 SIFT features

Exemplar labels

Database Step 0: Rough alignment & Top exemplar selection

100 top exemplars

Input




~150 SIFT features

Exemplar labels


Step 1: Nonrigid alignment

Input




~150 SIFT features

Exemplar labels



Input

Step 2: Exemplar label aggregation

*

+ *

+ …

=

12/6/2013

11




~150 SIFT features

Exemplar labels



Input

Step 2: Exemplar label aggregation

*

+ *

+

= Step 3: Pixel-wise label selection

w1 * + w2 *

+ w9 *

Label 1 Label 2

… Label 9

Output

∝

Face image parsing: quantitative

results


Face image parsing: qualitative

results


+ Input Soft segments Hard segments Ground truth

Face image parsing: qualitative

results


+ Input Soft segments Hard segments Ground truth

Documents

Face Detection Face Alignment Face Parsingpages.cs.wisc.edu/~dyer/cs534-spring11/slides/... · Face detection, face alignment, and face image parsing Brandon M. Smith Guest Lecturer,