Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
FACE DETECTIONMikael Nilsson
2
FACE DETECTION
Outline
• Introduction
• Skin Color
• Appearance Based Learning
– Timeline, ~1994 ->
• Non Maximum Suppression
• Bicycle Counting Project
3
FACE DETECTION
Introduction
• Goal of Face Detection
– Given an arbitrary image, the goal of face detection is to determine whether or not there are any faces in the image and, if present, return the image location and extent of each face.
[Yang et al, Detecting Faces in Images: a Survey, Pami 2002]
4
FACE DETECTION
Introduction
• Goal of Face Detection version 2
– Given an image, tell whether there is any human face, if there is, where is it (or where they are).
5
FACE DETECTION
Introduction
• Goal of Face Detection version 3
– Identify and locate human faces in an image regardless of their • position
• scale
• in-plane rotation
• orientation
• pose (out-of-plane rotation)
• and illumination
6
FACE DETECTION
Introduction
• Why is it hard?– Pose Variation
– Beards, Glasses
– Facial Expression
– Occlusion
– Image Orientation
– Lighting, Noise
– Resolution
7
FACE DETECTION
Introduction
• Related problems– Facial Feature Extraction (like eyes, nose, mouth localization)
– Face Recognition (who is the person)
– Facial Expression Analysis (example: only “smile-photos”)
– Head Pose Estimation
– Gender/Age/Race Classification
– Face Tracking
8
FACE DETECTION
Introduction
• Two methods discussed
– Skin Color
– Appearance Based Learning
9
FACE DETECTION
Skin Color
• Skin Color
– General Idea• Find skin color pixels in image
• Use found pixels and do blob analysis
• Skin color methods has been proposed as preprocessing before other face detection methods (all ”orange” objects detected)
10
FACE DETECTION
Skin Color
• A skin pixel classifier using likelihood ratio approach
)|(
)|(skinrgbP
skinrgbP
rgb
P(rgb)
skin non-skin
)|( skinrgbP
)|( skinrgbP
11
FACE DETECTION
Skin Color
• Example of skin pixel classifier
– Model andusing Gaussian Mixture Model (GMM)
– Train two GMMs, one with skin pixels and another with nonskin pixels
– Find likelihood ratio using GMMs and set threshold
)|( skinrgbP )|( skinrgbP
12
FACE DETECTION
Demo
• Skin Color– Michael J. Jones and James M. Rehg, ”Statistical Color Models with Application to Skin Detection”, International Journal of Computer Vision, 1999, pp.274—280.
13
FACE DETECTION
Appearance Based Learning
• Apperance Based Learning
– General Idea• Collect a large set of resized (e.g. 19x19, 20x20, 22x22 or 32x32) face and nonface images and learn a classifier to discriminate them.
• Given a test image, detect faces by applying the classifier at each position and scale of the image. 22x22
nonface
face
14
FACE DETECTION
Appearance Based Learning
• Handling position and scale
– Search all positions and relevant scales
15
FACE DETECTION
Appearance Based Learning
• Timeline
– Kah Kay Sung and Tomaso Poggio, ”Example-based Learning for View-based Human Face Detection”, MIT A.I. Memo No. 1521, 1994, C.B.C.L. Paper No. 112
– H. Rowley, S. Baluja, and T. Kanade, “Neural network-based face detection”, CVPR, 1996, pp. 203–208.
– H. Schneiderman, T. Kanade. "Probabilistic Modeling of Local Appearance and Spatial Relationships for Object Recognition”, CVPR,
1998, pp. 45-51.
16
FACE DETECTION
Appearance Based Learning
• Timeline– D. Roth, M. Yang, and N. Ahuja, “A snow-based face detector”, in NIPS 12, 2000, pp. 855–861.
– P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features”, CVPR, 2001, vol. 1, pp. 511–518.
17
FACE DETECTION
Appearance Based Learning
• Sung & Poggio 1994– They collected 1067 face patterns and for each one of these training images they did:
• Resizing to 19x19 pixels
• Masking to avoid background pixels
• Lighting correction / Histogram Equalization
19x19
18
FACE DETECTION
Appearance Based Learning
• Sung & Poggio 1994– In addition, for each one of the 1067face patterns, new virtual samples were generated by rotation and mirror operations, totalizing 4150 face patterns.
19
FACE DETECTION
Appearance Based Learning
• Sung & Poggio 1994– Each 19x19 pattern can be seen as a point in a high dimensional space.
– The K-means algorithm is used to cluster face and non-face samples into 6 (six) clusters.
– Each cluster is described by a multi-dimensional Gaussian with a centroid and covariance matrix.
20
FACE DETECTION
Appearance Based Learning
• Sung & Poggio 1994
21
FACE DETECTION
Appearance Based Learning
• Sung & Poggio 1994– Compute the distance of a sample to all face and non-face clusters
– Final classifier use a Multi-Layer Perceptron (MLP) trained using backpropagation on the distance vector
22
FACE DETECTION
Appearance Based Learning
• Sung & Poggio 1994
– Bootstrapping• 1. Start with a small set of
nonface examples in the training set
• 2. Train a MLP classifier with the current training set
• 3. Run the learned face detector on a sequence of random images (not containing any faces).
• 4. Collect all the nonface patterns that the current system wrongly classifies as faces (i.e., false positives)
• 5. Add these nonface patterns to the training set
• 6. Got to Step 2 or stop if satisfied
Test Image
False Detections
23
FACE DETECTION
Appearance Based Learning
• Sung & Poggio 1994
– Result• Able to detect upright frontal faces in complex scenes.
• Computationally
expensive
24
FACE DETECTION
Appearance Based Learning
• Rowley et al. 1996– Similar to Sung and Poggio
– 20x20 instead of 19x19, no mask used
– Same techniques for bootstrap, preprocessing, etc.
– Neural Network applied directly into the image.
– Different heuristics
– Faster than Sung & Poggio (but still far from real-time)
25
FACE DETECTION
Appearance Based Learning
• Rowley et al. 1996
26
FACE DETECTION
Appearance Based Learning
• Rowley et al. 1997
– Rowley extended his initial work in 1997 by adressing rotations also
27
FACE DETECTION
Appearance Based Learning
• Rowley et al. 1997
– Router network, input and output
28
FACE DETECTION
Appearance Based Learning
• Rowley et al. 1997
29
FACE DETECTION
Appearance Based Learning
• Schneiderman et al. 1998– Used parts of image
– Three different classifiers are created: frontal face detector, left profile, and right profile. The right profile classifier is the same as the left profile, but take as input mirrored images.
30
FACE DETECTION
Appearance Based Learning
• Schneiderman et al. 1998
31
FACE DETECTION
Appearance Based Learning
• Schneiderman et al. 1998
– Left-, right profile and frontal face
– Also for cars
32
FACE DETECTION
Appearance Based Learning
• Yang et al.
– Created sparse boolean primitive input space (intensity x position) from 20x20 patches
– Also evaluated to extend input space with (position x intensity mean x intensity variance) with some improvements
33
FACE DETECTION
Appearance Based Learning
• Yang et al.
– Example sparse input, 2 bits per value and 5 values => (2^2)*5=20 boolean vector
34
FACE DETECTION
Appearance Based Learning
• Utilized Winnow update rule
• Winnow
– Promotion: if
– Demotion: if
• Perceptron
– Promotion: if
– Demotion: if
thresholdactivation
35
FACE DETECTION
Appearance Based Learning
• Sparse Input and Winnow update rule
– Sparse Network of Winnows (SNoW)
+
= SNoW
36
FACE DETECTION
Appearance Based Learning
• Main conclusion of…
– Sung & Poggio
– Rowley et al.
– Schneiderman et al.
– Yang et al.
• …is that they works quite well in detecting faces but are too slowfor real-time operation (at leastseveral seconds per image)
37
FACE DETECTION
Appearance Based Learning
• Viola and Jones 2001
– Breakthourgh regarding detection speed ~15 FPS on 384x288 size image
– Rectangle features (“Haar-like features”)
– Adaboost learning
– Classifier cascade
38
FACE DETECTION
Appearance Based Learning
• Rectangle features– Uses the integral image
– The integral image (ii) computes a value at each pixel (x,y) that is the sum of the pixel values above and to the left of (x,y) including the point.
(x,y)
39
FACE DETECTION
• Integral image
Appearance Based Learning
40
FACE DETECTION
Appearance Based Learning
• Integral image
ii(x, y-1)
s(x-1, y)
i(x, y)
41
FACE DETECTION
Appearance Based Learning
• Integral image
– Cumulative row sum: s(x, y) = s(x–1, y) + i(x, y)
– Integral image: ii(x, y) = ii(x, y−1) + s(x, y)
• Integral image in MATLAB – ii = cumsum(cumsum(double(i)), 2);
42
FACE DETECTION
Appearance Based Learning
• Let a,b,c,d be corner coordinates
• Then the sum of original image values within the rectangle can be computed as:
– D=ii(d)+ii(a)–ii(b)–ii(c)
• Only 3 additions (2-, 1+) and 4 table lookups are required to find sum over area for any size of rectangle!
43
FACE DETECTION
Appearance Based Learning
• Rectangle features
• Value=sum(pixels in white area)-sum(pixels in black area)
• sums done with integral image
44
FACE DETECTION
Appearance Based Learning
• Rectangle features
45
FACE DETECTION
Appearance Based Learning
• Which rectangle features to use
• Design of classifier
• Adaboost
Strong classifier
Weak classifier
WeightFeaturesvector
46
FACE DETECTION
• Given a nested set of classifier hypothesis classes
• Computational Risk Minimization
Appearance Based Learning
FACEIMAGESUB-WINDOW
Classifier 1
F
T
NON-FACE
Classifier 3T
F
NON-FACE
F
T
NON-FACE
Classifier 2T
F
NON-FACE
vs false neg determined by
% False Pos
% D
etec
tion
0 50
0
100
47
FACE DETECTION
Appearance Based Learning
• Classifier cascade
FACEIMAGESUB-WINDOW
Classifier 1
F
T
NON-FACE
Classifier 3T
F
NON-FACE
F
T
NON-FACE
Classifier 2T
F
NON-FACE
Classifier 1 Classifier 2 Classifier 3
48
FACE DETECTION
Non Maximum Supression
• Non Maximum Supression
– Note that there will typically be several detections on a face• This is true for all appearance based methods discussed
49
FACE DETECTION
Non Maximum Supression
• Merge overlapping detection boxes using non maximum supression
– Set a threshold on overlap ratio (e.g. 0.2)
– Use the rectangle with the highest (max) score from the classifier
50
FACE DETECTION
Non Maximum Supression
• Non Maximum Supression (NMS)
NMS
51
FACE DETECTION
Demo
• Face Detection– M. Nilsson, J. Nordberg and I. Claesson, ”Face Detection using Local SMQT Features and Split Up SNoW Classifier”, ICASSP, 2007, pp. II-589 -II-592.
52
FACE DETECTION
Bicycle Counting Project
• Bicycle Counting Project
– Note that other objects than faces can be used• For example Schneiderman tested on cars also
– In the Bicycle Counting project bicycles are considered• in low resolution to keep sensor cost low
53
FACE DETECTION
Bicycle Counting Project
• Design system that can classify bicycles…
54
FACE DETECTION
Bicycle Counting Project
• …in videos like this
55
FACE DETECTION
Bicycle Counting Project
• Start making positive samples
– Example of workflow
1
2
3
Zoom
Mark three points
Warp to patch
32x32
56
FACE DETECTION
Bicycle Counting Project
• Negatvie samples
– Start with some random images
– Utilize Bootstrapping (see Sung & Poggio) once classifier method is in place and build up database with more samples
57
FACE DETECTION
Bicycle Counting Project
• Design method to find bicycles
– Suggested method• Histogram of Oriented Gradients (HOG)
• Support Vector Machine (SVM)
– Aim:• Calculate bicycles going in both directions
58
FACE DETECTION
Bicycle Counting Project
• Example bicycle couting final result
Time <<< >>>
09:00 1 0
09:30 1 6
10:00 0 3
10:30 2 2
11:00 2 2
11:30 2 2
12:00 1 3
12:30 1 2
13:00 3 1
13:30 2 6
14:00 3 1
14:30 5 1
15:00 3 2
15:30 10 3
16:00 12 3