CENTER FOR

MACHINE PERCEPTION

CZECH TECHNICAL

UNIVERSITY IN PRAGUE

RESEARCHREPORT

ISSN

1213-2365

Detector of landmark points onhuman face

Michal Uricar

uricamic@fel.cvut.cz

CTU–CMP–0000–00

September 30, 2010

Available athttp:∖∖cmp.felk.cvut.cz∖∼uricamic∖report.pdf

Supervisor: Ing. Vojtech Franc, Ph.D.

Acknowledge grants here. Use centering if the text is too short.

Research Reports of CMP, Czech Technical University in Prague, No. 0, 0000

Published by

Center for Machine Perception, Department of CyberneticsFaculty of Electrical Engineering, Czech Technical University

Technicka 2, 166 27 Prague 6, Czech Republicfax +420 2 2435 7385, phone +420 2 2435 7637, www: http://cmp.felk.cvut.cz

Detector of landmark points on human face

Michal Uricar

September 30, 2010

Abstract

The aim of thesis is implementation and experimental testing ofdetector of significant points on human face. Student will aim on twoapproaches. The first approach suppose detector built on existing im-plementation of method ”Active Appearance Models (AAM)”. Thesecond suppose solving of detection task by structural classifier learnedon examples. Thesis topic corresponds with project HUMAVIPS (Hu-manoids with auditory and visual abilities in populated spaces).

1 Active Appearance Models

Main task of the first experiment was modification of existing implementation[Kro10] of Active Appearance Models (AAM) [CET01]. Original MATLABimplementation was written for detection of human hands, so we had tomodify it to better fit our needs.

Whole experiment can be executed with MATLAB script experiment01.m.

1.1 Data description

For this method we used different databases for building AAM search model(training phase) and for testing it.

1.1.1 Training data set

For building AAM Search model we used existing IMM Face database, whichconsist of 240 annotated images (but only 40 unique faces) 640 × 480px.Unfortunately this database does not include ethnicity variations and thereis no face wearing glasses. Format of IMM Face database [NLSS04] wasmodified as required the MATLAB AAM implementation.

1

1.1.2 Test data set

The test was performed on our annotated LFW database. LFW databaseconsist of 13233 images 250× 250 px of various ethnicities, some faces wearglasses. Annotation is formed by XML file with positions (x, y) of 4 landmarkpoints— left eye, right eye, nose and mouth.

We have used face detector on this database and excluded images thatdoes not correspond with annotation (either there was more detected facesthan annotations or vice-versa). For simplicity we have also excluded imageswith more than one annotated face. As a result we have acquired the newdatabase with 12196 images that could be used for testing.

1.2 Results

All measurements were done in normalized frame (40× 40 px). Normalizedframe was constructed from bounding box provided by face detector andextended by 20% in each dimension. As initial point for AAM search modelwe chose centroid of bounding box.

Error is measured as distance between original annotated points and fromthe same points obtained from AAM search (i.e. centroids of contours foreyes, mouth and nose).

Table 1 depicts mean error, i.e. average of mean Euclidean distance fromall 4 landmark points and mean max error, i.e. average of maximum ofEuclidean distance from landmark points. On table 2 we can see the same,but for each landmark point apart.

Mean error Max error2.019 3.187

Table 1: Mean and max error. Values are in pixels. Base window 40× 40 px

Left eye Right eye Mouth NoseMean error 1.923 1.796 1.850 2.508Max error 16.819 12.905 16.444 19.564

Table 2: Mean and max errors on each landmark point. Values are in pixels.Base window 40× 40 px

1.3 Graphs

In this section we can find test measurements in graphs.

2

Figure 1: Left: Histogram of mean error. Right: Cumulative histogram ofmean error.

Figure 2: Left: Histogram of max error. Right: Cumulative histogram ofmax error.

Figure 3: Left: Mean error on each landmark point. Right: Max error oneach landmark point.

3

Figure 4: Left: Histogram of mean error on left eye. Right: Cumulativehistogram of mean error on left eye.

Figure 5: Left: Histogram of mean error on right eye. Right: Cumulativehistogram of mean error on right eye.

Figure 6: Left: Histogram of mean error on mouth. Right: Cumulativehistogram of mean error on mouth.

4

Figure 7: Left: Histogram of mean error on nose. Right: Cumulative his-togram of mean error on nose.

1.4 Image output on test data set

In this section we provide some image output of our AAM based detector.Yellow rectangle is bounding box obtained from face detector. Blue linesare contours from AAM model. Red points are ground truth positions ofannotated landmarks. Green points are the same points computed fromAAM as centroids of relevant contours.

Figure 8: Image output of aligned AAM model on test data set. Each imagemean error is from range 0.5–1.5 px. Base window 40× 40 px

5

Figure 9: Image output of aligned AAM model on test data set. Each imagemean error is from range 0.5–1.5 px. Base window 40× 40 px

Figure 10: Image output of aligned AAM model on test data set. Each imagemean error is from range 0.5–1.5 px. Base window 40× 40 px

2 Structural classifier

Second experiment was creation of new method for detection of significantpoints on human face based on structural classifier.

Whole experiment can be executed with MATLAB script experiment02.m.

2.1 Data description

For this method was used LFW face database. First we pruned database sim-ilarly as in previous AAM experiment (elimination of images with undetectedfaces or images with more than one annotation).

6

Then we specified some parameters for classifier— dimensions of basewindow, components (i. e. bounding boxes of landmark points) and basewindow margin (bw margin). These parameters were carefully set up to covermost of images in pruned database. Main request on parameters was thatall components fits in normalized image frame. Normalized image frame isoriginal image cropped to bounding box extended by bw margin and rescaledto dimension of base window.

Final setup of paramaters summarizes table 3. 97.5% of images frompruned database passed this configuration. That is 11929 images that formour new database.

base window (bw) 40× 40 pxcomponent - left eye bw(1)/3× bw(2)/3 pxcomponent - right eye bw(1)/3× bw(2)/3 pxcomponent - mouth bw(1)/2× bw(2)/3 pxcomponent - nose bw(1)/3× bw(2)/3 pxbw margin 20%× 20%

Table 3: Configuration of parameters

This database was randomly split into three parts— Training set (TRN),Validation set (VAL) and Test set (TST). Partition of database was done infollowing manner— 60% TRN, 20% VAL and 20% TST. Precise number ofimages in each set summarize table 4.

TRN 7157VAL 2386TST 2386

Table 4: Partition of database into TRN, VAL and TST set.

2.2 Results

For teaching classifier we used Bundle Method for regularized Risk Mini-mization (BMRM). This method was called for several values of parameter� ∈ {10−3, 10−2, 10−1, 1, 10}. Optimal parameter � was set to minimize vali-dation error as summarize table 5.

� 10−3 10−2 10−1 1 10RVAL 1.3375 1.2319 1.3261 1.7530 2.3875

Table 5: Validation error for all �-values. Optimum is at � = 10−2

7

For � = 10−2 mean error on test set was RTST = 1.2216 px, base window40× 40 px.

Table 6 summarize mean error, i. e. average of mean Euclidean distancefor each landmark point and mean max error, i. e. average of maximum ofEuclidean distance from each landmark point.

Left eye Right eye Mouth NoseMean error 1.123 0.973 1.327 1.463Max error 12.530 11.662 15.000 13.000

Table 6: Mean and max errors on each landmark point. Values are in pixels.Base window 40× 40 px

2.3 Graphs

In this section we can find test measurements in graphs.

Figure 11: Left: Histogram of mean error. Right: Cumulative histogram ofmean error.

8

Figure 12: Left: Histogram of max error. Right: Cumulative histogram ofmax error.

Figure 13: Left: Mean error on each landmark point. Right: Max error oneach landmark point.

Figure 14: Left: Histogram of mean error on left eye. Right: Cumulativehistogram of mean error on left eye.

9

Figure 15: Left: Histogram of mean error on right eye. Right: Cumulativehistogram of mean error on right eye.

Figure 16: Left: Histogram of mean error on mouth. Right: Cumulativehistogram of mean error on mouth.

Figure 17: Left: Histogram of mean error on nose. Right: Cumulative his-togram of mean error on nose.

2.4 Image output on test data set

10

Figure 18: Image output on TST data set. Green points— ground truthannotation. Red points— detected points.

11

References

[CET01] T. F. Cootes, G. J. Edwards, and C. J. Taylor. Active appearancemodels. IEEE Trans. Pattern Analysis and Machine Intelligence,23(6):681–685, June 2001.

[Kro10] Dirk-Jan Kroon. Active shape model (asm) and active appearancemodel (aam). MATLAB implementation, February 2010.

[NLSS04] M. M. Nordstrøm, M. Larsen, J. Sierakowski, and M. B. Stegmann.The IMM face database - an annotated dataset of 240 face images,may 2004.

12