IRCDL'07 - Padova, January 2007

3D Face Recognition using iso-Geodesic Surfaces

Stefano Berretti, Alberto Del Bimbo, Pietro Pala

Dipartimento di Sistemi e InformaticaUniversity of Firenze,

Firenze, ITALY

{berretti,delbimbo,pala}@dsi.unifi.it

Motivation

Detection and identification of human faces have been largely addressed mainly focussing on 2D still images and videos.

Recently, the increasing availability of three-dimensional (3D) data, has renewed the interest in 3D face models to improve the effectiveness of face recognition systems. Solutions based on 3D face models are expected to feature less

sensitivity to lighting conditions and pose variations.

3D face models are acquired through laser scanners, structured ligth scanners or multi camera systems, in the form of range images, cloud of points or polygonal meshes. Usually texture data is not available.

3D face recognition approaches

2D to 2D (using deformations of a 3D model): The input is 2D image; the database contains 2D images. Recognition is performed by

comparing the deformations of a 3D face model over the database images and the query image (using morphing) [Blanz, TPAMI03].

3D to 2D: 3D to 2D transformation. Techniques for 2D image matching (e.g., PCA, Gabor filters,

wavelets, etc.) are used to perform recognition in 2D [Pan, CVPR’05], [Wang, CVPR06]. Model based solutions. A 3D annotated face model is deformed to fit a 3D query model. The

deformed model is projected onto a 2D image. Recognition is performed by comparing 2D projected images [Jain, CVPR06], [Kakadiaris, BMVC06].

2D+3D: 3D face queries are matched with 3D face models; 2D face images are matched with 2D

face image models, separately. Results of the two matchings are combined to achieve better recognition accuracy [Beumier, PRL01], [Wang, IVC05].

3D to 3D (point clouds): 3D face rigid surfaces acquired as point clouds are compared using the Iterative Closest

Point (ICP) algorithm [Mian, BMVC05], [Chang, TPAMI06]. ......

Current research challenges: Recognition in the presence of expression variations. Matching efficiency.

Proposed solution

We propose a 3D to 3D face recognition approach in which the structural information of a 3D face model, given as a mesh, is directly captured in the 3D space by modeling the basic 3D shape and the relative arrangement of iso-geodesic stripes identified on the model surface.

Three steps: Extraction of iso-geodesic stripes according to the geodesic

distances from fiducial points. Modeling of the shape and 3D spatial relationships between stripes

(surfaces in the 3D space). Construction and matching of a graph based face model.

Geodesic distance from fiducial points

Iso-geodesic stripes are defined with reference to a function f defined on the face surface. Value of f at a generic vertex v is defined as

the geodesic distance (v,u) between v and a fiducial vertex u (the nose tip).

Due to the polygonal approximation of the model surface, the geodesic distance is computed as the shortest path on the edges of the mesh by using the Dijkstra’s algorithm. For regular meshes, this provides a fair approximation of

the real geodesic distance.

Normalized values of the distance are obtained dividing the geodesic distance by the Euclidean eye-to-nose distance. This guarantees invariance w.r.t. scaling and pose of the face. Since the eye-to-nose distance is invariant to face expressions, this

normalization does not bias values of under expression changes.

Euclidean distance

Geodesic distance

v

u

The nose tip and the eyes cavities are identified using mean and Gaussian curvatures according to state of the art solutions*. Models are aligned with the nose tip pointing along the z axis.

Distances from the nose tip are quantized into n intervals c1,..., cn. Accordingly, n level set stripes are identified on the model surface, the i-th stripe corresponding to the set of surface points on which the value of the distance falls in the limits of interval ci.

Stripes are surfaces in the 3D space. Information captured by the stripes is

modeled by their 3D mutual spatial relationships using the weighted walkthroughs approach.

Stripes extraction

(*) K.I. Chang, K.W. Bowyer, P.J. Flynn. “Multiple Nose Region Matching for 3D Face Recognition Under Varying Facial Expression,” IEEE Trans. on Pattern Analysis and Machine Intelligence, October 2006.

In a 3D reference system, with coordinate axes X, Y, Z, the projections of two points a = < xa,ya,za > and b = < xb,yb,zb > on each axis, can take three different orders: before, coincident, after. The combination of the three projections

results in 27 different 3D displacements which can be encoded by a triple of indexes <i,j,k>:

In general, points of 3D extended sets A, B can be connected through multiple different displacements. The triple <i,j,k>, is a walkthrough from A to B if it

encodes the displacement between at least one pair of points aA, bB.

Modeling 3D spatial relationships

ab

ab

ab

xx

xx

xx

i

1

0

1

ab

ab

ab

zz

zz

zz

k

1

0

1

ab

ab

ab

yy

yy

yy

j

1

0

1

x

a

b

z

y

za xa

ya

zb

xb

yb i=+1

k=+1

j=-1

z

y

j

A

B<1,0,0>

<1,-1,1> xk

i

3D weighted walkthroughs (WW)

Relevance of each walkthrough <i,j,k> is measured by a weight wi,j,k(A,B) evaluating the number of pairs of points aA, bB, whose displacement is captured by the direction <i,j,k>. Formally, weights wi,j,k(A,B) are evaluated as integral measures

over the six-dimensional set of point pairs in A and B:

Kijk acts as dimensional normalization factor; C±1(.) are the characteristic functions of the positive and negative real semi-axis (0,+oo) and (-oo,0), respectively. C0(.)= δ(.) denotes the Dirac's function and acts as the characteristic function of the singleton set {0}.

Weights between A and B are organized in a 3x3x3 matrix w(A,B), of indexes i, j, k.

bababaabk

A B

abjabiijk

ijk dzdzdydydxdxzzCyyCxxCK

BAw )()()(1

),(

W011W-111 W111

W001W-101 W101

W0-11W-1-11 W1-11

+1

j 0

-1-1 0 +1 i

W010W-110 W110

W100

W1-10

W01-1W-11-1 W11-1

W10-1

W1-1-1

+1

0 k

-1

w(A,B)

Compositional computation

Integral properties ensure that the 27 weights are reflexive (wi,j,k(A,B) = w-i,-j,-k(B,A)), and invariant with respect to shifting and scaling. In addition, they are compositional:

According to this, any complex entity is decomposed in a set of 3D rectangular voxels of a uniform partitioning of the space. The integral computation is cast to the linear

combination of a set of sub-integrals computed in closed form on rectangular domains arranged in a fixed number of possible different spatial positions.

),(),(

),(),(

),(

),(),( 2

21

21

21

121 BAw

BBAK

BAKBAw

BBAK

BAKBBAw ijk

ijk

ijkijk

ijk

ijkijk

0 0 0

wk=1(an,bm) = 0 0 0

¼ ½ ¼

0 0 0

wk=0(an,bm) = 0 0 0

½ 1 ½

0 0 0

wk=-1(an,bm) = 0 0 0

¼ ½ ¼z

y

an

x

bm

nnaA

10

1

mmbB

4

1

an

bm

WW between surfaces (stripes) in 3D

Computation of 3D spatial relationships between surfaces directly descends from the general case. Volumetric integrals become surface integrals extended to the

points of two surfaces.

Numerical computation of the surface integrals is avoided by exploiting the property of compositionality, and regarding surface entites as volumetric entities with infinitesimal thickness. Linear combination of integrals computed in

closed form between elementary rectangular voxels of a uniform partitioning of the space.

n mmnijkmnijk

ijkm

mn

nijk bawbaK

BAKbaw ),(),(

),(

1,

Given the w(A,B) matrix, measuring the relationship between entities A and B, three directional weights, taking values in [0,1], are computed on the eight corner weights:

These account for the degree by which B is on the right ( ), up ( ) and in front of A ( ).

Similarly, three weights account for the alignment along each of the three reference directions of the space:

Finally, three weights account for the joint alignment along two directions of the space:

Directional and alignment weights

1,1,11,1,11,1,11,1,1 wwwwwV1,1,11,1,11,1,11,1,1 wwwwwH

1,1,11,1,11,1,11,1,1 wwwwwD

w(A,B)

3x3x3 matrix

w-111 w111

w-1-11 w1-11

w-111 w111

w-11-1 w11-1

w111

w1-11

w11-1

w1-1-1

1,1,01,1,01,1,01,1,00 wwwwwH1,0,11,0,11,0,11,0,10 wwwwwV0,1,10,1,10,1,10,1,10 wwwwwD

1,0,01,0,00 wwwHV0,0,10,0,10 wwwVD0,1,00,1,00 wwwHD

Distance between WW

Dis-similarity in the arrangement of pairs of entities <A,B> and <A’,B’> captured by matrices w(A,B)=w and w(A’,B’)=w’, is evaluated by a distance D(w,w') which combines the differences between homologous coefficients in the space of 27-tuples of WW. Using the directional and alignment weights, this is expressed as:

where are non negative numbers with sum equal to 1.

000000000

000000000

'''

'''

''')',(

VDVDVDHDHDHDHVHVHV

DDDVVVHHH

DDDVVVHHH

wwwwww

wwwwww

wwwwwwwwD

Graph representation

WW are computed for every pair of stripes, including the pair composed by a stripe and itself. A face with N stripes is represented by N*(N+1)/2 relationship

matrixes. This model is cast to a graph representation by regarding face stripes as graph nodes and their mutual spatial relationships as graph edges:

assigns to a node the WW computed between the node and itself; associates to every edge the WW computed between the two nodes the edge connects to.

edgesofsetNNE functionlabelingnodesLN N ,:

nodesofsetN ,,,ENG

functionlabelingedgesLN E ,:

Efficient matching

Due to the normalization of geodesic distances, homologous stripes in different models capture the same distance interval. Face matching is reduced to the comparison between homologous

iso-geodesic stripes and their relationships: An interpretation Γ associates homologous nodes in the two representations, that is Γ(pk) = rk, k=1,...., N, being pk and rk nodes of the template and reference face graphs:

M

k

k

hhkhk

M

kkkkk

rrwttwD

rrwttwDRT

1

1

1

1

)),(),,(()1(

)),(),,((),(

t1

t2

t3

t4

t5

t6

t7

r1

r2

r3

r4

r5

r6

r7

3D face database

Face recognition experiments have been conducted on the 3D facial models of the GavabDB database (publicly available at http://gavab.escet.urjc.es).

The database includes 61 people (45 male and 16 female). They are Caucasian and most of them are aged between 18 and

40.

For each person, 7 different models are taken differing in pose or facial expression, resulting in 427 facial models. 2 frontal and 2 rotated models

with neutral facial expression.

3 frontal models in which the person laughs, smiles or exhibits a random gesture.

Recognition: neutral expression

In the recognition experiments, for each person one of the two frontal models with neutral expression is used as reference (gallery) model. A gallery of 61 neutral models is obtained. The remaining 366 models are used as templates for the

recognition (probes). A stripe extent of 0.08 is used in the experiments.

Dynamic of the distance using the first 8 stripes

#12 #8 #4

Recognition: non-neutral expression

The proposed approach (IGS) provides a relatively high recognition accuracy also for variations in face expression. This can be intuitively motivated by the fact that the geodesic

distance between vertices of the mesh is almost invariant to moderate facial expression changes.

Large variability inside the given categories of face expressions. In the “random” category,

each subject is free to provide a generic facial expression.

Baseline comparison

Baseline comparison against the Iterative Closest Point (ICP) registration algorithm is performed. ICP monotonically converges to a local minima with respect to the

mean-square distance objective function computed between vertices of two models.

To avoid local minima in the ICP registration, the algorithm is initialized with the correspondence between fiducial pointsof the two models.

Current and future work

Facial stripes computed with respect to different/multiple fiducial points. Improvement of recognition accuracy w.r.t. expression variations.

Experiments on larger and heterogeneous face databases. Florida State University database (115 persons with 6 different

expressions each). The Face Recognition Grand Challenge database - FRGC v.2

(4007 3D scans of 466 persons, includes expression changes). The BJUT-3D large scale chinese face database (500 models,

neutral expression). Useful to study ethnic variation.

Comparison w.r.t. different 3D to 3D recognition approaches on common benchmark databases.