Berk GökberkBoğaziçi University – Perceptual Intelligence Lab

Turkey

Principles of 3D Face Recognition

2

PromPromises and motivationsises and motivations 2D face recognition still requires help

Pose, expression, illumination variations

Possible solutions: other modalities? Video, infra-red, stereo, 3D

Promises of 3D facial recognition High-security applications 3D shape information invariance Pose and illumination problems can be solved Better facial feature localization

3

Some scenariosSome scenarios 3D-to-3D

2D-to-2D via 3D

2D-to-3D or 3D-to-2D

4

How do we get 3D facial How do we get 3D facial data?data?

Stereo cameras Quality: low to medium Speed: fast Problems: reconstruction

Structured-lights Quality: medium Speed: fast Problems: intrusive

Laser scanners Quality: high Speed: slow Problems: intrusive,

Shape from {shading, motion, video}

N/A

5

3D Facial Recognition 3D Facial Recognition PipelinePipeline

Features

Point Clouds

Depth Images

Pattern Classifie

r

3D Face Detection

Pre-proc. Face Normalization/Alignment

Fine Alignment

Noise Removal Hole FillingSmoothing

CroppingLandmark Finding

Coarse Alignment

6

3D Face Detection3D Face Detection This problem has not been touched so far! Simple heuristics such as nose tip In complex scenes, curvature analysis is generally

used

Ref: 3D face detection using curvature analysis, Alessandro Colombo, Claudio Cusano, Raimondo Schettini, Pattern Recognition 39 (2006) 444 – 455

7

Pre-processingPre-processing Artifact removal

Noise removal: spikes (filters), clutter (manually), noise (median filter)

Holes filling (Gaussian smoothing, linear interpolation, symmetrical interpolation)

8

Face Face Normalization/AlignmentNormalization/Alignment

Coarse alignment by Centre of mass, Plane fitted to the data Facial landmarks (eyes, nose tip)

Fine alignment ICP Warping Elastic deformations

9

3D Facial Features3D Facial Features

10

PC

-ICP gives the distance-Hausdorff-Too many points

11

PC

SN

-Enhanced Gaussian Image-Too many normals

-ICP gives the distance-Hausdorff-Too many points

12

PC

SN

PRO

-Sparse-Easy to compare-Not fully descriptive

-Enhanced Gaussian Image-Too many normals

-ICP gives the distance-Hausdorff-Too many points

13

PC

SN

PRO

CURV

Mean

Gaussian Shape Index

Principal directions

-Landmark detection-Segmentation-Sensitive to noise and the quality of data

-Sparse-Easy to compare-Not fully descriptive

-Enhanced Gaussian Image-Too many normals

-ICP gives the distance-Hausdorff-Too many points

14

PC

SN

PRO

CURV

Mean

Gaussian Shape Index

Principal directions

DI

PCA

-Landmark detection-Segmentation-Sensitive to noise and the quality of data

-Sparse-Easy to compare-Not fully descriptive

-Enhanced Gaussian Image-Too many normals

-ICP gives the distance-Hausdorff-Too many points

-Benefit from 2D literature-Easy to fuse with texture-Applicable to 2.5D only

15

PC

SN

PRO

CURV

Mean

Gaussian Shape Index

Principal directions

DI

TEX

PCA

PCA

Gabor

-Landmark detection-Segmentation-Sensitive to noise and the quality of data

-Sparse-Easy to compare-Not fully descriptive

-Enhanced Gaussian Image-Too many normals

-ICP gives the distance-Hausdorff-Too many points

-Benefit from 2D literature-Easy to fuse with texture-Applicable to 2.5D only

16

Baseline algorithmsBaseline algorithms When you design a new system, which algorithms

should be selected to show your algorithm’s superiority?

PCA

PCA

Texture & Shape

Match score for texture

Match score for shape

Use weighted sum

Perform ICP Output the surface matching errorFace A

Face B

Baseline Algorithm 1

Baseline Algorithm 2

17

What are the scenarios What are the scenarios tested?tested?

The discriminative power of texture and shape?

Taken from: “Three-dimensional face recognition”, Bronstein, A.M., and Bronstein, M.M., and Kimmel, R. International Journal of Computer Vision 2005, Vol.64 No.1 p.5-30

18

What are the scenarios What are the scenarios tested?tested?

The discriminative power of texture, shape, or texture+shape?

Pose variations No disciplined analysis to compare 2D and 3D under pose

variations Expression variations

Most of the databases do not contain expression variations No comparison to 2D

Illumination variations Image relighting Albedo estimation

19

Open Issues & ChallengesOpen Issues & Challenges Uncontrolled acquisition

Non-cooperative Different lighting conditions

Texture map + shape map inconsistenies Real-time 3D video data Computational complexity Issues related to performance assesment

Publicly available standard face databases Quality (resolution) of the data? Artifacts such as eyeglasses

Images taken from: A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition, Kevin W. Bowyer, Kyong Chang, Patrick Flynn, Computer Vision and Image Understanding 101 (2006) 1–15

Quick notes on Quick notes on 3D acquisition systems & 3D acquisition systems &

3D face databases3D face databases

21

3D Acquisition Systems3D Acquisition Systems Face specific

Biometrics A4Vision Geometrix

Modeling Cyberware Genex Inspeck Medeim Breuckmann

General sensors Minolta V-910

22

3D Face Databases3D Face Databases UND

275 subjects, 943 scans Shape + texture

FRGC 400 subjects, 4007 scans Shape + texture

3D_RMA 120 subject, 6 scans Shape only

GavabDB 61 subjects (9 scans) Shape only Pose, expression variations

USF database 357 scans

3DFS generator Custom face databases

12 persons to ~6000 persons (A4Vision)

UND USF

GavabDB 3DFS

23

ConclusionsConclusions 3D face recognition systems were proposed to

overcome expression, illumination, and pose challenges

Illumination correction is simpler Facial landmark localization is better Baseline recognizers

Combining depth maps with texture channel at the decision level

The core algorithm, ICP, has limited capabilities Not suitable for non-rigid deformations

Lots of research on shape channel representation Few in combining shape + texture

24

ReferencesReferences Boğaziçi University

Perceptual Intelligence Lab (PILAB) Signal and Image Processing Lab (BUSIM)

Relevant Papers: Comparative analysis of decision-fusion methods for 3D face recognition

B. Gökberk, L. Akarun, submitted for publication. Exact 2D-3D Facial Landmarking for Registration and Recognition

Salah, A.A., H. Çınar, L. Akarun, B. Sankur, submitted for publication. 3D Shape-based Face Representation and Feature Extraction for Face Recognition

B. Gökberk, M. O. İrfanoğlu, L. Akarun, Image and Vision Computing (accepted). 3D Face Recognition by Projection Based Methods

H. Dutağacı, B. Sankur, Y. Yemez, in SPIE Conf. on Electronic Imaging, 2006. 2D/3D facial feature extraction

Çınar Akakın, H., A.A. Salah, L. Akarun, B. Sankur, in SPIE Conf. on Electronic Imaging, 2006. Selection and Extraction of Patch Descriptors for 3D Face Recognition

B. Gökberk, L. Akarun, ISCIS 2005, LNCS, Vol. 3733 Springer 2005, pp. 718-727. 3D Face Recognition for Biometric Applications

L. Akarun, B. Gökberk, A. A. Salah, EUSIPCO2005, Antalya, Turkey. Rank-based Decision Fusion for 3D Shape-based Face Recognition

B. Gökberk, A. A. Salah, L. Akarun, AVBPA 2005, LNCS, Vol.3546 p.1019-1028. 3D Shape-based Face Recognition using Automatically Registered Facial Surfaces

M. O. İrfanoğlu, B. Gökberk, L. Akarun, ICPR2004, pp.183-186

Contact: Berk Gökberk, e-mail: gokberk@boun.edu.tr

Additional MaterialAdditional Material

26

High Resolution 2D vs 2D

3D Shape+Texture

3D Texture Only

3D Shape Only

Reference: Jonathon Phillips, FRGC Workshop,CVPR’05

Face Recognition Grand Face Recognition Grand ChallengeChallenge