View
217
Download
4
Embed Size (px)
Citation preview
Face Recognition from Face Motion Face Recognition from Face Motion Manifolds using Robust Kernel RADManifolds using Robust Kernel RAD
Ognjen Arandjelović
Roberto Cipolla
Funded by Toshiba Corp. and Trinity College, Cambridge
Face RecognitionFace Recognition
• Single-shot recognition – a popular area of research since 1970s
• Many methods have been developed
• Bad performance in presence of:
– Illumination variation
– Pose variation
– Facial expression
– Occlusions (glasses, hair etc.)
Eigenfaces
Wavelet methods3D MorphableModels
Face Recognition from VideoFace Recognition from Video
• Face motion helps resolve ambiguities of single shot recognition – implicit 3D
• Video information often available (surveillance, authentication etc.)
Recognition setup Training stream Novel stream
Face ManifoldsFace Manifolds• Face patterns describe manifolds which are:
– Highly nonlinear, and
– Noisy, but
– Smooth
Facial features Face pattern manifold Face region
Limitations of Previous WorkLimitations of Previous Work
• In this work we address 3 fundamental questions:
– How to model nonlinear manifolds of face motion
– How to choose the distance measure
– How and what noise sources to model
?
Comparing Nonlinear ManifoldsComparing Nonlinear Manifolds
Closest-neighbour
Majority vote + Eigen/Fisherfaces
Mutual Subspace Method
Principal angles
Our method, KLD method of Shakhnarovich et al.
Information-theoretic measures
RAD: How well can we distinguish between P(x) and Q(x)?
1 1 1( , ) ( ( || ) ( || ) )RAD KL KLD P Q D P Q D Q P
KLD vs. RAD vs. …KLD vs. RAD vs. …
KLD: How well does P(x) explain Q(x)?
P(x) Q(x)
( || )KLD P Q
Q(x)
P(x)
Nonlinear RADNonlinear RAD
• Use closed form expression for KLD between Gaussians in KPCA space
Kernel PCA
Approximately linear manifolds
Highly nonlinear manifolds
Input space KPCA space
1 12|| log
Tqq p q pKL p q q
p
D p q Tr N
ΣΣ Σ Σ x x x x
Σ
0.6 ,( , ) ( , ) i j
i j i jk e x xx x x x
RBF Kernel
RegistrationRegistration
• Linear operations on images are highly nonlinear in the pattern space
• Translation/rotation and weak perspective can be easily corrected for directly from point correspondences
– We use the locations of pupils and nostrils to robustly estimate the optimal affine registration parameters
Translationmanifold
Skewmanifold
Rotationmanifold
Registration Method UsedRegistration Method Used
• Feature localization based on the combination of shape and pattern matching (Fukui et al. 1998)
Detect features
Crop & affineregister faces
Feature Tracking ErrorsFeature Tracking Errors
• We recognize two sources of registration noise:
– Low-energy noise due to the imprecision feature detector
– High-energy noise due to incorrectly localized features
20 automatically cropped and registered faces from a video sequence
Outliers – highenergy noise
Imperfect alignment offacial features – low energy noise
Low Energy NoiseLow Energy Noise
• Estimate misregistration manifold noise energy
• Augment data with synthetically perturbed samples = thickening of the motion manifold
• Synthetic data explicitly models the variation
Original data Original + synthetic data
Outliers – High Energy NoiseOutliers – High Energy Noise
• Outliers are due to incorrect feature localization
• High energy noise – far from the ‘correct’ data mean in KPCA space
Outliers
Manifold of correctly registered faces (+low energy noise)
RANSAC for Robust KPCARANSAC for Robust KPCA
Minimal, randomsample
Kernel PCAprojection
Valid datacount
IterationOutliers
Algorithm: The Big PictureAlgorithm: The Big Picture
Input frames
Distance
Original + syntheticdata
Valid data inKPCA space
Face Video DatabaseFace Video Database
• No standard database exists – we collected our own data
• 160 people, 10 different lighting conditions (each condition twice i.e. 20 video sequences per person)
Evaluation ResultsEvaluation Results
• Robust Kernel RAD outperformed other methods on all databases
• Average recognition rate of 98%
Method Limitations / Future WorkMethod Limitations / Future Work
• Less pose sensitivity (why should input and reference distributions be similar?)
• Illumination invariance is not addressed
Same person,different illumination
Novel personSee Arandjelović et al., BMVC 2004
For suggestions, questions etc. please contact me at: [email protected]