Face Image Analysis: Recognition and Presentation Attack

Face Image Analysis:Recognition and Presentation Attack Detection

Shervin R. Arashloo

March 12, 2021

Shervin R. Arashloo Face Image Analysis March 12, 2021 1 / 41

Outline

1 Face RecognitionGraph MatchingStatistical Shape PriorMultiresolution and parallel optimisationClass-specific Discriminant Analysis

2 Face Presentation Attack DetectionOne-Class FormulationClient-specific one-class learningClassifier FusionOne-Class Fisher Discriminant AnalysisKernel Fusion


Face Recognition Major Challenges

Pose (out-of-plane rotation)

Illumination

Expression

etc.


Deformable Matching

Object Recognition and Matching

Bag of words

Discarding relational information between object primitives

Graph matching

Incorporating structural information for matching


Graphs and Hypergraphs

Nodes represent object primitives

Edges/hyperedges encode their dependencies


Deformation Model

E (x ; θ) =∑u∈V

θu(xu) +∑

(u,v)∈E

θuv (xu, xv )

labels are 2D displacements

θu measures the degree of similarity between graylevel contents of twoblocks

θuv enforces smoothness over deformation map


Decomposed Model

Complexity: O(νL2)L: Cardinality of the label set for the horizontal and vertical displacementsν: Number of nodesThe complexity scales quadratically in number of labels!

Decomposed model: modelling horizontal and vertical labels separately

Complexity of the decomposed model: O(νL)scales linearly in number of labels.


Decomposed Model

Complexity: O(νL2)L: Cardinality of the label set for the horizontal and vertical displacementsν: Number of nodesThe complexity scales quadratically in number of labels!

Decomposed model: modelling horizontal and vertical labels separately

Complexity of the decomposed model: O(νL)scales linearly in number of labels.


Statistical Shape Prior

Deformable models are broadly classified into two categories:

Free-form

Only general continuity and smoothness constraints are considered; can bematched to an arbitrary shape (e.g. Snake Model)

Parametric

Incorporate a general shape of the object of interest and encode specialattributes of an object and its variations-are more robust to occlusions andspurious structures (e.g. Active Shape Model)


Deformation Energy

The updated objective function:

E (x ; θ) =∑s∈V

θs(xs) +∑

(s,u)∈E

θsu(xs , xu)+θg (x)

where

θg (x) measures deviation from mean shape in the PCA space


Multiresolution Optimisation: Supercoupling Approach

Moving from coarser towardsfiner scales

Challenge: maintaining aconsistency between energyfunctions at different scales

The Supercoupling algorithm consists of two main steps:renormalisation and processing.

Renormalisation

Iteratively constructing coarser and coarser grids of nodes and acorresponding sequence of energy functions

Processing

Performing a multi-scale coarse-to-fine optimisation starting from thecoarsest scale moving towards the finest one


Multiresolution Optimisation: Supercoupling Approach

Moving from coarser towardsfiner scales

Challenge: maintaining aconsistency between energyfunctions at different scales

The Supercoupling algorithm consists of two main steps:renormalisation and processing.

Renormalisation

Iteratively constructing coarser and coarser grids of nodes and acorresponding sequence of energy functions

Processing

Performing a multi-scale coarse-to-fine optimisation starting from thecoarsest scale moving towards the finest one


Optimisation: Dual Decomposition

Two steps:

Decompose the problem into anumber of subproblems andsolve each one separately

Enforce consistency betweensubproblems

A large number of independently solvable subproblems motivate a parallelprocessing!

Graphical Processing Units

Array of highly threaded streaming multiprocessors

High speed shared memory visible to all processing elements as wellas a number of other types of memory



Two steps:









Two steps:








Speed-up Gains

Parallel Processing ∼ 24x

Multiresolution Analysis ∼ 5x

Other Techniques ∼ 1.8x

Overall ∼ 218x


Some Matching Results


Class-specific Discriminant Analysis

Subspace methods:

PCA, LDA, etc. (insufficiency of linear models)

Kernel methods: KPCA, KDA, etc.

Class-Specific Kernel Discriminant Analysis

Learns a class from a single labelled example (one-shot learning)

Results in subject-specific projections


Class-specific Discriminant Analysis

Subspace methods:

PCA, LDA, etc. (insufficiency of linear models)

Kernel methods: KPCA, KDA, etc.

Class-Specific Kernel Discriminant Analysis

Learns a class from a single labelled example (one-shot learning)

Results in subject-specific projections


Classification

Conventional hand-crafted descriptors used:

Local Binary Pattern (LBP)

Local Phase Quantisation (LPQ)

Binarised Statistical Image Features (BSIF)

For classification:

Kernel fusion over the three descriptors

Correspondences are taken into account


Labelled Faces in the Wild (LFW) Dataset

Real world variations of facial images such as pose, illumination,expression, occlusion, low resolution, blur, etc.Contains 13,233 images of 5,749 subjectsPair-matching problem

Figure: Sample images from the LFW dataset.


Evaluation Protocols


Results: Unsupervised


Results: Image-Restricted, No Outside Data


Observations

In the most relaxed scenario, state-of-the-art deep methods achievemore than 98% accuracy

Very large labelled data required to exploit the potential of deepnetworks

Graph-based methods very efficient in terms of the number of trainingdata

Typically high performance computing resources required to traindeep nets

Computationally demanding MAP inference in MRF vs. relatively fastoutput generation in deep networks


Journal Articles Relevant to Face Recognition

Arashloo, S.R. and Kittler, J., ”Energy Normalization for Pose-Invariant Face Recognition Based on MRF Model ImageMatching”, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 33, no. 6, pp. 1274-1280, Jun. 2011.

Arashloo, S.R., Kittler, J. and Christmas, W.J., ”Pose-Invariant Face Recognition by Matching on MultiresolutionMRFs Linked by Super-coupling Transform”, Computer Vision and Image Understanding , Elsevier, special issue ongraph-based representations in computer vision, vol. 115, issue 7, pp. 1073-1083, July 2011.

Arashloo, S.R. and Kittler, J., ”Fast Pose Invariant Face Recognition Using Supercoupled Multi-resolution MarkovRandom Fields on a GPU”, Pattern Recognition Letters, Elsevier, special issue on celebrating the life and work of MariaPetrou, vol. 48, pp. 49-59, Oct. 2014.

Arashloo, S.R. and Kittler, J., ”Class-Specific Kernel Fusion of Multiple Descriptors for Face Verification UsingMultiscale Binarised Statistical Image Features”, Information Forensics and Security, IEEE Transactions on, special issueon facial biometrics in the wild, vol. 9, no.12, pp. 2100-2109, Dec. 2014.

Arashloo, S.R., ”Incorporating Point Distribution Model Priors into MRFs Using Convex Quadratic Programming”,Machine Vision and Applications, Springer, vol. 27, no. 6, pp. 821-832, Aug. 2016.

Arashloo, S.R., ”A Comparison of Deep Multilayer Networks and MRF Matching Models for Face Recognition in theWild”, Computer Vision, IET , vol. 10, no. 6, pp. 466-474, Sep. 2016.

Arashloo, S.R., ”Multiscale Binarised Statistical Image Features for Symmetric Face Matching Using MultipleDescriptor Fusion Based on Class-Specific LDA”, Pattern Analysis and Applications, Springer, pp. 1-14, May 2015.


Face Presentation Attack Detection

Problem:An unauthorised subject tries to get illegitimate access to a facerecognition system by presenting fake biometrics traits

Typical presentation attack instruments:

Print

Video Replay

Mask

etc.


Face Presentation Attack Detection

Problem:An unauthorised subject tries to get illegitimate access to a facerecognition system by presenting fake biometrics traits

Typical presentation attack instruments:

Print

Video Replay

Mask

etc.


Points of Attack to a Biometrics System


Samples captured by a recognition system

(a) corresponds to genuine (bona fide) samples(b),(c) and (d) represent presentation attacks 1

1images from the ”MSU Mobile Face Spoofing Database (MSU MFSD)” dataset.


Samples captured by a recognition system

(a) corresponds to genuine (bona fide) samples(b),(c) and (d) represent presentation attacks 1

1images from the ”MSU Mobile Face Spoofing Database (MSU MFSD)” dataset.


Conventional approach

The Conventional approach is Two-Class Classification:

Training samples include both bona-fide (genuine) and attack samples

A binary classifier is trained to classify an image (sequence) as eitherbona-fide or attack

Drawbacks:

High cost of collecting attack samples

Low generalisation

Different imaging conditionsNovel attack types unseen during training!


Conventional approach

The Conventional approach is Two-Class Classification:

Training samples include both bona-fide (genuine) and attack samples

A binary classifier is trained to classify an image (sequence) as eitherbona-fide or attack

Drawbacks:

High cost of collecting attack samples

Low generalisation

Different imaging conditionsNovel attack types unseen during training!


One-Class Formulation of Face PAD problem

Genuine samples considered as target observations and attacks asanomalies

Our approach learns from genuine data only: not biased towardsspecific attack types!

Goal: Characterise the support domain of probability density function ofgenuine samples


One-Class Formulation of Face PAD problem

Genuine samples considered as target observations and attacks asanomalies

Our approach learns from genuine data only: not biased towardsspecific attack types!

Goal: Characterise the support domain of probability density function ofgenuine samples


Presentation Attack Detection: Common approach

The common approach is Subject-Independent Detection:

A common classifier is trained to detect PA w.r.t. all subjects

Drawback:

Ignores any class-specific information useful for PAD


Client-specific modelling

Deploying client-specific information for face spoofing detection

Subject-specific score distributions motivate a distinct threshold foreach client



Classifier Fusion

Motivation: Different one-class learners+diverse representations

2

2J. Kittler, M. Hatef, R. P. W. Duin and J. Matas, ”On combining classifiers,” in IEEE Transactions on Pattern Analysis

and Machine Intelligence, vol. 20, no. 3, pp. 226-239, March 1998, doi: 10.1109/34.667881.


Classifier Fusion

Motivation: Different one-class learners+diverse representations

2

2J. Kittler, M. Hatef, R. P. W. Duin and J. Matas, ”On combining classifiers,” in IEEE Transactions on Pattern Analysis

and Machine Intelligence, vol. 20, no. 3, pp. 226-239, March 1998, doi: 10.1109/34.667881.


Diversity in Experts

Multiple regions:

Multiple Deep CNN’s:

GoogleNet

ResNet50

VGG16

Multiple One-class learners:

One-class Support Vector Data Description

Mahalanobis distance

Gaussian mixture model


The Impact of Classifier Fusion


One-Class Fisher Discriminant Analysis

The Fisher classifier:

F(β) =β>Σbβ

β>Σwβ

Σb: between-class scatter matrixΣw : within-class scatter matrixβ: the Fisher discriminant

Originally developed for two-classclassification but can be adapted to aone-class setting!




F(β) =β>Σbβ

β>Σwβ






F(β) =β>Σbβ

β>Σwβ




Regression-Based Formulation

Solving for the Fisher discriminant requires costly eigendecompositionof dense matrices

Not convenient to impose regularisation on the discriminant forimproved generalisation performance

Regularised regression-based reformulation in the Hilbert space

minθ‖θ‖2

2 +δ

n

n∑i=1

(1− θ>υ(xi ))2

Tikhonov regularisation

Or its dual problem as

maxω−ω>Kω − σω>ω + 2ω>1

σ = n/δK: kernel matrix1: denotes an n-dimensional vector of ones






minθ‖θ‖2

2 +δ

n

n∑i=1

(1− θ>υ(xi ))2










minθ‖θ‖2

2 +δ

n

n∑i=1

(1− θ>υ(xi ))2










minθ‖θ‖2

2 +δ

n

n∑i=1

(1− θ>υ(xi ))2










minθ‖θ‖2

2 +δ

n

n∑i=1

(1− θ>υ(xi ))2






Kernel Fusion

Fusing multiple representations via a sum rule:K = K1 + K2 + · · ·+ KJ

Diversity in the representationsMultiple Regions

Different Deep CNN’s

GoogleNetResNet50VGG16


Kernel Fusion Evaluation Results

Unseen attack evaluation protocol


Multiple Kernel Learning

The ideaInstead of using fixed combination rules, learn linear combination weights

Objective function

minβ

maxα

−α>(∑J

j=1 βjKj)α− δα>α+ 2α>1

s.t. β ≥ 0,R(β)

kernel weights

Different possibilities for regularisation R(β):

`p-norm ‖β‖pp ≤ 1; p ≥ 1

induces sparsity

mixed (r , p)-norm∥∥ββ>∥∥

r ,p≤ 1; r , p ≥ 1

induces sparsityenables interaction between kernels

Both regularisations lead to convex optimisation problems!




Objective function

minβ

maxα

−α>(∑J


s.t. β ≥ 0,R(β)

kernel weights


`p-norm ‖β‖pp ≤ 1; p ≥ 1

induces sparsity


r ,p≤ 1; r , p ≥ 1






Objective function

minβ

maxα

−α>(∑J


s.t. β ≥ 0,R(β)

kernel weights


`p-norm ‖β‖pp ≤ 1; p ≥ 1

induces sparsity


r ,p≤ 1; r , p ≥ 1






Objective function

minβ

maxα

−α>(∑J


s.t. β ≥ 0,R(β)

kernel weights


`p-norm ‖β‖pp ≤ 1; p ≥ 1

induces sparsity


r ,p≤ 1; r , p ≥ 1






Objective function

minβ

maxα

−α>(∑J


s.t. β ≥ 0,R(β)

kernel weights


`p-norm ‖β‖pp ≤ 1; p ≥ 1

induces sparsity


r ,p≤ 1; r , p ≥ 1






Objective function

minβ

maxα

−α>(∑J


s.t. β ≥ 0,R(β)

kernel weights


`p-norm ‖β‖pp ≤ 1; p ≥ 1

induces sparsity


r ,p≤ 1; r , p ≥ 1




Abnormality and Novelty Detection

Abnormality Detectiondetect abnormal observations whenthe classifier is trained using a set ofnormal samples of the correspondingclass

Novelty Detectionassess the novelty of a new samplebased on previously observed samples

Figure: (a) Abnormal image detection: Top three rows arenormal images from the PASCAL dataset. Bottom three rowsare abnormal images from the Abnormal 1001 dataset. (b)Novelty detection: images from the Caltech256 dataset.


`p-norm Evaluation Results


(r , p)-norm Evaluation Results


Journal Articles Relevant to Presentation Attack Detection

Arashloo, S.R. and Kittler, J., ”An Anomaly Detection Approach to Face Spoofing Detection: A New Formulation andEvaluation Protocol”, IEEE Access, vol. 5, pp. 13868-13882, 2017.

Arashloo, S.R. and Kittler, J., ”Robust One-Class Kernel Spectral Regression”,Neural Networks and Learning Systems, IEEE Transactions on, vol. 32, no. 3, pp. 999-1013, March 2021, doi:

10.1109/TNNLS.2020.2979823.

Fatemifar, S., Arashloo, S.R., Awais, M., Kittler, J., ”Client-Specific Anomaly Detection for Face Presentation AttackDetection”, Pattern Recognition, Elsevier, vol. 112, 107696, 2021.

Arashloo, S.R., ”Unseen Face Presentation Attack Detection Using Sparse One-Class Multiple Kernel FusionRegression”, Circuits and Systems for Video Technology, IEEE Transactions on, doi: 10.1109/TCSVT.2020.3046505,2020.

Arashloo, S.R., ”`p -Norm Multiple Kernel One-Class Fisher Null-Space”, under review.

Arashloo, S.R., ”Mixed (r, p)-Norm One-Class Multiple Kernel Fisher Null-Space”, in preparation.


Documents

Face Image Analysis: Recognition and Presentation Attack