Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Data Insufficiency in Sketch Versus Photo Face Recognition
Jonghyun ChoiAbhishek Sharma, David W. Jacobs, Larry S. Davis
Ins=tute of Advanced Computer StudiesUniversity of Maryland, College Park
CVPR Workshop in Biometrics 2012
17 June 2012
Sketch-Photo Face Recognition•Why is it important?-‐ Automated criminal search by forensic sketch can reduce the 8me of crime inves8ga8on
Matching
GalleryProbe
Image Courtesy by B. Klare from “Matching Forensic Sketches to Mug Shot Photos”, PAMI 2011
Popular Benchmark in Literature
• CUFS dataset[1]
-‐ Public benchmark dataset-‐ Promo8ng the ini8al research‣ A controlled dataset
✓Well lit photos, neutral expression, frontal poses
-‐ Many approaches evaluated so far
[1] Tang and Wang, “Face Photo Recogni8on using Sketch”, ICIP 2002
Timeline of Research
2002
First CUFS dataset released (188) Baseline
ICIP2003
CUFS dataset expanded (608),
Bayesian Approach
ICCV
Non-linear approach
CVPR
Baseline
T.CSVT
Journal Extension
2006ECCV
Common Discriminant
Feature Extraction
2007 2008 2010 2011 2012ICASSP
E-HMM and selected ensemble
T.CSVT
E-HMM and selected ensemble
Journal Extension
ECCV
Lighting and Pose Invariant
CUFS dataset expanded (1,800),
Coupled Information
Theoretic Encoding
CVPR
Coupled Spectral Regression
2009CVPR
20052004
Local Feature based Discriminant
Analysis (LFDA)
PAMI
• On the CUFS Dataset
65
76.667
88.333
100
Accuracy
Summary of Previous ResultsApproach #Train #Test Rate (%)
Sketch SynthesisTang and Wang 88 100 71Tang and Wang 306 300 81.3
Nonlinear 306 300 87.67E-HMM 306 300 95.24
MS MRF+LDA 306 300 96.3MS MRF+LDA 88 100 96
MS MRF+W.PCA 88 100 99Modelling Modality Gap
PLS-subspace 88 100 93.6Klare et al. 306 300 99.47
CITP 306 300 99.87
Nearly perfect result!
Is the problem solved?
Yes
Yes CUFS Datasetfor
CUFS Dataset• 606 photo-‐sketch pairs-‐ Random par88oning for training/tes8ng
• Combined with CUFSF (2011) from FERET-‐ Total 1,800 photo-‐sketch pairs• viewed sketch dataset-‐ Provides well-‐aligned photo-‐sketch pairs‣ Good for analysis of difference in sketch
and photo domains without any other factors’ interven8ons
• viewed sketch dataset
CUFS Dataset (Cont’d)
• viewed sketch dataset-‐ Sketch is drawn by ar8sts‣ Capturing subtle edge similarity (e.g. hair style)-‐ Pre-‐processed to make them well aligned as well
CUFS Dataset (Cont’d)
• viewed sketch dataset-‐ Sketch is drawn by ar8sts‣ Capturing subtle edge similarity (e.g. hair style)-‐ Pre-‐processed to make them well aligned as well
Real-World Scenario for Sketch-Photo Face Recognition1. Eye-‐witness describes the criminal’s
facial traits verbally2. Forensic ar;sts draw the sketch
according to the verbal descrip;on• Forensic sketch[1]
[1] B. Klare et al.,”Matching Forensic Sketches to Mug Shot Photos”, PAMI 2011
➡ Viewed sketch is not realistic
*Images courtesy from B. Klare
Insufficiency of the CUFS Dataset• Well-‐alignment of CUFS dataset-‐ Good for ini8al research on domain difference
(photo-‐sketch) w/o interven8on of other factors-‐ But simplifies the problem too much‣ Simple edge matching techniques might work
‣ Ignores true variability of sketch-‐face recogni8on:
✓ Mis-‐alignment of fiducial components
✓ Seman8c descrip8on of shapes of fiducial component
✓ No precise descrip8on for subtle difference (e.g. hair style)
Today, We Show• To obtain good result on CUFS dataset-‐ Discrimina8ve edge matching technique is
enough‣ Outperforms state of the art in face
iden8fica8on sebng-‐ No effort in reducing modality gap is required‣ Thus no training set except gallery set is
required
‣ But even outperforms state of the art in bigger set
Discriminative Edge Matching
• Edge features-‐ Gabor wavelet response‣ Blurred edge tendency: Macro edge
-‐ CCS-‐POP[1]
‣Micro-‐edgelet
• Discrimina8ve weight on the feature-‐ Build a one-‐vs-‐all PLS model
[1] Choi et al.,”A Complementary Local Feature for Face Iden8fica8on”, WACV 2012
Partial Least Sqaures• A supervised dimension reduc8on technique by
maximizing covariance of weighted independent variable (X) and weighted dependent variable (Y)
• Using NIPALS algorithm[1] to obtain the regression solu8on from X to Y
[1] H. Wold, Par8al Least Squares, 1985
feature label
w = max
|w|=1cov(Xw, Y )
2
Overall System Diagram[1]
GalleryBuild “One-‐vs-‐All” PLS regression models
Posi=ve Samples
Nega=ve Samples
…Model
A…
Posi=ve Samples
Nega=ve Samples
…
ModelZ
…
…
A B
DE
F
G
Z
Tes=ng Regression responses
Model A Model B Model C Model D Model Z
…
C
Iden=fica=on Result
…
Model Building (Training)
PLSR
PLSR
ProbeRegression
C
[1] Schwartz et al., “A Robust and Scalable Approach to Face Iden8fica8on”, ECCV 2010
Experimental Setup• CUFS+CUFSF (CUFS) dataset-‐ 1,800 pairs of sketch-‐face-‐ No extra training set: Use all pairs for test-‐ Photo to Sketch / Sketch to Photo experiments
• Comparison to previous work• Various image cropping-‐ Tight/Loose crop-‐ Horizontal/Ver8cal strip crop-‐ Fiducial component crop
Experimental ResultsApproach #Train #Test Rate (%)
Sketch SynthesisTang and Wang [24,26] 88 100 71Tang and Wang [25] 306 300 81.3
Nonlinear [13] 306 300 87.67E-HMM [4,35] 306 300 95.24
MS MRF+LDA [30] 306 300 96.3MS MRF+LDA (from [31]) 88 100 96MS MRF+W.PCA [31] 88 100 99
Modelling Modality GapPLS-subspace [21] 88 100 93.6Klareet al. [9] 306 300 99.47CITP [32] 306 300 99.87
Ours (Gabor only) 0 300 99.80 ± 0.44Ours (CCS-POP only) 0 300 95.53 ± 0.90
Ours(CCS-POP+Gabor) 0 300 100Ours (Gabor only) 0 1,800 99.50
Ours (CCS-POP only) 0 1,800 96.28Ours(CCS-POP+Gabor) 0 1,800 99.94
• Test with 100 samples
70
77.5
85
92.5
100
71
9699
93.6
100
2002 [24]2006 [26] 2010 [31]
2010 [31]2011
Ours
Accuracy (%)
Approach #Train #Test Rate (%)Sketch Synthesis
Tang and Wang [24,26] 88 100 71Tang and Wang [25] 306 300 81.3
Nonlinear [13] 306 300 87.67E-HMM [4,35] 306 300 95.24
MS MRF+LDA [30] 306 300 96.3MS MRF+LDA (from [31]) 88 100 96MS MRF+W.PCA [31] 88 100 99
Modelling Modality GapPLS-subspace [21] 88 100 93.6Klareet al. [9] 306 300 99.47CITP [32] 306 300 99.87
Ours (Gabor only) 0 300 99.80 ± 0.44Ours (CCS-POP only) 0 300 95.53 ± 0.90
Ours(CCS-POP+Gabor) 0 300 100Ours (Gabor only) 0 1,800 99.50
Ours (CCS-POP only) 0 1,800 96.28Ours(CCS-POP+Gabor) 0 1,800 99.94Our Method
Experimental Results
Experimental Results• Test with 300 samples
80
85
90
95
100
81.3
87.67
95.24 96.3
99.47 99.87 100
2003 [25]2005 [13]
2007,8 [4,35]2009 [30]
2011 [9]2011 [32]
Ours (G+C)
Accuracy (%)
Approach #Train #Test Rate (%)Sketch Synthesis
Tang and Wang [24,26] 88 100 71Tang and Wang [25] 306 300 81.3
Nonlinear [13] 306 300 87.67E-HMM [4,35] 306 300 95.24
MS MRF+LDA [30] 306 300 96.3MS MRF+LDA (from [31]) 88 100 96MS MRF+W.PCA [31] 88 100 99
Modelling Modality GapPLS-subspace [21] 88 100 93.6Klareet al. [9] 306 300 99.47CITP [32] 306 300 99.87
Ours (Gabor only) 0 300 99.80 ± 0.44Ours (CCS-POP only) 0 300 95.53 ± 0.90
Ours(CCS-POP+Gabor) 0 300 100Ours (Gabor only) 0 1,800 99.50
Ours (CCS-POP only) 0 1,800 96.28Ours(CCS-POP+Gabor) 0 1,800 99.94Our Method
Experimental Results• Test with 1,800 samples-‐ Compared to the results tested with 300 samples
99.4
99.55
99.7
99.85
100
99.47
99.8799.94
2011 [9] w/3002011 [32] w/300
Ours (G+C) w/1800
Accuracy (%)
Approach #Train #Test Rate (%)Sketch Synthesis
Tang and Wang [24,26] 88 100 71Tang and Wang [25] 306 300 81.3
Nonlinear [13] 306 300 87.67E-HMM [4,35] 306 300 95.24
MS MRF+LDA [30] 306 300 96.3MS MRF+LDA (from [31]) 88 100 96MS MRF+W.PCA [31] 88 100 99
Modelling Modality GapPLS-subspace [21] 88 100 93.6Klareet al. [9] 306 300 99.47CITP [32] 306 300 99.87
Ours (Gabor only) 0 300 99.80 ± 0.44Ours (CCS-POP only) 0 300 95.53 ± 0.90
Ours(CCS-POP+Gabor) 0 300 100Ours (Gabor only) 0 1,800 99.50
Ours (CCS-POP only) 0 1,800 96.28Ours(CCS-POP+Gabor) 0 1,800 99.94
Our Method
Different Cropping
• Tight/Loose Crop
• Horizontal/Ver8cal Strip Crop
• Fiducial Component Crop
Results on Tight/Loose Cropping
97
97.75
98.5
99.25
100
97.6
98.8
99.94
TightMedium
Loose
Accuracy (%)
Results on Fiducial Component Cropping
77
82.75
88.5
94.25
10095.11
77.67 79.585.22
Ocular Nose MouthHair
Accuracy
Results on Strip Cropping
77
82.75
88.5
94.25
100
1 2 3 4
H V Hc Vc
Discussion & Conclusion
• A simple discrimina8ve edge analysis can perform well in overly-‐reduced problem of sketch-‐photo matching
• Now is the 8me to move on to more challenging dataset
•We suggest a guideline for new dataset (Please refer to our paper)
Q/AThank you!
Authors are supported byMURI Grant N00014-08-10638 from Office of Naval Research