Preliminary Assessment of Discrimination of Twins in

Photographs based on Facial Blemishes

Nisha Srinivas1, Matthew Pruitt1, Gaurav Aggarwal1, Patrick Flynn1, Richard Vorder Bruegge2

1University of Notre Dame2Frederal Bureau of Investigation, Digital Evidence Lab

Problem Statement

• Investigate the usefulness of facial blemishes to distinguish between identical twins– Moles, Freckles, Scars, etc    

• Determine– Whether facial blemishes and locations can be

used to distinguish between identical twins– Whether the distributions of facial blemishes are

“more similar” for identical twins than unrelated persons?

Facial Blemishes

• Types of facial blemishes – Mole– Freckle– Freckle Group– Pimple– Darkened Patch– Lightened Patch– Splotchiness– Birthmark– Raised Skin– Pockmark– Scars

• Linear• Round

Mole Freckle and Freckle Group

Lightened Patch

Darkened Patch

Raised Skin Scar (Round) Pockmark Pimple

Proposed System Overview

Manual Annotation

Feature Extraction

Geometric Normalization

Point Cloud Matching

Biometric Verification

Performance Evaluation

Manual Annotation

Display Module

Annotation Module

Tool Module

Facial Blemishes Identified

by Observer 1

Facial Blemishes Identified

by Observer 2

Facial Blemishes Identified

by Observer 3

Facial Blemishes Identified by Observers

Moles

Freck

les

Freck

le Gro

up

Pimple

Birthmark

Darkened Patch

Lighte

ned Patch

Splotchiness

Raised Skin

Pockmark

Scar R

ound

Scar L

inear0

500

1000

1500

2000

2500

Observer 1Observer 2Observer 3

Types of Facial Blemishes

Coun

t

Total Number of Facial Blemishes Annotated by each ObserverObserver 1: 3785Observer 2: 2311Observer 3: 5100

Facial Blemishes Matching

N Nodes M Nodes

Moles

Matching Contd.

• The Edges in the bipartite graph correspond to potential matches• Each potential match has a cost associated with it which is a function of

the euclidean distance between the centroids of the blemishes being compared.

Matching Contd.

Match

Match

Similarity metric=Number of matches/Max(N,M)

Data

• Twin face images were collected at the Twins Days Festival in Twinsburg, Ohio in August 2009.

• High Resolution Images: 4310 rows x 2868 columns• Dataset Attributes

– Frontal (yaw=0), Indoor, No Glasses, Neutral Expression

• Number of Images: 295– Number of Subjects: 152– Number of Twins Pairs: 76

• Terminology– Target set: “gallery” of persons to be recognized– Query set: a set of images of unidentified persons to be

matched against the target set

Experimental Setup

• Perform two different experiments– Individual Observer Analysis

• Query set and Target set are annotated by same observer

– Inter-Observer Analysis• Query set is annotated by one observer and the Target

set is annotated by another observer– Observer 1 vs Observer 2– Observer 2 vs Observer 3– Observer 3 vs Observer 1

Subset of Facial blemishes

• FM={moles, freckles, freckle group, pimple, birthmark, darkened patch, lightened patch, splotchiness, raised skin, pockmark, scar round, scar linear}

• FM1=FM-{pimple}• FM2={moles, freckles}• FM3={moles, freckles, pimple}

Twins vs Twins Setup: Query Set Target Set

Subject 1, Twin A

Subject 2, Twin B

Subject 3, Twin A

Subject 4, Twin B

Match Comparison

Non-Match Comparison

Individual Performance Evaluation- Observer 3

Match Comparison

Non-Match Comparison

Query Set Target Set

Subject 1, Twin A

Subject 2, Twin B

Subject 3, Twin A

Subject 4, Twin B

Subject 5, Twin A

All vs All Setup:

Individual Performance Evaluation- Observer 3

Comparison: All vs All and Twins vs Twins

Inter-Observer Performance

Degradation in Performance when comparing facial marks annotated by different observers

Conclusion

• There appears a correlation between the distribution of facial blemishes across twins.

• The number of facial blemishes across twins appears to be similar.

• Facial blemishes can be used as a potential biometric signature.

• Consistent annotation is a challenging process – It is difficult to achieve consistency

Thank You

This research was supported by– NIJ/OJP award 2009-DN-BX-K231– FBI through TSWG/ARMY RDECOM contract

W91CRB-08-C-0093