A Computational Theory of Writer Recognition Catalin I. Tomai Department of Computer Science and Engineering Dissertation Defense

A Computational Theory of Writer Recognition

Catalin I. TomaiDepartment of Computer Science and EngineeringDepartment of Computer Science and Engineering

Dissertation Defense

Outline

• Problem Domain• A Computational Theory of Writer Recognition • Algorithms and Representations

– Pictorial Similarity Examination– Identification of Document Elements

• Complete Information• Partial Information

– Extraction of Document Elements• Classifier Combination • Dynamic Classifier Selection

– Determination of Discriminating Elements– Pattern Examination and Classification

• Writer Verification• Writer Identification

• Conclusions

Writer Recognition

• Biometrics:– Physiological: face, iris pattern, fingerprints– Behavioral: voice, handwriting

• Forensic Sciences: Court testimony: Daubert vs. Merrell Dow (1993 -Supreme Court) – forensic techniques need to be based on testing, error rates,

peer review and acceptability

• Practiced by Forensic Document Examiners (FDE’s)– Experts perform significantly better than non-professionals

[Kam et. al 1994,1997]

• Semi-automatic computer-based systems:– FISH (Germany 1970), SCRIPT (Holland, 1994)– Used by: Government agencies (IRS,INS,FBI)

Handwriting Analysis Taxonomy

Handwriting Analysis

Synthesis Recognition Personality identification(Graphology)

On-line Off-line Writer VerificationWriter Identification

Natural Writing Forgery Disguised Writing

Handwriting Recognition

WriterRecognition

Text Dependent

Text Independent

• Writer Recognition

• Handwriting Recognition

Problem Domain

Identification Model

Verification Model

Which writer?1,…,n

Yes, Same Writer No, Different Writer

Recognition Model started

Problem Domain

Individuality: no two writings by different persons are identicalVariability: no two writings by the same person are identical

Writer A

Writer B

. . .

. . .

. . .

. . .

Problem Domain – Previous WorkAuthors Writer

Identification (%)

Writer Verification

(%)

Writer Sets Features Classification

[Steinke 1981]

99 - 20 writers/40 documents each

- -

[Said

1999]

96 - 40 writers/1000 documents

Textural KNN

[Zhu

2001]

95.7 - 17 writers/chinese alphabet

Textural Euclidean Classifier

[Srihari 2001]

89 95 1000 writers/3000 documents

Character and Document

Neural Nets

[Bensefia 2002]

97.7 - 88 writers Stroke-Based KNN

[Kam 1997]

- 93.5-professionals

61.7-nonprofessionals

144 pairs of documents

- 100 document examiners

- partial solutions, no integrated framework

- features do not reflect the experience of human examiners

- small number of documents/writers

- document elements recognition/extraction overlooked

Computational Theory of Writer Recognition

EvaluationAnalysis

Query Documents

Comparison

Likelihood Ratio

Writer Identity

2. Representation

1. Theory: developed based on studies on how human experts discriminate between

handwritings of different people

Determination of Discriminating

Elements

Pictorial Similarity

Examination

Comparison

Identification of Document Elements

Pattern Examination

Classification

ComparisonAnalysis Analysis Evaluation

2. Algorithms: pattern recognition/machine learning/computer vision

Document elements (characters/words)

Discriminating Elements (Habits)

Textural/Statistical/Structural features

Inspired by the Computational Theory of Computer Vision [Marr,1980]

3. Hardware/Software



Elements


Examination

Query Documents

Comparison


Pattern Examination

Classification

Likelihood Ratio


Writer Identity

Pictorial SimilarityTheory : eliminate dissimilar candidates Representation: handwritten documents = texturesAlgorithms:

– Wavelets - humans process images in a multi-scale way. – Gabor Filters – reasonably model most spatial aspects of the visual cortex

Pre-processing

…

…

Gabor filter bank

θ – [0,45,90,135]

f – [2,4,8,16,32]

and combinations

DCT

Daubechies

Haar

Antonini

…

Odagard

What is the most descriptive Wavelet Transform/Filter for a given handwriting?

GLCM

1 60

…

12

28

Feature Vector


ApproximationAlgorithm[GLCM]

TrainExemplars Database

Applyalgorithms on

each image

MeanFeaturesDatabase

Training Process

Query Process

Query Document

QueryFeatures Database

Decision Fusion/Find BestAlgorithm

Query Exemplars Database

TestExemplars Database

Return Most Similar Exemplars

Features Database

……

A2

A1

A28

A3

A15

A7

…

A1

A8

A3…

…

Ranked Algorithmsfor each Handwriting Exemplar

Classsifier/

Class

Results

Algorithm Set Retrieval Performance

Adaptive Scheme vs.

Best Algorithm

Train Set: 1927 images (167 writers)Test Set: 1985 images (173 writers)

Results

Document a15 Adaptive

Scheme

Best

Scheme

Document a15 Adaptive

Scheme

Best

Scheme

0001 0.341 0.330 0.375 0023 0.348 0.500 0.500

0003 0.288 0.205 0.288 0025 0.280 0.265 0.280

0005 0.144 0.091 0.265 0027 0.182 0.197 0.227

0007 0.402 0.470 0.500 0029 0.220 0.212 0.227

0011 0.265 0.333 0.333 0031 0.023 0.053 0.053

0015 0.189 0.220 0.220 0033 0.295 0.439 0.455

0017 0.144 0.182 0.220 … … … …

0019 0.424 0.424 0.455 Average 0.258 0.288 0.320

0

10

20

30

40

50

60

70

a5 a6 a7 a8 a9 a10 a11 a12 a13 a14 a15 a16 a17 a18 a19

rank1

rank2

rank3

rank4

rank5

rank6

rank7

rank8

rank9

rank10

rank11

Algorithm

Frequency

Conclusion

– Filtering properties: Allows us to eliminate approx 70% of the test set exemplars (the ones non-similar to the query image).

– Performance: outperforms the use of a unique filter for all query images.

– Extensibility: more algorithms/decision schemes can be added to the mix.

– Limited overhead: by using pre-computed features and mean feature vectors database.



Elements


Examination

Query Documents

Comparison


Pattern Examination

Classification

Likelihood Ratio


Writer Identity

Complete Information(Transcript Mapping)

Partial Information(Heuristic and Bayesian)

16

Country name Postal Code structure

Country name Postal Code Structure

Country name Postal Code structure

Canada LDL DLD Ireland Czech Republic DDDDD

England L/LA/ADLL Brasil DDDDD-DD Latvia DDDD

Germany DDDDD New Zealand DDDD Singapore DDDDDD

… … … … … …

Document Elements Extraction

Theory: Extract document elements (characters/words/other)

From Nov 10 1999 Jim Elder ….

CompleteInformation

Partial Information

+

From

Nov...

No Information

+

Mexico

+ ? From

...

Transcript

17

Complete Information-Transcript Mapping

. . .

From Nov 10 1999 Jim Elder …

Word Recognition

Build Word-Hypothesis Set

Next Line/Refine Results

Dynamic Programming

From

Nov

1999

WordRecognition

18

Transcript Mapping

19

Transcript Mapping

20

Results

From 3000 document images

– almost ½ million word images extracted

– more than 2 million character images extracted

0

10

20

30

40

50

60

8 9 4 2 k j b B E F

90-99

80 - 89

70-79

60 - 69

50 - 59

40 - 49

30 - 39

20 - 29

10 to 19

0 - 9

0

2

4

68

10

12

14

16

8 9 4 2 k j b B E F

ErrorError

Rate

Error

Rate

21



Elements


Examination

Query Documents

Comparison


Pattern Examination

Classification

Likelihood Ratio


Writer Identity


Partial Information(Heuristic and Bayesian)

22

Document Element Extraction-Partial Information

Theory: Partial information is available

• combine heterogeneous information• missing data/noisiness• interpretability

Script

Structure

Partial recognition results

Example: Foreign Mail Recognition (FMR) - extract character/word images from mail pieces sent to foreign addresses

Mail Stream

23

Foreign Mail Recognition

Domestic Mail Recognition

(DMR)

Foreign Mail Recognition (FMR)

Lexicon Large Large

Postal Code Structure Small variation Large variation

Postal Code Length Usually Fixed Variable

Script Small set Large set

Address Elements Positioning

Small configuration set

Large configuration set

24

Partial Information-Foreign Mail-Samples

25

Partial Information – Heuristic Solution

PC candidate Position PC- Format

Country Confidence PC FormatFrance 3.04 dddddLatvia 4.58 dddd

75014 France

2:1:4 ddd

2:1:5 invalid

2:2:5 dd ddddd

FKANCE

REORDER/ELIMINATE

WMR

HNNCR

2:1:2 ddddd

Country Confidence PC-FormatFrance 2.04 dddddGreece 5.62 ddddd

Greece 5.62 ddddd …. ….. ...

... ... ...

Id Country

Configuration Frequency

Country

City State

PC

1 France 1,1,1 2,2,2

0,0,0

2,1,2 0.40

2 France 1,1,1 3,2,2

0,0,0

3,1,2 0.11

3 France 1,1,1 3,1,1

0,0,0

2,1,1 0.39

4 France 2,1,1 1,1,2

0,0,0

1,2,2 0.05

5 Greece 1,1,1 2,1,2

0,0,0

2,2,2 0.5

6 Greece 2,1,1 0,0,0

0,0,0

1,1,1 0.3

… … … … … … …

Country Candidates

26

Partial Information – Bayesian SolutionStructural Features:

2 No Of Lines3,4,5 No Of Strokes (Lines

1,2,3)8,9,10,11 Line Length (Lines

1,2,3)14 IsGap – gap on the last

line

Script Differentiation Features

12 IsLatin(0/1) 13 IsCKJ(0/1)

Address Block Component Features

15 PostalCodeLength16 PostalCodeLine – line on which the PostCode is

located18 PostalCodeFormat (e.g. ####,####,#a#a#a, etc)19 PostalCodeHasDash (e.g. ####-###)17 PostalCodeIsMixed(0/1) – is the Postcode of

digits only or not20 PostalCodeCountryOrder (0/1) – is the

PostCode located before/after the country name

1

2

3

4

Recognition Features

6 CountryNameFirstLetter (from character recognizers)

7 CountryNameLastLetter (from character recognizers)

21 Continent22 CountryName

27

Results

Method/Performance

HM-T

HM-NT

BNM-R

BNM-NR

Acceptance Rate

0.376

0.576 0.552 0.408

Rejection Rate 0.607

0.000 0.000 0.000

Error rate 0.017

0.424 0.448 0.592

• 9270 mail piece images• 22 country destinations (> 30 train set samples)

– almost 3090 word images extracted

28



Elements


Examination

Query Documents

Comparison


Pattern Examination

Classification

Likelihood Ratio


Writer Identity


Partial Information(Foreign Mail)

Classifier Combination Classifier Selection

29

Classifier Combination – Decision Level

Input

C1

C2

…

CN

A7 A8 A3…AN

A8 A4

A7 A8 A3…AN

s7 s8 s3… sN

“Ensemble”Information

“Local” information

e1

e2

…eN

“Global” information

-heterogeneity-uncertainty

Global - [Xu et. al, 1998], [Zhang et. al 2003]Local: [Rogova et. al, 1997]Ensemble: BKS [Huang et. al, 1995]Feature and Classifier: FIM [Tahani et. al, 1990]

…

Dempster-Shafer-based unified framework for heterogeneous ensembles of classifiers

30

Classifier Combination

Motivation:– Current DST-based combination methods use only

global/local/ensemble information or combined local + global– Current solutions not always suitable for combining Type-III

classifiers (assume we have scores attached to each class)

Goals:– Adapt classic DST mass-computation methods for Type-III classifiers– Integrate “ensemble” information into the DS Theory of Evidence– Combine “local”, “global” and “ensemble” classifier information

into one unique framework– Estimate impact of affirming uncertainty regarding the top choice

(double hypotheses)

31

Classifier Combination-Adaptation to Type-III Classifiers Computation of Evidences

Classifier Level Class Level

Method 1 Method 2 Method 3

Use recognition/substitution rates for each top class

For each classifier ek the sum of masses for all classes add up to 1

Use recognition/substitution rates over all classes

Distance to a mean vector

Membership

function

Use uncertainty in the combination

32

Classifier Combination-Integrate BKS in DST

BKS

(Behavior

Knowledge

Space

- a unit of the K-dimensional knowledge space

- no. of training patterns for which T = Cm in

- no. of training patterns for the configuration

RKS

Problem: - as the number of classifiers increases, BKS becomes sparseSolution: - RKS (Reduced Knowledge Space)

- addresses sparseness - models joint behavior of recognizers, irrespective of the class

- set of groups of classifiers that agree on the top choice

- no of train patterns whose output configurations belong to L

- the classifiers set

- no of training patterns whose output configurations belong to L for which

- no of training patterns whose output configurations belong to L for which

33

Classifier Combination – Unified FrameworkDecision

Component Classifier C1

Global

information

Local

information

Component Classifier C2

Global

information

Local

information

Component Classifier CK

Global

information

Local

information

K classifiers:

Output:

“Global” performance:

“Local” performance:

Frame of Discernment:

)(, xeyRy kkM

k

…

Dempster-ShaferCombination Algorithm

Ensemble

information

Component ClassifiersC1 C2 … CK

&

34

Classifier Combination – Methods used• s1 - first classifier

• s2 - second classifier

• s3 - third classifier

• X – original method in [Xu et al 1998]• P – original method in [Parikh et al. 2001]

• R1 – original method in [Rogova el al. 1997], cosine measure

• R2 – original method in [Rogova el al. 1997], Euclidean distance based measure

• FIM – Fuzzy Integral Method [Tahani et. al 1990]• BKS – Behavior Knowledge Space [Huang et. al, 1995]

• M1 – “global” – recognition, substitution rates for each class

• M2 – “local” – sum of masses for all classes add up to 1

• M3 – “local” – use membership functions instead of distances to mean vectors

• RKS – Reduced Knowledge Space

• X+M2

• M2 +DH – M2+double hypotheses

• X+M2+DH – X+ M2 + double hypotheses

• M3+DH – M3+double hypotheses

• M2+BKS – M2+ ensemble BPA obtained from BKS

• M2+RKS – M2+ ensemble BPA obtained from RKS

• X+M2+BKS – X + M2 + ensemble BPA obtained from BKS

• X+M2+RKS – X + M2 + ensemble BPA obtained from RKS

35

Results

Local, Global,

Local + Global Information

75

80

85

90

95

100

s1 s2 s3

Recognizers

Re

co

gn

itio

n R

ate

96

96.5

97

97.5

98

98.5

X P R1 R2 M1 M2 M3 FIMX+M2

Methods

Re

co

gn

itio

n R

ate

e1+e2+e3

e1+e3

Original

Recognizer

Performance

36

Results

Method

e1+e2+e3 e1+e3

Recog. Rate(%)

Error Rate(%)

Reject Rate(%)

Recog. Rate(%)

Error Rate(%)

Reject Rate(%)

e1 94.47 4.73 0.08 94.47 4.73 0.08

e3 97.66 2.34 0.00 97.66 2.34 0.00

e2 83.96 16.04 0.00

BKS 97.86 1.49 0.65 97.81 2.16 0.03

RKS 97.21 2.79 0.00 97.66 2.34 0.00

Ensemble

Double Hypotheses

Method e1+e2+e3 e1+e3

Recog. Rate(%)

Error Rate(%)

Reject Rate(%)

Recog. Rate(%)

Error Rate(%)

Reject Rate(%)

e1 94.47 4.73 0.08 94.47 4.73 0.08

e3 97.66 2.34 0.00 97.66 2.34 0.00

e2 83.96 16.04 0.00

M2+DH 98.14 1.86 0.00 97.96 2.04 0.00

X+M2+DH

98.05 1.95 0.00 97.73 2.27 0.00

37

Results

Local + Global + Ensemble Information

97.697.797.897.9

9898.198.298.3

e1+e2+e3

e1+e3

38



Elements


Examination

Query Documents

Comparison


Pattern Examination

Classification

Likelihood Ratio


Writer Identity


Partial Information(Foreign Mail)

Classifier Combination Classifier Selection

39

Classifier Selection- When to use which Classifier?

Dynamic ClassifierSelection

Input A1,A2,…,AK

C1,C2,…,Cn

Class Set

Cj

[Woods et. al, 1997]

[Kanai et. al, 1997]…

[Xue et. al, 2002]…

Classifiers “Best” Classifier

Decision

Goal: Choose the classifier based on the class set (alphabet) size and composition

40

Classifier Selection

• How to measure the confusability of an alphabet?– recognizer confusion matrices [Kanai et al, 1994] , deformation

energy of elastic nets [Revow et al, 1996] , image-to-image distances [Huttenlocher et al, 2002], degradation model [Baird et. al 1993] –no “perfect” handwriting sample

• Drawbacks: classifier dependency and vulnerability to outliers• Approach: eliminate outliers and variances of shape by looking

at the character skeletons

IranIraqZair

IranOran≠Word Lexicons:

Alphabets o O, PD,t≠

size,“confusability”

ac

vwt

edit-distance

?

41


• Structural Features [Xue et. al, 2002]– Loop, Cusp, Arc, Circle, Cross, Gap - with different

attributes: Position, Orientation, Angle

Extract structural features from each character and build a profile HMM model for each character (of different sizes)

Match Match

Insert Insert

EndBegin

Delete

Upward Arc Upward CuspLoop

Downward Arc

42


HMM model for Digit ’2’ Emission Probabilities for State 3

43


Characters Confusability (Similarity) : distance between their corresponding HMM models– measure the probability that two models generate the same sequence

Match Match

Insert Insert

EndBegin

Delete

Match Match

Insert Insert

EndBegin

Delete

M1 M2

Co-emission probability

Alphabet confusability

c1 c2 s(c1c2)0 O 0.9862

71 I 0.9840

3U V 0.9103

3… … …

ac

vwt

Alphabet APairs of Characters Confusability ranking

+

44

Performance

Results obtained using the R1-based confusability measure

N CI R1

(%)

R2

(%)

R3

(%)

DCS

(%)

N CI R1

(%)

R2

(%)

R3

(%)

DCS

(%)

3 1 97.29 97.77 95.99 97.29 7 1 94.78 95.51 92.03 95.51

3 2 97.72 96.96 96.72 96.96 7 2 93.97 94.50 91.79 94.50

3 3 95.64 96.36 96.04 96.36 7 3 93.93 93.97 92.19 93.97

3 4 93.02 94.11 93.31 94.11 7 4 93.42 94.01 91.87 94.01

5 1 96.40 96.03 93.77 96.03 Average performance

5 2 95.59 95.55 93.89 95.55 95.00 95.38 93.68 95.33

5 3 94.82 94.78 93.85 94.78

5 4 94.42 94.94 92.72 94.94

94

95

96

97

98

99

100

1 2 3 4

R1

R2

R392

94

96

98

100

1 2 3 4

R1

R2

R3

Alphabet Size 5 Alphabet Size 7

Confusability Interval Confusability Interval

Train

Test

45

Performance

N CI R1

(%)

R2

(%)

R3

(%)

DCS

(%)

N CI R1

(%)

R2

(%)

R3

(%)

DCS

(%)

3 1 97.05 96.97 96.89 96.97 7 1 94.10 94.26 91.92 94.26

3 2 97.41 97.53 96.08 97.41 7 2 93.94 94.50 91.51 94.50

3 3 96.20 96.40 94.99 96.40 7 3 94.26 94.30 91.72 94.30

3 4 97.01 97.01 95.47 97.01 7 4 94.30 94.18 91.96 94.30

5 1 95.39 95.63 94.18 95.63 Average performance

5 2 94.83 95.43 93.29 95.43 95.43 95.57 93.77 95.61

5 3 95.15 95.51 93.45 95.51

5 4 95.51 95.15 93.81 95.51

Results obtained using the HMM-based confusability measure

94

95

96

97

98

99

1 2 3 4

R1

R2

R3

92

94

96

98

100

1 2 3 4

R1

R2

R3

Confusability IntervalConfusability Interval

Alphabet Size 5 Alphabet Size 7

Train

Test

46



Elements


Examination

Query Documents

Comparison


Pattern Examination

Classification

Likelihood Ratio


Writer Identity

Document ElementsWord and Document

Features

47

Character Discriminability

Di

ww,

Di

wx,

• Theory: Individuality is exhibited by writers in the execution of more complex forms

• Characters/Words differ in their discriminability power

For each character:

Distance Space

w1-w1

w2-w2

w3-w3

…wn-wn

w1-w2

w2-w3

...wi-wj

(μ,σ)

48

Character Discriminability

di

B

Di

ww,

Discriminability:– Bhattacharya distance– Area below Receiver Operating Curve (ROC)

w1-w1

w2-w2

w3-w3

…wn-wn

w1-w2

w2-w3

...wi-wj

49

Character Discriminabilityrank(c

)Character

c

1 G

2 B

3 N

4 I

5 K

6 J

5 W

6 D

7 h

8 F

9 r

10 H

11 B

…

62 1Discriminabilityranking of characters by their Bhattacharya distance/ROC area between the SW and DW distance distributions

Writer Verification

50

Word DiscriminabilityWord w d(w)

Queensberry

0.560

Allentown 0.356

Virginia 0.302

Grant 0.287

Parkway 0.282

Street 0.237

From 0.205

Elder 0.203

West 0.171

… …

10 0.002

Discriminability ranking of thefirst 25 words of the CEDAR letter

Queensberry Allentown

Virginia Grant Parkway

- length of word

Word Discriminability

Queensber r y

012345

WM R GSCW SC SCON

Al lentown

012345

WMR GSCW SC SCON

Virginia

01

23

45

WMR GSCW SC SCON

Grant

0

1

2

3

4

5

WMR GSCW SC SCON

51

Group-dependent Character Discriminability

Handwriting: Influenced by Age/Handedness/Gender?

In some cases partial information about the writer of the query document is available from other sources (e.g. gender, age group, etc)

what is the discriminability of characters written by writers of a certain group?

Males Females Bachelor High School LH RH Under 24

Above 45

C s C s C s C s C s C s C s C s

G 0.58 G 0.55 D 0.55 N 0.59 J 0.55 G 0.48 G 0.60

N 0.53

I 0.60 W 0.57 I 0.56 G 0.60 D 0.56 h 0.50 d 0.61

D 0.56

h 0.60 I 0.58 N 0.60 I 0.61 h 0.56 I 0.53 J 0.62

G 0.56

b 0.60 D 0.59 b 0.60 F 0.63 S 0.57 A 0.53 I 0.63

M 0.57

J 0.61 w 0.59 G 0.61 W 0.63 b 0.60 b 0.55 W 0.63

W 0.57

y 0.62 N 0.60 W 0.61 D 0.64 f 0.60 H 0.56 b 0.63

F 0.57

Group-dependent discriminability ranking of characters

52

Group-dependent Character Discriminability

G

01234567

N

01234567

I

0123456

F

0123456

W

0123456

b

0123456

J

0123456

53

Accumulated Writer Verification Performance

Accumulated Writer Verification

Performance for different groups

Base Case - letters are considered in alphabetical orderOther cases - letters are considered in decreasing order of discriminability

54



Elements


Examination

Query Documents

Comparison


Pattern Examination

Classification

Likelihood Ratio


Writer Identity

Document ElementsWord and Document

Features

55

Word and Document Features

• Theory: Use interpretable features • Current features:

– holistic view, without considering the relationships between document elements

– mostly content-dependent

• Proposed Features– Word Features– Document features:

• Lexeme (writing style)• Lexeme (writing style) context• Relative character proportions – height and slant• Handwriting legibility +inter-character distance

56

Curvature-based Word Feature

Original Image

ExteriorContour

Extraction

Curvature distribution

Curvature Computation

… …

… …‘o’

DTW

…3.45 …

Curvature: how much the curve "bends" at each point

- information preserving feature - rotation invariant.

/,

)(

22/322

22

rrrr

rrrr

57

Results – Word Features

Writer verification

58

Lexeme Distribution

Theory: writers use one or more forms for each character

Soft ClusteringHard Clustering

…

ratkowskyscottmarriottballtrcovwtracewfriedmanrubinssilikelihoodcalinskidbcindexhartigan

Clustering

Validity Indices

+

Lexemes

59

Lexeme Distribution

…

‘1’ ‘4’

1 2 3

4

‘0’

…01a 0

2a 11a 1

2a 41a 4

2a 43a 4

4a

Lexemes

;0)4(1 n ;0)4(2 n ;0)4(3 n ;3)4(4 n

Writer W2

1 2 3

4

Writer W1

… …01a 0

2a 11a 1

2a 41a 4

2a 43a4

4a

‘1’ ‘4’‘0’

)(

)()(

i

ilil cn

cnca

;0)4(1 n ;2)4(2 n ;1)4(3 n ;0)4(4 n ;3)4( n

22

21

221

21 ||),(||||),(||

||),(),(||),(

cDacDa

cDacDaDDd

cc

)4,,( 12 DWa

)4,,( 11 DWa

Lexemes

60

Lexeme Context Distributions

223322

3333

22

11

22

1133

44

…

11 22 33…… 6/106/10

dd

…

00 11 22 33

…

ZZ

… …

ee ll oo…nn

… …

2/102/10 …

11 22 33 44

44

Theory: neighboring allograph shapes influence a given allograph

)),(),((()),(),(((),( '2

)(

0

'12

)(

1121

'

'

' cDbcDbcDacDaDDd j

cNL

jj

ccl

cNL

ll

cc

aa

…

a

b

61

9.1 8.8 8.1 8.2 8.5 8.9 9.1 7.9

8.8 9.1 8.7 8.6 8.2 8.5 8.1 8.3

8.3 7.9 8.8 8.8

Avg. score: 8.6

Character/Word Legibility

…

Writer A

……

……

10.3 11.312.5 13.7

Writer B

8.9 7.7 7.9 8.5 8.6 5.7 7.8 7.9 8.1

7.6 7.9 7.0 7.1 8.0 6.8 7.9

7. 7 7.7 7.7

Average score : 7.2

…

…

……

9.5 8.210.1 12.5

Avg. Character LegibilitiesAvg. Character Legibilities

|)()(

|),(2

)(

1

1

)(

121

21

Dn

o

Dn

oDDd

i

Dn

j

ij

ij

i

Dn

j

ij

ij

ww

ii

62

Inter-Character Distance

Word W1

123

1ca

Word W2

123

1cb

13

12

)1(3

11 )()1(

1

)()1(

1)2,1(

N

c

ic

i

N

c

ic

i

bWareaN

aWareaN

WWd

63

Relative Slant and Height• Theory: Individuality is given mostly by relative not absolute measures like size

and slant

• ratios of letters are maintained despite changes in size, speed or intent of writing (normal or disguised)

a

d

ascenders: l,d,h

descenders: g y

n none: x a n

…

…

aa ad UU

hpp

1 2 9

p positive angle

n negative angle

ax-pp

xx-pn

hpn hnp hnn hpp hpn hnp hnn

Example:

hpp hpn hnp hnn

Compute RelativeSlant/Height

U Uppercase: A,B

NiDn

rh

i

ji

ij

i ,1;)(

,

64

Experimental Settings

• Data Set: 3000 documents written by 1000 Writers, representative of the US population

• Writer Verification:– Train/Test set: 1,500 same-writer document pairs, 1,500

different-writer document pairs.

• Writer Identification:– Test Set: 1000 documents (1000 writers)– Train Set: 2000 documents(1000 writers)

Same Content Different ContentCEDAR Letter

65

Results – Individual features

020406080100

Same Content

Different Content

05

101520

Same Content

Different Content

Writer

verification

Writer

identification

66

Results –Accumulated Features

Features Writer Verification Writer Identification

Same Content(%)

Different Content(%)

Same Content(%)


Original(12) 93.07 90.40 64.18 11.34

Proposed(6) 80.50 80.30 35.62 16.04

Features Writer Verification

Same Content(%)


Original+Proposed 93.77 90.60

Original+Characters 93.77 90.23

Proposed+Characters 91.47 84.23

Original+Proposed+Characters

93.90 94.30

67

Results – Writer Identification - Accumulated Features

CMC curves for original+proposed+character features



Elements


Examination

Query Documents

Comparison


Pattern Examination

Classification

Likelihood Ratio


Writer Identity

P: GNB

NP: KNN

Verification Identification

-

NP: KNN P:GNB, BNWriter independent

Writer dependent

Classification Models

• Theory: “The Bayesian approach is the best measure we have for assessing the value of evidence” [Aitken 1986] -> likelihood must be returned !

• Challenges:– variable number of features – model interpretation LR > 1 same writer

LR < 1 different writer

K

k

M

m

M

n

jkn

ikm

jkn

ikm

i j

DWssdpSWssdpLLR1 1 1

,,,, )))|),((ln()|),(((ln(

),(),|( ,,,,,, tkrtkrtrkt NCdp

),|()|()|( ,,1

,, trktt

R

rtrkt CdpCPCdp

k

Classification Models

Proposed approach–Verification:

– Non-Parametric: KNN - character/document features– Parametric:

–Gaussian Naïve Bayes – character features–Bayesian Networks – document features

–Identification – Non-Parametric: KNN - character features

Writer Verification Model - Train Process

di

B

Di

ww,

Di

wx,

...

Any Character

WriterWk

...

Writer-Independent

DWSW

...

...

Writer-Dependent

DWSW

TrainWriters

Set

Any Character

di

B

wk-wk

wk-w2

wk-w3

...wk-wj

Writer Verification Model-Test Process

...

‘0’

d(Wk ,Wx )

...

...

...….

‘Z’

N

iiSW cSWpP

1

)|(

‘SW’ ‘DW’

…

‘SW’ ‘DW’

…

……

‘0’ ‘1’ ‘Z’

N

iiDW cDWpP

1

)|(

LikelihoodRatio

WriterWk

Writer Verification-Document features

Id Feature Name Node Type

Id Feature Name Node Type

1 SW/DW discrete 7 Number of vertical slope components

continuous

2 Entropy continuous

8 Number of horizontal slope components

continuous

3 Gray-level threshold continuous

9 Number of negative slope components

continuous

4 Number of black pixels continuous

10

Number of positive slope components

continuous

5 Number of interior contours

continuous

11

Slant continuous

6 Number of exterior contours

continuous

12

Height continuous

13

Word gap continuous

Results-Writer Verification

Writer Independent Writer Dependent

GNB KNN GNB KNN

No. of Mixture

s

Accuracy

No. of Neighbor

s

Accuracy

No. of Mixture

s

Accuracy

No. of Neighbor

s

Accuracy

1 93.76 3 95.30 1 79.39 3 84.68

2 94.26 5 95.93 2 80.37 5 85.50

3 94.33 7 95.80 3 80.78 7 85.89

4 94.40

variable 94.30 variable 81.09

Character features

Writer independent

GNB BN KNN

93.96 93.93 90.60

Document features

Conclusion

– A framework for Writer Recognition in the form of a Computational Theory

– Adaptive handwriting retrieval system for Pictorial Similarity Examination

– Algorithms for extracting characters/words from documents for complete and partial information scenarios

– Framework for combining global, local and ensemble of classifiers information using the DS Theory of Evidence

– Classifier selection scheme based on an alphabet confusability measure based on distance between structure-based HMM models of characters

– Character and word ranking by their authorship discriminability– Parametric and Non-parametric models for writer recognition

Publications

Writer Recognition• Catalin I. Tomai, Devika Ksirhagar and Sargur N. Srihari "Group

Discriminatory power of handwritten characters" in Proceedings of SPIE, Document Recognition and Retrieval XI,San Jose, California, USA, Jan 18-22 2003

• Catalin I. Tomai , Bin Zhang and Sargur N. Srihari, "Discriminatory power of handwritten words for Writer Recognition“ in Proceedings of ICPR'04

• Sargur N. Srihari , Catalin I. Tomai, Bin Zhang and Sanjik Lee, Individuality of Numerals in Proceedings of ICDAR'03, Edinburgh, Scotland, August 2002

• Sargur N. Srihari, Bin Zhang Catalin I. Tomai, Sanjik Lee, Zhixin Shi and Yong-Chul Shin "A System for Handwriting Matching and Recognition" in Proceedings of the 2003 Symposium on Document Image Understanding Technology (SDIUT'03),Greenbelt, Maryland, April 2003

• Sargur N. Srihari, S. Lee, Bin Zhang and Catalin I. Tomai, “Recognition-based System for Handwriting Verification and Identification” , in Proceedings of the Intl. Graphonomics Conference (IGS'03) Scottsdale, Arizona, 2-5 November 2003

• Sargur N. Srihari, Catalin I. Tomai and Sanjik Lee, "Quantitative Assesment of Handwriting Individuality" in Proceedings of the Fifth International Conference On Advances In Pattern Recognition (ICAPR-2003),Calcutta, India, December 10-13, 2003 invited paper

• Sargur N. Srihari, Anantharaman Ganesh, Catalin I. Tomai and Yong-Chul-Shin "Information Retrieval System for Handwritten Documents" in Proceedings of DAS'04

Publications

Handwriting recognition• Catalin I. Tomai, Kristin Allen and Sargur N. Srihari, Foreign Mail

Recognition, in Proceedings of ICDAR’01, Seattle, WA, September 2001

Classifier Combination• Catalin I. Tomai and Sargur N. Srihari, Combination of Type-III classifiers

using DS Theory of Evidence, Proceedings of ICDAR'03, Edinburgh, Scotland, August 2002

Digital Libraries• Catalin I. Tomai, Bin Zhang and Venu Govindaraju, Transcript Mapping for

Historical Documents, in Proceedings of IWFHR’02, Niagara On The Lake, 2002

• Bin Zhang, Catalin I. Tomai, Venu Govindaraju and Sargur N. Srihari, Construction of Handwriting Databases Using Transcript-based Mapping" in Proceedings of the Intl. Workshop on Document Image Analysis for Libraries 2004 (DIAL'04),January 23-24, 2004, PARC, Palo Alto, CA, USA

Image Retrieval• Bin Zhang, Catalin I. Tomai and Aidong Zhang, Adaptive Texture Image

Retrieval in Transform Domain , in Proceedings of ICME’02, Lausanne, Switzerland, September 2002

• Bin Zhang, Catalin I. Tomai and Aidong Zhang, An Adaptive Texture-Information Retrieval System using Wavelets, in Proceedings of the Seventh International Conference on Control, Automation, Robotics and Vision (ICARCV 2002),Singapore, December 2002

Thank You

Complete Information-Transcript Mapping

From (0.73)

10 (0.00)

Nov (0.36)

1999 (0.77). . .

Nov (0.88)

Jim (0.37)

1999 (0.77)

From (0.73)

From Nov 10 1999 Jim Elder …

Word Recognition

Build Word-Hypothesis Set

Next Line/Refine Results

Dynamic Programming

10 (0.26)

Nov (0.42)

From

Nov

1999

WordRecognition

Results

Query Image Image 1 Image 2 Image3

0005b-0

005a-2 0339b-2

0021a-0

0013a-0

0013a-1

0313c-2

0015b-0

0059b-0

0145c-3

0237c-3

0145c-1


• Inspired by the Computational Theory of Computer Vision [Marr,1980]

• Three Levels:– Theory - developed based on studies on how human

experts discriminate between handwritings of different people

• Analysis, Comparison, Evaluation (Law of the ACE’s)– Algorithms and Representation

• pattern recognition/machine learning/computer vision• Document elements: character/words• Discriminating elements: elements of handwriting that vary

measurably with its author– Hardware/Software

Discussion

– Proposed a DST-based framework for combining global, local and ensemble of classifiers information which outperforms the FIM method

– Adapted classic BPA-computation methods to combine Type-III recognizers

– Integrated “ensemble of classifiers” information into the DST Theory of evidence

– Investigated the use of double hypotheses when combining recognizers