On Evaluating Open Biometric Identification Systems

Master’s Dissertation

by

Michael Gibbons

School of Computer Science and Information Systems

Pace University

May 2005

2

We hereby certify that this dissertation, submitted by Michael Gibbons, satisfies the dissertation requirements for the Master’s Degree in Computer Science and has been approved.

_____________________________________________-________________

Sung-Hyuk Cha Date Chairperson of Dissertation Committee

_____________________________________________-________________

Charles Tappert Date Dissertation Committee Member

_____________________________________________-________________

Sungsoo Yoon Date Dissertation Committee Member

School of Computer Science and Information Systems Pace University 2005

3

Abstract

This dissertation concerns the generalizability of biometric identification results from small-sized closed systems to larger open systems. Many researchers have claimed high identification accuracies on closed system consisting of a few hundred or thousand members. Here, we consider what happens to these closed identification systems as they are opened to non-members. We claim that these systems do not generalize well as the non-member population increases. To support this claim, we first take a look at the visualization of pattern classification using Support Vector Machines (SVM), Nearest Neighbor (NN) and Artificial Neural Network (ANN). Next, we present experimental results on writer and iris biometric databases using the afore mentioned classifiers. We find that system security (1-FAR) decreases rapidly for closed systems when they are tested in open-system mode as the number of non members tested increases. We also find that, although systems can be trained for greater closed-system security using SVM rather than NN classifiers, the NN classifiers are better for generalizing to open systems due to their superior capability of rejecting non-members.

4

Acknowledgments

I would like to express my sincere appreciation to Professors Sung-Hyuk Cha,

Charles Tappert and Sungsoo Yoon for their guidance and support during this dissertation

study.

5

Table of Contents

Abstract.................................................................................................................. 3

Acknowledgments ................................................................................................. 4

Chapter 1 Introduction...................................................................................... 10

Motivation............................................................................................................. 10

Approach............................................................................................................... 11

Chapter 2 Error Rate Types and Classifiers in Biometric Identification...... 12

Error Rate Types ................................................................................................... 12

Classifiers.............................................................................................................. 15

Visualization in Pattern Classification.................................................................. 18

Using Separated Training Data............................................................................. 18

Nearest Neighbor (Separated Training Data)................................................ 19

ANN 1-vs-all (Separated Training Data) ...................................................... 21

SVM 1-vs-all (Separated Training Data) ...................................................... 24

Identification by Verification (Separated Training Data) ............................. 26

Analysis on Classifications on Separated Data............................................. 28

Using Overlapping Training Data......................................................................... 29

6

Nearest Neighbor (Overlapping Training Data) ........................................... 30

ANN 1-vs-all (Overlapping Training Data).................................................. 31

SVM 1-vs-all (Overlapping Training Data).................................................. 32

Identification by Verification (Overlapping Training Data)......................... 32

Analysis on Visualization of Classification on Overlapping Data ............... 33

Chapter 3 Experiments....................................................................................... 34

Biometric Databases ............................................................................................. 34

Experiment Setup.................................................................................................. 35

Results and Analysis ............................................................................................. 37

Security Convergence ........................................................................................... 42

Chapter 4 Conclusions........................................................................................ 45

References............................................................................................................ 47

7

List of Figures

Figure 1. (a) The graph displays classification boundaries for a hypothetical two-member

database using the Nearest Neighbor classifier with threshold t. False accepts and

false rejects can occur between members of the system, and false accepts can also

occur as non-members enter the system. (b) Same as (a) but with a three-member

database and SVM classifier..................................................................................... 15

Figure 2. Three separated classes with points randomly generated. The yellow clusters

of asterisk (*) points represent non-members........................................................... 19

Figure 3. Nearest Neighbor visualization with 0 threshold on the separated data........... 20

Figure 4. Nearest Neighbor visualization with threshold of 8 on the separated data ...... 20

Figure 5. Nearest Neighbor visualization with a threshold of 2 on the separated data. This

is the best threshold choice based on the training data for NN................................. 21

Figure 6. (a) Class 1 vs. all dichotomy (b) Class 2 vs. all dichotomy (c) Class 3 vs. all

dichotomy (d) The 1-vs-all ANN implementation trained on separated data........... 23

Figure 7. A 1-vs-all SVM with gamma = 2, C=100 trained on separated data ............... 25

Figure 8. A 1-vs-all SVM with gamma = 2-3 and C = 50 trained on separated data ....... 25

8

Figure 9. The normal distribution to determine threshold to be used in the identification

by verification approach ........................................................................................... 26

Figure 10. An Identification by Verification approach with threshold measure of 19

trained on separated data........................................................................................... 27

Figure 11. An Identification by Verification approach with threshold measure of 10

trained on separated data. Notice the rejection of non-members improved as

threshold decreased (compared to figure 10)............................................................ 28

Figure 12. Three overlapping classes with points randomly generated. The yellow

clusters of asterisk (*) points represent non-members.............................................. 29

Figure 13. Nearest Neighbor visualization with a threshold of 8 on overlapping data.

Compare this to the separated case where the best option was a threshold of 2....... 30

Figure 14. The 1-vs-all ANN implementation trained on the overlapping data .............. 31

Figure 15. A 1-vs-all SVM with gamma=2-3, C=100 trained on overlapping data ......... 32

Figure 16. Identification by Verification with a threshold of 14 on overlapping data .... 33

Figure 17. The ROC curve for the Nearest Neighbor classification of 100 members. The

operating threshold was chosen close to equal error rate, but favoring FAR or

security...................................................................................................................... 37

9

Figure 18. Security results for writer data using SVM as non-members are introduced to

the system.................................................................................................................. 38

Figure 19. Security results for iris data using SVM as non-members are introduced to the

system ....................................................................................................................... 39

Figure 20. Security results for writer data using Nearest Neighbor as non-members are

introduced to the system ........................................................................................... 40

Figure 21. A comparison of the performance of the Nearest Neighbor and SVM

classifiers on the writer data of 100 members as non-members enter the system .... 41

Figure 22. A comparison of the performance of the Nearest Neighbor, SVM and ANN

classifiers on the iris data of 15 members................................................................. 42

Figure 23. Curve fitting for writer data of 50 and 200 members ..................................... 43

10

Chapter 1

Introduction

Biometric applications are becoming commonplace in today’s society.

Technology continues to improve, providing faster processors, smaller sensors and

cheaper materials, all of which are contributing to reliable, affordable biometric

applications. The most common use of biometrics is for verification. In biometric

verification systems, a user is identified by an ID or smart card and is verified by their

biometric, i.e., a person’s biological or behavior characteristic such as their finger-print,

voice, iris, or signature. This is analogous to a user at an ATM machine using a bank

card to identify and a PIN to verify. Another use of biometrics is for identification,

which is the focus of this study. Identification can be applied in a closed system such as

employee positive identification for building access, or in an open system such as a

national ID system. Positive biometric identification, a 1-to-many problem, is more

challenging than verification, a 1-to-1 problem. As stated in [1], “positive identification

is perhaps the most ambitious use of biometrics technology.”

Motivation

There have been many promising results reported for closed identification

systems. Although high accuracies have been reported in writer, iris, and hand geometry

studies [4, 9, 11, 15, 19], these accuracies may lead to a false impression of security. One

11

may ask if there really are any situations that correspond to closed worlds[1]. For

example, take an employee identification system. Can it be guaranteed that a biometric

of a guest (a non-member) visiting the facility does not match one of an employee (a

member)? This paper will investigate the generalizability of biometric identification as it

pertains to the security of a system. Our hypothesis is that the accuracies reported for

closed systems are relevant only to those systems and do not generalize well to larger,

open systems containing non-members.

Approach

Since it is impractical to test a true population, we use a reverse approach to sup-

port the hypothesis. We will work with a database M of m members, but assume a closed

system of members, where'm 'm m< , and train the system on the members. We

then have members to test how well the system holds up when non-members

attempt to enter the system. This approach is used on two biometric databases, one

consisting of writer data and the other of iris data.

'm

'mm −

In chapter 2 of this study, positive identification and the associated error rates will

be explained. Also, the classifiers used in the study are presented, along with a

visualization of pattern classification. In chapter 3, the biometric databases are

explained, followed by the experiments to support the hypothesis. Chapter 4 concludes

with a summary and considerations for future work.

12

Chapter 2

Error Rate Types and Classifiers in Biometric Identification

This chapter will discuss error rate types that occur for positive identification in

both the closed and open environments, followed by an overview of classifiers used in

this study.

Error Rate Types

Consider the positive identification model. Positive identification refers to

determining that a given individual is in a member database [1]. This is an m-class

classification problem – that is, given data from m subjects in a biometric database

1 2{ , , , }mM s s s= L

q

i

, the problem is to identify an unknown biometric sample d from a

person, s , where . In this model, a classifier can be trained based on all

exemplars in M to find decision boundaries, e.g., support vector machine. If a similarity-

based classifier such as a nearest neighbor is used, an unknown sample d is compared to

each d of M. The error rate in this model is simply a number of misclassified instances

divided by the testing set size.

qs M∈

Consider an unknown biometric sample d from a person, s , where . If

this biometric data of a non member enters directly to the above model, can it be

classified correctly? If the classifier has no reject capability, the unknown will be

q qs M∉

13

classified into one of the decision regions or as the closest matching subject. However, if

the classifier has a reject capability, the number of classes in M becomes m , i.e., m

member classes + 1 reject class. Therefore, if the questioned instance is in a reject area in

SVM or the closest match is outside the nearest neighbor thresholds, the unknown will be

classified as none of the members. The study investigates the reject capability of three

classifiers: Support Vector Machines, Nearest Neighbor and Artificial Neural Networks.

We claim that a classifier with the lowest error rate is not necessarily the best for

security, and that classifier designers might consider the following three types of error

rates.

1+

s ∈

In the latter scenario with members and non-members, there are three kinds of

error. A ‘false reject’, FR, error occurs when a classifier identifies an unknown biometric

sample d from a person, , where qs qs M∈ , as a reject. The other errors are ‘false

accepts’, FA, of which there are two types – those that can occur between members of the

system, FA (1), and those that can occur as non-members enter the system, FA (2).

FA(1) occurs when a classifier identifies an unknown biometric sample d from a person,

to where and qs is ,q is s M∈ qs si≠ . FA (2) occurs when a classifier identifies an

unknown biometric sample d from a person, to , where qs is and q is M M∉ . The

error types are illustrated in figure 1.

The frequencies at which the false accepts and false rejects occur are known as

the False Accept Rate (FAR) and the False Reject Rate (FRR), respectively. These two

14

error rates are used to determine the two key performance measurements of a biometric

system: convenience and security [1]:

FARSecurityFRReConvenienc

−=−=

11

(1)

In this paper, we will pay close attention to the Security measurement as we test our

hypothesis.

15

Figure 1. (a) The graph displays classification boundaries for a hypothetical two-member

database using the Nearest Neighbor classifier with threshold t. False accepts and false rejects

can occur between members of the system, and false accepts can also occur as non-members

enter the system. (b) Same as (a) but with a three-member database and SVM classifier

Classifiers

Although there are many classifiers to choose from in the field of pattern

classification, we used three pattern classification techniques: Support Vector Machines

(SVM), Nearest Neighbor (NN) and Artificial Neural Networks (ANN).

In recent years, the SVM classifier has emerged as a popular classifier. The

objective of the SVM classifier is to separate data with a maximal margin, which tends to

16

result in a better generalization of the data. Generalization helps with the common

classification problem of over-fitting.

The points that lie on the planes that separate the data are the support vectors.

Finding the support vectors requires solving the following optimization problem (details

of this method can be found in [2, 12]):

0,1))(( :subject to

min1

21

,,

>−≥+

∑+=

ξξξ

φξ

iiiT

i

i

l

i

T

bw

bxwy

Cww (2)

The geometric representation of the SVM can be easily visualized when the data

falls into the linear separable case. When the data falls into linear non-separable case and

non-linear separable cases, the complexity increases. For more information on the

different cases, please refer to [10] which devotes a chapter on SVMs.

Real world data tends to fall into the non-linear separable case. To solve the non-

linear separable problem, the SVM relies on pre-processing the data to represent patterns

in a higher dimension than the original data set. The functions that provide the mapping

to higher dimensions are known as phi functions or kernels. Common kernels include

Radial Basis Function (RBF), linear, polynomial, and sigmoid. The RBF kernel is used

in this study and additional information on this kernel follows in chapter 3.

17

The next classifier we consider is the Nearest Neighbor classifier, NN, which is

much simpler mathematically than SVM and other classifiers. To classify a test subject,

the NN computes distances from a test subject d to each member di of the database, and

classifies the test subject as the subject that has the closest distance. The distances can be

computed using various methods such city-block distance or Euclidean distance.

A reject threshold can be introduced into the Nearest Neighbor classification. If

the distance between test subject d and its’ nearest neighbor d is within the threshold,

the classification is that of the closest member. However, if the distance is greater than

the threshold, the subject is rejected and classified as a non-member. In this study, we

use a reject threshold, which is necessary for an open world environment.

i

The third classifier that is used in this study is the Artificial Neural Network, or

ANN. We chose to implement a 1-vs-all implementation of the ANN to provide reject

capability.

The 1-vs-all approach becomes a series of dichotomy problems, i.e., class 1 vs.

class 2 and class 3, class 2 vs. class 1 and class 3, etc. Any points that fall into more than

one of the dichotomy decision regions will be rejected.

The next section will examine these classifiers on a two-dimensional dataset to

allow analysis of classifier behavior visually.

18

Visualization in Pattern Classification

In this section we explore pattern classification in two-dimensions in order to

produce visualizations to help understand the decision boundaries of the classifiers

mentioned in the previous section: SVM, Nearest Neighbor and Artificial Neural

Network. In addition, we include an identification-by-verification approach.

Two sample datasets were created. Both contain three classes, however one

dataset is purely separated, the other overlapping. The overlapping dataset is presented to

closer resemble real-world data.

Using Separated Training Data

To produce the separated data, we choose three center points and randomly

generate 200 points within radius r of the center points. The separated datasets are

presented in figure 2. Non-member data is also included in the figure. The non-member

data is also separated from the member classes.

The test points will consist of all points (i, j) where i = {1:100} and j = {1:100}.

In the next sections, we will take a look at how the mentioned classifiers classify the test

points based on the separated training data sets.

19

Figure 2. Three separated classes with points randomly generated. The yellow clusters of

asterisk (*) points represent non-members

Nearest Neighbor (Separated Training Data)

Although a very simple classifier, the Nearest Neighbor is still very powerful.

Figure 3 shows the Nearest Neighbor without using a threshold. For an identification

system where we a considering an open environment, a 0 threshold is not the right

choice, as any non-member will automatically be falsely accepted. Figures 4 and 5

follow with a threshold of 8 and 2 respectively. For the larger threshold, many non-

members (outlined in black circle) are getting improperly classified. The NN using a

threshold of 2 resembles the 3 classes of figure 2 the best.

20

Figure 3. Nearest Neighbor visualization with 0 threshold on the separated data

Figure 4. Nearest Neighbor visualization with threshold of 8 on the separated data

21

Figure 5. Nearest Neighbor visualization with a threshold of 2 on the separated data. This is the

best threshold choice based on the training data for NN

ANN 1-vs-all (Separated Training Data)

Next, we investigate a 1-vs-all implementation of the Artificial Neural Network.

The reason we implement 1-vs-all is the same reason we choose a Nearest Neighbor

implementation with a threshold: reject capability.

The 1-vs-all approach becomes a series of dichotomy problems, i.e., class 1 vs.

class 2 and class 3, class 2 vs. class 1 and class 3, etc. Figures 6 (a), (b) and (c) show the

dichotomy classifications. Figure 6 (d) is the three dichotomies merged together, where

any overlap between dichotomies are rejected.

22

23

Figure 6. (a) Class 1 vs. all dichotomy (b) Class 2 vs. all dichotomy (c) Class 3 vs. all dichotomy

(d) The 1-vs-all ANN implementation trained on separated data

24

The non-member points fall nicely outside the reject boundaries as illustrated in

the figure. But what would happen if there were additional non-member clusters? For

example, if there were an additional cluster of points centered around the point (15, 15).

All of these points would fall directly into class 1’s classification boundaries. So,

although the ANN visually looks good, it could be a little risky in the open environment.

SVM 1-vs-all (Separated Training Data)

Under proper training, the Support Vector Machine should be able to provide a

high-quality generalization that other classifiers cannot match. However, if the SVM is

not properly trained, the classifier can actually lead to over-fitting, as shown in figure 7.

The non-member data is properly rejected, but so are many points within the clusters.

The decision boundaries are too close to training data.

The gamma parameter of the SVM can be adjusted to shrink or grow the

neighborhood around the member class points. More information will follow in chapter 3

regarding SVM parameter tuning. Figure 8 shows another SVM implementation where

the clusters are more generalized and the non-member points are still rejected, producing

an improved generalization of the classes of figure 2.

25

Figure 7. A 1-vs-all SVM with gamma = 2, C=100 trained on separated data

Figure 8. A 1-vs-all SVM with gamma = 2-3 and C = 50 trained on separated data

26

Identification by Verification (Separated Training Data)

Last we present a classifier that uses verification to identify. In this approach, we

first find a threshold for each member to determine what threshold best separates within-

distance, i.e., between members of the same class, and between-distance, i.e., between

members of different classes.

The normal distributions of the within-distance and between-distance values are

used to determine a threshold. As displayed in figure 9, we choose a threshold that

minimizes FAR and FRR, aiming for close to equal-error rate.

Figure 9. The normal distribution to determine threshold to be used in the identification by

verification approach

27

Test subjects are then compared to each member instance and votes are tabulated

on whether the test subject falls within the threshold for the member class. Figure 10

below illustrates this approach. The approach has sub-par performance. If the threshold

is chosen to be much smaller as in figure 11, the approach will have better performance.

More analysis will have to be done on threshold selection before using this classifier in

real world situations.

Figure 10. An Identification by Verification approach with threshold measure of 19 trained on

separated data

28

Figure 11. An Identification by Verification approach with threshold measure of 10 trained on

separated data. Notice the rejection of non-members improved as threshold decreased (compared

to figure 10)

Analysis on Classifications on Separated Data

Based on the member data presented in figure 2, we find the SVM and Nearest

Neighbor approaches to provide the best performance in regards to rejecting non-

members. Also, it was seen that an SVM can be trained to make a proper generalization

around the cluster points, equaling the Nearest Neighbor. As we move on to the next

section containing overlapping data, we will see the differences between the SVM and

the Nearest Neighbor.

29

Using Overlapping Training Data

To produce the overlapping data, we choose three center points and randomly

generate 200 points within radius r of the center points. However, the radius has been

chosen to be large enough to create an overlap between classes. The overlapping datasets

are presented in figure 12. Non-member data is also included in the figure. The non-

member data also overlaps with the member classes.

Note, because the number of points has not changed, 200, and the radius has been

increased, the clusters of data points contain large empty areas within the boundaries.

Figure 12. Three overlapping classes with points randomly generated. The yellow clusters of

asterisk (*) points represent non-members

30

As in the separated case, the test points will consist of all points (i, j) where i =

{1:100} and j = {1:100}. In the next sections, we will take a look at how the mentioned

classifiers classify the test points based on the overlapping training data sets.

Nearest Neighbor (Overlapping Training Data)

As mentioned above, the overlapping data sets contained large holes within the

clusters. Because of that, a larger threshold had to be chosen to fill in the areas. The

downside of that, as illustrated in figure 13 is the very large proportion of non-member

data points getting classified as one of the three classes, as indicated by the circle

outlines.

Figure 13. Nearest Neighbor visualization with a threshold of 8 on overlapping data. Compare

this to the separated case where the best option was a threshold of 2

31

ANN 1-vs-all (Overlapping Training Data)

Next, we investigate a 1-vs-all implementation of the Artificial Neural Network

for the overlapping data. Figure 14 shows the classification. Notices, the boundary lines

are no longer clearly defined and straight as was in figure 6 on the separated data.

Figure 14. The 1-vs-all ANN implementation trained on the overlapping data

32

SVM 1-vs-all (Overlapping Training Data)

The SVM results on the overlapping data are illustrated in figure 15. The outer

boundaries are not as rounded as the Nearest Neighbor, but the number of valid rejections

is greater, which makes this classifier the best so far on the overlapping data.

Figure 15. A 1-vs-all SVM with gamma=2-3, C=100 trained on overlapping data

Identification by Verification (Overlapping Training Data)

Last, we present the method which uses verification to identify. This approach

resembles the Nearest Neighbor, but instead of making branches at the intersecting

sections of the clusters, straight lines mark the edges. Clearly, the false accepts among

members, what was labeled FA (1) earlier in this chapter 2 will be very large.

33

Figure 16. Identification by Verification with a threshold of 14 on overlapping data

Analysis on Visualization of Classification on Overlapping Data

Based on the overlapping member data presented in figure 12, we find the SVM

to be the best candidate for classification in the overlapping data case. This is important

as most real world data will not be as clearly defined as the separated data in previous

section, i.e., figure 2.

34

Chapter 3

Experiments

Our hypothesis is that biometric identification on closed systems does not

generalize well to larger, open systems containing non-members. In the previous section,

we saw a visualization of the various classifiers against a simple 2-dimensional dataset.

We now investigate the hypothesis further by conducting experiments on subset database

'M M⊂ from both the writer and iris databases. We describe those databases first.

Biometric Databases

In a previous study, Cha et al. [3] studied the individuality of handwriting using a

database of handwriting samples from 841 subjects representative of the United States

population. Each subject copied three times a source document containing 156 words

carefully constructed to have each letter of the alphabet used in the starting (both upper

and lowercase), middle (lowercase), and ending position (lowercase) of a word. Each

document was digitized and features were extracted at the document, word, and character

level. For the purposes of this study, we used the same database but focus only on the

document features: entropy, threshold, number of black pixels, number of exterior

contours, number of interior contours, slant, height, horizontal slope, vertical slope,

negative slope, and positive slope. A detailed explanation of these features can be found

in [4].

35

From the iris biometric image database [8], we selected 10 left bare eye samples

of 52 subjects. In comparison to the writer database, the iris database has many fewer

subjects, but a much larger number of samples per subject. This will allow for more

samples to be trained.

After the images are acquired, they are segmented to provide a normalized

rectangular sample of the iris. Features are extracted using 2-D multi-level wavelet

transforms. For this experiment, 3 levels are used producing a total of 12 parts. The 12

parts produce 12 feature vectors consisting of the coefficients from the wavelet trans-

form. The mean and variance of each vector are obtained to produce a total of 24

features for each sample. See [18] for more information on the 2-D wavelet transforms

used.

Experiment Setup

For each of the databases, training sets were created. Training sets for the writer

data consisted of m 50, 100, 200 and 400 members. Training sets for the iris data

consisted of m 5, 15, 25 and 35 members. These sets included all instances per

member, i.e., 3 per member for writer and 10 per member for iris.

' =

' =

For the first part of the experiment, an SVM was trained on the members.

Parameter tuning, or SVM optimization, was performed prior to training. The first

parameter tuned is the penalty parameter C from equation (2), and depending on the

36

kernel used, there are additional parameters to tune. For this experiment we used an RBF

kernel of the form:

0,||||),(2

>−−= γγe jiji xxxxK (3)

The γ parameter in equation (3) is the only kernel parameter requiring tuning. A grid-

search method as defined in [7] was used to optimize these two parameters.

Tuning the parameters gives 100% accuracy on each of the training sets.

Therefore, we have 0% FAR and FRR, or equivalently, 100% security and convenience.

The next step is to test non-members to determine the true security of the trained SVM.

For each training set we created a combined evaluation set consisting of the trained

members plus an increasing number of non-members. The evaluation sets for the 50-

writer trained SVM consisted of 50, 100, 200, 400, 700 and 841 subjects, where the first

50 subjects are the members and the remaining subjects are non-members. Similarly, the

evaluation sets for the 25-iris trained SVM consisted of 25, 35, 45 and 52 subjects, where

the first 25 subjects are the members and remaining subjects are non-members.

In the second part of the experiment, the Nearest Neighbor classifier was used.

For this classifier, threshold tuning was required. The threshold has to be large enough to

allow identification for the known members, but small enough not to allow non-members

to be classified as members. As the threshold increases, the FAR increases and FRR

decreases.

37

The Receiver Operating Characteristic (ROC) curve for the Nearest Neighbor

classifier is presented in figure 17. The ROC curve is a plot of FAR against FRR for

various thresholds. When designing an identification system, there is a trade off between

the convenience (FRR) and security (FAR) of the system. For this experiment, we have

chosen an operating threshold that is close to equal error rate, but leaning towards a

higher security system.

Figure 17. The ROC curve for the Nearest Neighbor classification of 100 members. The

operating threshold was chosen close to equal error rate, but favoring FAR or security

Results and Analysis

In the positive identification model we consider two errors: false accepts and false

rejects. Since the SVM was able to train the members to 100% accuracy, we eliminate

the false accepts and false rejects for members. The remaining tests are non-members

38

and therefore can only produce false accepts. The false accepts correlate to the security

measurement of the system, a measure of extreme importance to the system. In figure 18,

the security results are shown for the writer data.

Figure 18. Security results for writer data using SVM as non-members are introduced to the

system

39

Figure 19. Security results for iris data using SVM as non-members are introduced to the system

As hypothesized, for each curve, as the number of non-members increases, the

security monotonically decreases (or equivalently, the FAR monotonically increases). It

might also be noted that the final rates to which the security curves decrease appear to

converge – that is, to approach asymptotes. To ensure that this is not an artifact of the

particular handwriting data used, we obtained similar experiment results on the iris data

as presented in figure 19. The iris data in figure 19 follows the same pattern as the writer

data in figure 18, although convergence is not as evident for these data.

Next, we present the results for the Nearest Neighbor classifier. As can be seen in

figure 20, the same pattern emerges, although in this experiment we did not obtain 0%

FAR for the members. When using the Nearest Neighbor approach, a one-versus-all

method was used to obtain the accuracy for the closed environment.

40

Figure 20. Security results for writer data using Nearest Neighbor as non-members are

introduced to the system

We now present a comparison of the results from the two classifiers used in this

experiment. Figure 21 illustrates the security performance for 100 members of the writer

database. Notice, although Nearest Neighbor does not perform as well on the closed

environment, it eventually meets and surpasses the performance of the SVM as non-

members enter the system.

41

Figure 21. A comparison of the performance of the Nearest Neighbor and SVM classifiers on the

writer data of 100 members as non-members enter the system

The finding that NN outperforms SVM in the open environment for writer data is

a key observation. Last, we go one step further bringing in an additional classifier, the

Artificial Neural Network or ANN. Figure 22 illustrates the security performance for 15

members of the iris database. We notice the Nearest Neighbor again does not perform

well on the closed environment, but does out perform the SVM performance as non-

members enter the system. However, we introduced a 1-versus-all approach ANN. This

approach, like SVM, provides excellent performance on the closed environment, and

actually surpasses both the SVM and NN on the open environment. This is an interesting

42

find, however, additional work still needs to performed to test against various sized iris

and writer data.

Figure 22. A comparison of the performance of the Nearest Neighbor, SVM and ANN classifiers

on the iris data of 15 members

Security Convergence

Based on the security results of figures 18, 19 and 20, we recognize that the

curves appear to be of exponential form and that we might be able to extrapolate the

43

security of a system for large populations containing non-members. After some fitting

trials, we find the curve most similar to be:

daey cbx += −− )/)(( 21

(4)

where the constant b is the number of members; the constants a, c, and d vary based on b;

and the security converges to d. Figure 23 displays the curve fitting for the 50 and 200

trained writer data samples. Once the constants are found, equation (4) can then be used

to project results for even larger populations.

Figure 23. Curve fitting for writer data of 50 and 200 members

44

Other attempts have been made to model biometric identification in large

environments. In [6], the researchers attempt to model large systems using a binomial

modeling approach.

45

Chapter 4

Conclusions

In this study, we found that system security (1-FAR) decreases rapidly for closed

systems when they are tested in open-system mode as the number of non members tested

increases. Thus, the high accuracy rates often obtained for closed biometric identification

problems do not appear to generalize well to the open system problem. This is important

because we believe that no system can be guaranteed to remain closed. This hypothesis

was validated by experiments on both writer and iris biometric databases. An estimate of

the expected error was also projected based on the asymptote of an exponential curve

fitted to the data. Furthermore, because the FAR in these studies was obtained from

normal biometrics of other subjects, the so-called "zero-effort" rate, the true FAR of these

systems would be greater, resulting in even poorer generalization.

We also found that, although systems can be trained for greater closed-system

security using SVM rather than NN classifiers, the NN systems are better for generalizing

to open systems through their capability of rejecting non members. Thus, it appears that

the reject thresholds of NN classifiers do a better job of rejecting non members than the

reject regions of SVM classifiers.

In summary, we demonstrated that the generalization capability of closed

biometric systems in open environments is poor, and that the significantly larger error

46

rates should be taken into account when designing biometric systems for positive

identification. For increased security, for example, multi-modal biometrics might be

considered [14].

When designing an identification system, there is a trade off between the

convenience and security of the system. Most systems would choose security over

convenience. However, in our implementation of SVM for the writer data, we imply

choosing convenience over security (guarantee 0 false rejects). In our study on writer

data, there just are not enough samples per member to put into the testing set. It would

therefore be beneficial to run further experiments against larger biometric databases.

47

References

1. Bolle, R.M., Connell, J.H., Pankanti, S., Ratha, N.K., Senior, A.W.: Guide to Biometrics. Springer (2004)

2. Burges, C.: A Tutorial on Support Vector Machines for Pattern Recognition. Data Mining and Knowledge Discovery. 2:121-167 (1998)

3. Cha, S.-H., Srihari, S.N.: Writer Identification: Statistical Analysis and Dichotomizer. Proceedings International Workshop on Structural and Syntactical Pattern Recognition (SSPR 2000), Alicante, Spain. pp. 123- 132 (2000)

4. Cha, S.-H., Srihari, S.N.: Handwritten Document Image Database Construction and Retrieval System. Proceedings of SPIE Document Recognition and Retrieval VIII, Vol. 4307 San Jose (2001)

5. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification. John Wiley & Sons (2001) 6. Grother, P., Phillips, P.J.: Models of Large Population Recognition Performance. 2004

IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'04) Vol. 2. Washington, D.C. pp. 68-77 (2004)

7. Hsu, C.-W., Chang, C.-C., Len, C.-J.: A Practical Guide to Support Vector Classification.

8. Kee, G., Byun, Y., Lee, K., Lee, Y.: Improved Techniques for an Iris Recognition System with High Performance. Lecture Notes Artificial Intelligence (2001)

9. Krichen, E., Mellakh, M.A., Garcia-Salicetti, S., Dorizzi, B.: Iris Identification Using Wavelet Packets. Pattern Recognition, 17th International Conference on (ICPR'04) Vol. 4. pp. 335-338 (2004)

10. Kung, S.Y., Mak, M.W., Lin, S.H.: Biometric Authentication: A Machine Learning Approach. Pearson Education Company (2004)

11. Ma, Y., Pollick, F., Hewitt, W.T.: Using B-Spline Curves for Hand Recognition. Pattern Recognition, 17th International Conference on (ICPR'04) Vol. 3. Cambridge UK. pp. 274-277 (2004)

12. Osuna, E., Freund, R., Girosi, F.: Support Vector Machines: Training and Applications. MIT Artificial Intelligence Laboratory and Center for Biological and Computational Learning Department of Brain and Cognitive Sciences. A.I. Memo No 1602, C.B.C.L. Paper No 144 (1997)

13. Pankanti, S., Bolle, R.M., Jain, A.: Biometrics: The Future of Identification. Computer (2000)

14. Prabhakar, S., Pankanti, S., Jain, A.K.: Biometric Recognition: Security & Privacy Concerns. IEEE Security and Privacy Magazine. Vol. 1, No. 2. pp. 33-42 (2003)

15. Schlapbach, A., Bunke, H.: Off-line Handwriting Identification Using HMM Based Recognizers. Pattern Recognition, 17th International Conference on (ICPR'04) Vol. 2. Cambridge UK. pp. 654-658 (2004)

48

16. Snelick, R., Uludag, U., Mink, A., Indovina, M., Jain, A.: Large-Scale Evaluation of Multimodal Biometric Authentication Using State-of-the-Art Systems. IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol. 27, No. 3 (2005)

17. Wayman, J.L.: Error-Rate Equations for the General Biometric System. IEEE Robotics & Automation Magazine (1999)

18. Woodford, B.J., Deng, D., Benwell, G.L.: A Wavelet-based Neuro-fuzzy System for Data Mining Small Image Sets.

19. Zhang, D., Kong, W.-K., You, J., Wong, M.: Online Palmprint Identification. IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol. 25, No. 9. pp. 1041-1050 (2003)