Multimodal Biometrics Recognition by Dimensionality Reduction Method

Cheng Lu, Computer School Jilin University

Changchun, China kongj435@nenu.edu.cn

Di Liu*, Jisong Wang, Shuyan Wang Computer School

Northeast Normal University Changchun, China

wangjz019@nenu.edu.cn

Abstract—Multimodal biometric system utilizes two or more individual modalities, e.g., face, ear, and fingerprint, to improve the recognition accuracy of conventional unimodal methods. We propose a new dimensionality reduction method called Dimension Reduce Projection (DRP) in this paper. DRP can not only preserve local information by capturing the intra-modal geometry, but also extract between-class relevant structures for classification effectively. Experimental results show that our proposed method performs better than other algorithms including PCA, LDA and MFA.

Keywords: 2DPCA; Multimodal biometrics; Dimensionality reduction; Face recognition; Palmprint recognition.

I. INTRODUCTION Biometrics is an emerging technology [1] that is used to

identify people by their physical and/or behavioral characteristics and, so, inherently requires that the person to be identified is physically present at the point of identification. The physical characteristics of an individual that can be used in biometric identification/verification systems are fingerprint, hand geometry, palm print [2], face [3], iris, retina, and ear; the behavioral characteristics are signature, lip movement, speech, keystroke dynamics, gesture, and gait [1]. Biometric systems based on a single biometric characteristic are referred to as unimodal systems. They are usually more cost-efficient than multimodal biometric systems. However, a single physical or behavioral characteristic of an individual can sometimes fail to be sufficient for identification. For this reason, multimodal biometric systems that integrate two or more different biometric characteristics are being developed to provide an acceptable performance, to increase the reliability of decisions, and to increase robustness to fraudulent technologies [4].

Hong et al. [5] developed a prototype multimodal biometric system, which integrates faces and fingerprints at the identification stage. Ribaric and Fratric [6] presented a multimodal biometric system based on features extracted from fingerprint and palmprint data.

Recently, subspace methods, which select low dimensional features to represent raw data, have been widely studied in biometrics researches [7, 8]. It is shown that

subspace selection is one of the most important steps for entire biometric systems. These methods which carry out fusion at feature level obtain the low-dimensional features integrated the information of all modalities.

In this paper, we propose a new dimensionality reduction method called Dimension Reduce Projection (DRP) for multimodal biometric recognition. The rest of this paper is organized as follows: Section 2 describes the proposed algorithm DRP. In Section 3, experimental results on the face and palmprint databases are presented. We conclude this paper in section 4.

II. Dimensionality Reduce Projection To take face images as an example, there are two kinds

of face difference: intra-class face difference and extra-class difference. The intra-class difference contains transformation difference such as illumination, pose changes while the extra-class difference contains intrinsic difference (identity changes), transformation difference.

In this section, DRP is proposed as a new dimensionality reduction method, which can be used to extract features from multimodal biometric data. It modeling intra-class metric matrix to characterize the distribution of transformation difference and extra-class metric matrix to characterize the distribution of intrinsic difference. Figure 1 shows the block diagram of the proposed system for multimodal biometrics system.

It is well known that the manifold or distribution of one face (or palmprint) under variations such as viewpoint, deformation, and illumination is highly nonlinear and nonconvex [11]. However, we can recover the nonlinear structure by locally linear fits [12]. We adopt linearization extension of locally linear embedding (LLE) [13] to model the intra-class metric matrix. LLE can preserve the local geometry of the faces in the same class. Because of the high-dimensional raw data, the step of preprocessing is recommended for compressing the raw data. In this work, we adopt a straightforward image projection technique, Two-Dimensional Principal Component Analysis (2DPCA) [9] for initial dimension reduce.

*Corresponding author

2009 Second International Symposium on Electronic Commerce and Security

978-0-7695-3643-9/09 $25.00 © 2009 IEEE

DOI 10.1109/ISECS.2009.166

113

Figure 1. The block diagram of the proposed system.

A. Two-Dimensional Principal Component Analysis Two-dimensional principal component analysis (2DPCA)

is developed for image feature extraction. As opposed to conventional PCA, 2DPCA is based on 2D matrices rather than 1D vectors. That is, the image matrix does not need to be previously transformed into a vector. In contrast to the covariance matrix of PCA, the size of the image covariance matrix using 2DPCA is much smaller. As a result, 2DPCA has two important advantages over PCA. First, it is easier to evaluate the covariance matrix accurately. Second, less time is required to determine the corresponding eigenvectors.

Let P denote an n-dimensional unitary column vector. Our idea is to project image A, an nm× random matrix, onto P by the following linear transformation:

.B AP= (1) Thus, we obtain an m-dimensional projected vector B,

which is called the projected feature vector of image A. The projection vector P can be determined by solving

eigenvector of image covariance (scatter) matrix Gt corresponding to the largest eigenvalue [10], where

( )( ) ( )( )[ ]AEAAEAEG Tt −−= .We can evaluate Gt directly

using the training image samples. Suppose that there are M training image samples in total, the jth training image is denoted by an nm× matrix Aj (j=1, 2, …, M), and the average image of all training samples is denoted by A . Then, Gt can be evaluated by:

1

1 ( ) ( )M

Tt j j

j

G A A A AM =

= − −∑ (2)

In general, it is not enough to have only one optimal projection axis. We usually need to select a set of projection axes, P1, P2,…, Pd, subject to the orthonormal constraints. In

fact, the optimal projection axes, P1, P2,…, Pd, are the orthonormal eigenvectors of Gt corresponding to the first d largest eigenvalues.

Then the optimal projection vectors P1, P2,…, Pd are used for initial dimensionality reduction. For a given image sample A, let

, 1,2,..., .k kB AP k d= = (3) then, we obtain a family of projected feature vectors, B1, B2,…, Bd, which are called the principal component (vectors) of the sample image A.

The principal component vectors obtained are used to form a nm× matrix F=[ B1, B2,…, Bd], which is called the feature matrix of the image sample A. For each image A (face or palm), we can transform the feature matrix F into 1D feature vectors x and prepared for the following dimensionality reduction steps.

B. Modeling of intra-class metric matrix After preprocessing all raw biometric data from various

modalities, the images are compressed and sorted by its feature vectors x. Denote the multimodal biometric data set nm

N RxxX ×∈= ],,[ 1r

Kr , and each datum ixr belongs to

one class, each of which has two or more modalities. The data X is the training data set, on which the DRP algorithm is developed. The system obtains a transformation matrix

dmRU ×∈ that maps the set X of N points to the set nd

N RyyY ×∈= ],,[ 1r

Kr

, such that Y=UTX, where d<m. Given the testing set P, we can project it to the subspace via the transformation matrix U as Q=UTP. Finally, we classify the dimension-reduced testing data Q in the subspace by matching with the corresponding training data Y.

Roweis et al. [12] proposed the LLE algorithm. LLE tries to recover the global nonlinear structure from locally linear fits. LLE supposes that each data point and its neighbor points lie on a locally linear patch of the manifold. Each data point can be reconstructed by a linear combination of its k nearest neighbors. LLE constructs a neighborhood-preserving mapping based on the weight matrix. As a result, LLE can preserve the local manifold structure of one face class in the dimensionality reduced space. LLE computes the weight matrix by minimizing the reconstruction error:

2min || ||i ij ji j

x xα−∑ ∑ (4)

where 0=ijα if xi and xj belong to different classes and the

rows of the weight matrix sum to one: ∑ =j

ij 1α .

The constrained weights that minimize the reconstruction error obey an important symmetry: for any particular data points, they are invariant to rotations, rescaling, and translations of that data point and its neighbors [12]. In order to preserve the neighborhood structure of the original image space, we should minimize the following cost function [13]:

Subspace after 2DPCA

Low-dimension subspace

2DPCA

DRP

Classification

Low-dimension subspace

Testing set

Result

2DPCA , DRP

Training set

114

2min

2

( ) min || ||

min || ||

min ( ( ) ( ) )

min ( )

i ij ji j

T Ti ij jU i j

T T T

UT

U

J U y y

U x U x

tr U X I A I A X U

tr U WU

α

α

= −

= −

= − −

=

∑ ∑

∑ ∑

(5) where yi is the project vector of xi. U is the projection matrix and A is the weight matrix. W=X(I-A)T(I-A)XT denotes the intra-class metric matrix and tr() stands for the operation of trace. ],,[ 21 nxxxX K= is the original data matrix and

)1,1,1( KdiagI = .

C. Modeling of extra-class metric matrix Specifically, in the low-dimensional subspace, we aim to

map the raw biometric data from different individuals as far apart as possible, while the intra- modal data from same class remain nearby as before. Hence we map the neighbor different class points far apart:

2

1 1

max || ||N N

Ti j ij

i jU x U x s

= =

−∑∑uur uur

(6) where sij=1 if ixr and jxr are from the different classes, and ixr is one of k nearest neighbors of jxr or jxr is one of k nearest neighbors of ixr ,otherwise sij=0. Eq. (6) reduces to:

1 1

max {( )( ) }

max 2 ( )max ( ( ) ),

N NT T T T T

i j i j iji j

T T T T

T T

tr U x U x U x U x s

tr U XDX U U XSX Utr U X D S X U

= =

− −

= −

= −

∑∑uur uur uur uur

(7) where tr() denotes the trace operator, S is the matrix with S(i, j)=sij and D is a diagonal matrix with ∑=

jijii sd .

Combining Eqs. (5) and (6) together, we can write the final objective function which needs to be solved as follows:

2

1

2

1 1

min( || ||

1 || || )2

NT T

i ij ji j

N NT T

i j iji j

U x U x

U x U x s

α=

= =

−

− −

∑ ∑

∑∑

uur uur

uur uur

(8) according Eqs. (5) and (7), Eq. (8) can reduce to

1min { ( ( )) }2

T Ttr U X W D S X U− −. (9)

In order to uniquely determine U, we impose the constraint UTU=1, that is, the columns of U are orthonormal. Now, the objective function has the final form:

1arg min { ( ( )) }2

T T

Utr U X W D S X U− −

, w.r.t. UTU=I (10)

Obviously, the above optimization problem can be converted to solving a standard eigenvalue problem:

1( ( )) .2

TX W D S X β λβ− − =ur ur

(11) Let the column vectors dβββ

rK

rr,, 21 be the solutions of

Eq. (11), ordered according to the eigenvalue dλλλ <<< K21 . Thus, the optimal transformation matrix U is given by ],,[ 21 dU βββ

rK

rr= .

The proposed algorithm successfully avoids the singular problem since it never computes the inverse of one matrix, different from the algorithms which lead to the generalized eigenvalue problem.

Note that, the raw data were pre-compressed by the 2DPCA firstly and thereby the eigen-decomposition

on ( ) TXSDWX

−−

21 would not be computationally

expensive. We can choose the optimal value for the reduced dimension in the 2DPCA step, without the discriminative information loss. Thus, we perform DRP in the 2DPCA-projected subspace and the ultimate transformation matrix is as follows:

2DPCA DRPU = U U (12)

D. Classification in the low-dimensional space By using the DRP algorithm, the transformation matrix U

is yielded, and then the training data can be projected into the low dimensional space: ],,[],,[ 2121 N

TN xxxUyyy r

Krrr

Krr

= . The testing data P are projected onto Q=UTP. The dimension-reduced testing data Q and the dimension-reduced training data Y have the same low-dimensional space where we can perform the classification.

Given a testing individual constituted by the multimodal features

TNqqq rK

rr,, 21 , where NT is the number of the

modalities of the individual. It is permitted that some modalities of testing samples are not available. For each modality jqr , we measure the Euclidean distance between the given feature and all the dimension-reduced training samples

Nyyy rK

rr,, 21 . So we have distance vector Dj for modality jqr :

1[ ( , ),..., ( , )]j j j ND d q y d q y=uur r ur r ur

, (13) where j=1, 2,…,NT. Considering all the modalities of the testing individual, we have the following decision:

1 ,... ( )TN iq q label y∈

r r ur

(14) If ],,[minmin),( 1 TNij DDyqd K

rr= , for j=1, 2,…, NT,

and i=1, 2,…, N.

III. Experiments In this section, the performance of DRP was evaluated on

the ORL face databases and the PolyU palmprint database [14]. Our system can be extended to include more modalities.

115

There are 10 different images of 40 distinct subjects contained in ORL database. For some of the subjects, the images were taken at different times, varying lighting slightly, facial expressions and facial details.. The size of each image is 92× 112, 8-bit grey levels. The PolyU palmprint database contains 7752 grayscale images corresponding to 386 different palms.

We select 25 individuals and for each individual select 10 face images and 10 palmprint images, 5 for training and the other 5 for testing. That is, we have 125 images for training and 125 images for testing in one modal. We compared DRP with the performances of other methods including PCA [8], LDA, MFA [15]. In the preprocessing step, we take d=5 (5 projection vectors), for the face image size of 92×112, then the dimension after 2DPCA is reduced to 560. The nearest extra-class neighbors parameter k in DRP is chosen as k=4 and 5, so we can obtain 2 results DRP4 and DRP5. Table 1 gives the recognition rates with the corresponding reduced dimensions of the employed algorithms on ORL and PolyU database. Figure 2 shows the average recognition rates versus subspace dimensions in multimodal tests. We can see that the proposed algorithm DRP performed much better than other algorithms. The performances of DRP4 and DRP5 are nearly the same.

We also take another experiment on the variance of d (number of projection vectors). Table 2 gives the recognition rates and recognition time.

Figure 2. Recognition results on the ORL and PolyU database

TABLE I. RECOGNITION RESULTS (%) ON THE ORL AND POLYU DATASET

Method Recognition Rate(%) Dimension (d) PCA 73.6 400 LDA 83.2 200 MFA 89.6 350

4DRP 93.6 30

5DRP 95.2 30

TABLE II. RECOGNITION RESULTS(%) VERSUS THE NUMBER OF PROJECTION VECTORS ON THE ORL AND POLYU DATA SET.

d Recognition Rate(%) Recognition Time(s) Dimension

3 78.2 5.41 100

5 93.6 9.65 30

10 95.8 26.34 30

IV. CONCLUSION In this paper, a new dimensionality reduction method

called Dimension Reduce Projection (DRP) is proposed for multimodal biometric recognition. DRP preserves local information by capturing the intra-modal geometry and extract between-class relevant structures which is useful for classification. Experimental results have shown the effectiveness and the advantages of the proposed method.

REFERENCES [1] Biometrics: Personal Identification in Networked Society, A.K. Jain,

R. Bolle, and S. Pankanti, eds. Kluwer Academic, 1999. [2] W. Shu and D. Zhang, “Automated Personal Identification by

Palmprint,” Optical Eng., vol. 37, no. 8, pp. 2359-2362, Aug. 1998. [3] Intelligent Biometric Techniques in Fingerprint and Face Recognition,

L.C. Jain, U. Halici, I. Hayashi, and S.B. Lee, eds. CRC Press, 1999. [4] A.K. Jain and A. Ross, “Multibiometric Systems,” Comm. ACM, vol.

47, no. 1, pp. 34-40, Jan. 2004. [5] L. Hong, A.K. Jain, Integrating faces and fingerprints for personal

identification, IEEE Trans. Pattern Anal. Mach. Intell. 20 (12) (1998) 1295–1307.

[6] S. Ribaric, I. Fratric, A biometric identification system based on eigenpalm and eigenfinger features, IEEE Trans. Pattern Anal. Mach. Intell. 27 (11) (2005) 1698–1709.

[7] S. Yan, D. Xu, B. Zhang, H. Zhang, Q. Yang, S. Lin, Graph embedding and extensions: a general framework for dimensionality reduction, IEEE Trans. Pattern Anal. Mach. Intell. 29 (1) (2007) 40–51.

[8] M. Turk, A. Pentland, Eigenfaces for recognition, Int. J. Cognitive Neurosci. 3 (1) (1991) 71–86.

[9] Jian Yang, David Zhang, Alejandro F. Frangi, and Jing-yu Yang. Two-Dimensional PCA: A New Approach to Appearance-Based Face Representation and Recognition, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 1, JANUARY 2004.

[10] J. Yang, J.Y. Yang, “From Image Vector to Matrix: A Straightforward Image Projection Technique—IMPCA vs. PCA,” Pattern Recognition, vol. 35, no. 9, pp. 1997-1999.

[11] S.Z. Li, A.K. Jain, Handbook of Face Recognition, Springer, New York, 2005.

[12] S. Roweis, L.K. Saul, Nonlinear dimensionality reduction by locally linear embedding, Science 290 (5500) (2000) 2323–2326.

[13] X. He, D. Cai, S. Yan, H.J. Zhang, Neighborhood preserving embedding, in: Proceedings of the IEEE International Conference on Computer Vision, 2005, pp. 1208–1213.

[14] PolyU Palmprint Palmprint Database, /http://www.comp.polyu. edu.hk/_biometrics/S, 2006.

[15] S.C. Yan, D. Xu, B. Zhang, H.J. Zhang, Graph embedding and extensions: a general framework for dimensionality reduction, IEEE Trans. Pattern Anal. Mach. Intell. 29 (1) (2007) 40–51.

116