LNCS 3851 - Modelling the Effect of View Angle Variation on Appearance-Based Gait RecognitionModelling the Effect of View Angle Variation on Appearance-Based Gait Recognition
Shiqi Yu, Daoliang Tan, and Tieniu Tan
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences,
P.O.Box 2728, Beijing, 100080, China {sqyu, dltan, tnt}@nlpr.ia.ac.cn
Abstract. In recent years, many gait recognition algorithms have been devel- oped, but most of them depend on a specific view angle. However, view an- gle variation is a significant factor among those that affect gait recognition per- formance. It is important to find the relationship between the performance and the view angle. In this paper, we discuss the effect of view angle variation on appearance-based gait recognition performance. A multi-view gait database (124 subjects and 11 view directions) is created for our research. We propose two mod- els, a geometrical one and a mathematical one, to model the effect of view angle variation on appearance-based gait recognition. These models will be valuable for designing robust gait recognition systems.
1 Introduction
Gait, as an attractive biometric for human identification at a distance, has received an increasing interest from researchers in the computer vision community. The study by Murray et al. [1] is supportive of the uniqueness of gait for a person. More importantly, gait has the advantages of being non-contact, non-invasive and easily acquired at dis- tance in contrast with other biometrics.
Among the factors which affect gait recognition performance, such as view angle, clothing, shoe type, carrying condition and surface type [2], view angle variation is a significant one since for a given gait recognition system, it is impossible to expect all the subjects to walk in a particular direction. Furthermore, most appearance-based gait recognition algorithms depend upon a specific view angle. Some researchers in computer vision have been devoting their efforts to designing view-invariant and multi- view algorithms. Johnson et al. [3] propose a multi-view algorithm, which recovers static body parameters of subjects and uses these view-invariant parameters to recognize people. Kale et al. [4] use a sophisticated method to eliminate the effect from view angle change. They synthesize the side view from another arbitrary view using a single camera through the perspective projection model or the optical flow-based structure from motion equations.
Although many gait recognition algorithms for human identification have been pro- posed and developed over the past years, most of them are view-dependent, which will limit their practical applications. In addition, there are two remaining open problems.
P.J. Narayanan et al. (Eds.): ACCV 2006, LNCS 3851, pp. 807–816, 2006. c© Springer-Verlag Berlin Heidelberg 2006
808 S. Yu, D. Tan, and T. Tan
One is which view is the most suitable for gait recognition and why it is. Kale et al [4] declare that the side view is the best choice in practice, but no theoretical results, for the time being, have been given to prove that. Another open problem is how view angle variation affects the performance of gait recognition. Intuitively, the greater the angle between the gallery (training) set and the probe (test) set is, the worse the recognition performance. However, there are not been experimental or theoretical results on the relationship between gait recognition performance and view angles. It is obvious that the answers to the above questions are significant for designing robust gait recognition systems.
This paper proposes two models, a geometrical one and a mathematical one, in an attempt to address these two questions. The purpose of this paper is to investigate and analyze the effect of view angle on the performance of appearance-based gait recogni- tion.
The remainder of this paper is organized as follows. Section 2 presents our definition of performance function. In Section 3, we discuss a multi-view gait database. Then, Section 4 introduces our experiments and gives experimental results. Two models are given in Section 5. Finally, this paper is concluded in Section 6.
2 Performance Evaluation Function
In our experiments, the correct classification rate (CCR) is used to evaluate the per- formance of gait recognition. Suppose, without the loss of generality, that the angle between the view direction of gallery set and the walking direction is θg , and the angle constituted by the view direction of probe set and the walking direction is θp. Obviously the CCR is a function of variables θg and θp.
CCR = f(θg, θp) θg ∈ [0, 360), θp ∈ [0, 360) (1)
If we can get the analytic expression of the function f(θg, θp), then how θg and θp
affect the recognition performance can be solved with ease. Discovering the expression of f(θg, θp), however, can not be easily achieved. It is impossible to precisely obtain the value of f(θg, θp) at any point in space P = [0, 360) × [0, 360) by way of experi- mental methods as P is a continuous space. A sophisticated way to solve this problem is to compute the value of f(θg, θp) through experiments at a discrete and limited subset of P. The subset can be P = {θ, 2θ, · · · , 360} × {θ, 2θ, · · · , 360}, and θ is a small angle.
As mentioned above, the video data ought to be collected from view angles ranging from θ to 360 at an increment θ. When the camera is far from the subject, the sil- houette taken from the left hand side of the subject is, from the perspective of geometry, basically similar to that from the right hand side. Therefore, the video data just need to be collected from only one side (the left side in this paper). The video data in our ex- periments is collected at view angles {0, θ, 2θ, · · · , 180}, and θ = 18. The CCR in the discrete subset {0, 1, 2, · · · , 180
θ } × {0, 1, 2, · · · , 180 θ } can be obtained
by experiments and be formulated as Equation(2).
Modelling the Effect of View Angle Variation on Appearance-Based Gait Recognition 809
CCR = F (n, k) = f(n · θ, k · θ)
n = 0, 1, 2, · · · , ⌊180
θ
⌋ (2)
θ
⌋
An algebraic formula f(θg, θp), which is an approximation to f(θg, θp) and satisfies f(θg, θp) ≈ f(θg, θp), can be acquired by data fitting and interpolation to F (n, k). A simple yet useful and reasonable model f(θg, θp) is presented in Section 6 on the basis of numerically analyzing the experimental results. The CCR at arbitrary θg and θp can be predicted or estimated with this model. It is easy to imagine that this work has a great practical meaning.
3 A Multi-view Gait Database
To analyze the impact of view angle changes on gait recognition performance, a multi- view gait database is needed. In addition to consisting of a great number of subjects, the database should be composed of the gait data collected from many view angles. The minimum angle interval ought to be relatively small.
For the purpose of developing gait recognition algorithms, a variety of gait databases have been created by many research units, such as USF [2], Soton [5], CASIA [6],
Computer 1
Computer 3
Computer 2/controller
Fig. 1. The schematic diagram of gait data collection system
Fig. 2. Sample frames from 11 view angles
810 S. Yu, D. Tan, and T. Tan
UMD, etc. The existing gait databases are either not large enough, or only captured from few view angles. These gait databases do not fulfill the requirements of view angle variation research. We develop a gait data collection system for creating a multi- view gait database which meets the requirements of view angle variation research. 11 cameras were used to capture gait videos as illustrated in Fig. 1.
All the subjects are asked to walk naturally on the concrete ground along a straight line in an indoor environment. The videos can be simultaneously captured by 11 cam- eras from different view directions. At last we successfully collect 124 subjects’ gait data (94 males and 30 females). The view angle θ between the view direction and the walking direction takes on the values of 0, 18, 36, · · · , and 180, as delineated in Fig. 1. Each subject walks along the straight line 10 times (6 for normal walking, 2 for walking in a coat and 2 for walking with a bag), and 11 video sequences are captured each time. Thus, there are 110 sequences for each subject, and a total of 110 × 124 = 13640 video sequences in our database. All the video sequences have the same resolution of 320 × 240 pixels. Some sample frames from 11 cameras are shown in Fig. 2.
Our database comprises those factors affecting gait recognition: view angles (11 views), clothing (with or without in a coat), and carrying condition (with or without a bag). Only view angles is studied here, though other factors are interesting to study too.
4 Gait Feature Extraction
There are many appearance-based gait features in the literature. Most of them are ex- tracted from human silhouettes or outer contours. Here we choose one typical feature from each category. One is gait energy image (GEI), and it is extracted from human silhouettes. GEI is introduced in [7], which is the average of all silhouettes in a video sequence. The other is key Fourier descriptors (KFDs), and it is extracted from human outer contours. KFD method is proposed by Yu et al. [8], which is the key compo- nent of Fourier descriptors computed from human contours. Finally, we use the nearest neighbor classifier to perform classification.
4.1 Silhouette Segmentation
Given a fixed camera, the human silhouette can be extracted by background subtraction and thresholding. We take advantage of the method given in [9] to segment human sil- houette from image sequences. The sizes of the silhouettes we extracted are not unique, and the silhouettes need to be normalized to the same size.
4.2 GEI Feature Extraction
The gait energy image is reported as a good feature which is robust to silhouette errors and image noise, and is defined by [7]
G(x, y) = 1 N
I(x, y, t) (3)
where N is the number of frames in the sequence I(x, y, t), t is the frame number, x and y are the image coordinate [7].
Modelling the Effect of View Angle Variation on Appearance-Based Gait Recognition 811
4.3 KFD Feature Extraction
To extract KFD feature, the outer contour first needs to be obtained. The outer contour can be easily derived using a border-following algorithm based on connectivity. Then all the contours and the gait cycle are normalized to have the same number (N ) of sample and the same number (T ) of frame, respectively. All Fourier descriptors g(i) can be obtained by discrete Fourier transform. The KFDs are defined as in [8]:
G = [ |g(2T )| |g(T )| ,
] (4)
where N is the number of sample points of each contour, and T is the number of frames in a gait cycle.
5 Experimental Results and Analysis
Each subject has 6 normal walking sequences at each view angle. Wet put the first 4 sequences into the gallery set, and the last 2 sequences into the probe set. A number of experiments are carried out to discover the relationship between gait recognition performance and view angles. Fig. 3 and Table 1 show the CCRs of the experiments which take GEI as a feature. It can be noticed from Fig. 3 that there exist two peaks on the CCR curves in each subfigure. and that CCR reaches the first peak at θp = θg and the second peak at θp = 180 − θg. A geometrical model is proposed for explaining the existence of these two peaks. Fig. 4 and Table 2 display the CCRs of the experiments which take KFDs as a feature. As in Fig. 3 and Table 1, a similar phenomenon can be found in Fig. 4 and Table 2.
We can get that CCRs basically remains a constant CM along the major diagonal line (θg = θp) of Tables 1 and 2, and a constant Cm along the major skew diagonal line (θg = 180 − θp) except (θg, θp) ∈ {(108, 72), (90, 90), (72, 108)}.
From Figures 3(f) and 4(f), it can be seen that the CCR basically remains high when θp varies around 90. Thus, the CCR at the side view is robust to view angle change with respect to other views.
Table 1. CCR table (%) for GEI (rank=1)
Gallery Probe angle θp
angle θg 0 18 36 54 72 90 108 126 144 162 180
0 99.2 31.9 9.3 4.0 3.2 3.2 2.0 2.0 4.8 12.9 37.9 18 23.8 99.6 39.9 8.9 4.4 3.6 3.6 5.2 13.7 33.5 10.9 36 4.4 37.9 97.6 29.8 11.7 6.9 8.1 13.3 23.4 13.3 2.0 54 2.4 3.6 29.0 97.2 23.0 16.5 21.4 29.0 21.4 4.8 1.2 72 0.8 4.4 7.3 21.8 97.2 81.5 68.1 21.0 5.6 3.6 1.6 90 0.4 2.4 4.8 17.7 82.3 97.6 82.3 15.3 5.2 3.6 1.2
108 1.6 1.6 2.0 16.9 71.4 87.9 95.6 37.1 6.0 2.0 2.0 126 1.2 2.8 6.0 37.5 33.5 22.2 48.0 96.8 26.6 4.4 2.0 144 3.6 5.2 28.2 18.5 4.4 1.6 3.2 43.1 96.4 5.6 2.8 162 12.1 39.1 15.7 2.4 1.6 0.8 0.8 2.4 5.2 98.4 28.6 180 41.1 19.8 8.1 3.2 2.0 0.8 1.6 3.6 12.5 51.2 99.6
812 S. Yu, D. Tan, and T. Tan
Table 2. CCR table (%) for KFD (rank=1)
Gallery Probe angle θp
angle θg 0 18 36 54 72 90 108 126 144 162 180
0 71.8 5.2 5.2 2.4 0.8 1.2 1.2 0.8 1.6 3.2 33.1 18 3.6 49.2 14.5 4.4 2.8 3.2 4.0 3.6 4.0 8.9 4.0 36 2.8 12.1 72.6 11.7 3.6 2.8 2.0 3.2 14.1 10.9 2.4 54 2.0 3.2 10.5 69.4 7.7 2.4 4.4 14.9 10.9 3.2 0.8 72 0.4 0.8 2.8 12.9 77.8 16.9 25.0 8.9 2.4 1.2 0.0 90 0.4 0.8 3.6 4.4 23.0 75.0 20.6 4.0 2.0 0.8 1.2
108 0.4 2.4 3.2 5.2 20.6 21.4 69.8 10.5 2.8 1.2 0.8 126 0.8 3.6 4.8 14.9 11.7 5.6 14.1 71.4 10.5 3.6 1.6 144 2.0 6.9 16.1 12.1 4.0 2.4 2.8 12.5 71.0 11.7 3.2 162 2.8 10.9 10.9 1.6 2.0 2.4 2.8 6.0 11.3 72.2 3.6 180 30.6 3.2 4.8 1.6 2.0 2.4 1.2 2.8 3.6 7.3 67.7
Gallery View Angle=0
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
Rank=1
Rank=3
Rank=5
Fig. 3. CCRs(%) for GEI(the abscissa is θp, and the ordinate is CCR)
5.1 A Geometrical Model
Why are there two symmetrical (though not strictly) peaks on the curves in Fig. 3 and 4? In our opinion, it is the human body symmetry that results in this phenomenon. Suppose that 3 images are, respectively, taken from 3 different view angles θ, 180 − θ, and 180+θ, as illustrated in Fig. 5, and that the cameras are far away from the subject.
Modelling the Effect of View Angle Variation on Appearance-Based Gait Recognition 813
Gallery View Angle=0
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
CCR
0 20 40 60 80 100 120 140 160 180 0
10
20
30
40
50
60
70
80
90
100
Rank=1
Rank=3
Rank=5
Fig. 4. CCRs(%) for KFD(the abscissa is θp, and the ordinate is CCR)
180-
180+
Fig. 5. Images at 3 view angles
S(θ), S(180 − θ) and S(180 + θ) represent the 3 human silhouettes extracted from the 3 corresponding images. For the distance between the cameras and the subject is large, we can reasonably consider:
S(θ) ≈ −→ S (180 + θ) (5)
where the symbol → means flip horizontally. On the other hand, human body has the symmetry, so we have −→
S (180 + θ) ≈ S(180 − θ) (6)
Walking can partly break this symmetry with respect to a fixed view angle, which is the reason for using the symbol ≈ in Equation (6).
814 S. Yu, D. Tan, and T. Tan
From Equations (5) and (6), it is straightforward to get Equation (7):
S(θ) ≈ S(180 − θ) (7)
This implies that the silhouette from view angle θ is similar to that from view angle 180 − θ. Thus, when the gallery angle is θ, using the data from 180 − θ as probe can produce relative higher CCR, from which a local peak positioned at 180 − θ naturally occurs, compared with the values of CCR at other view angles distant from θ.
5.2 A Mathematical Model
Based on the analysis in Section 6.1, there are two peaks on CCR curves in Figures 3 and 4. By observing the shapes of curves, we can reasonably use a mixed Gaussian function to model CCR curves.
The analytical expression of the mathematical model is defined as:
f(θg, θp) = CMe− (θg−θp)2
2σ2 + Cme− (180−θg−θp)2
2σ2
2σ2
] (8)
where CM and Cm are the same as previous definitions. and σ is treated as a constant in our experiments, which indicates the level of performance deterioration when the probe view departs from the gallery view. The value of σ is optimized by the Curve Fitting
Toolbox in Matlab, and it takes 15 here. CM e− (θg−θp)2
2σ2 and Cme− (180−θg−θp)2
2σ2 are
two Gaussian functions which simulate the two ridges in Figures 6 and 8. 1−e− (θg−θp)2
2σ2
is a weighting term which makes sure that f(θg, θp) does not exceed unity. The CCRs in Table 1 are shown in Fig. 6. Fig. 8 displays the CCRs in Table 2.
Fig. 7 and Fig. 9 are the continuous versions of Fig. 6 and Fig. 8 obtained from our mathematical model (equation (8)), respectively. The theoretical results computed from Equation (8) generally conform to the experimental ones in Table 1 and Table 2.
0 18 36 54 72 90 108 126 144 162 180
0 18
36 54
72 90
108 126
144 162
180 0
)
Fig. 6. CCRs for GEI Fig. 7. The modelled CCRs for GEI
From Equation (8), Fig. 7 and Fig. 9, there are two perpendicular ridges which super- pose each other. It is this superposition that makes the CCR around the site (90, 90) much higher than in other regions. Thus, the CCR at the side view is robust to view angle variation.
Modelling the Effect of View Angle Variation on Appearance-Based Gait Recognition 815
0 18 36 54 72 90 108 126 144 162 180
0 18
36 54
72 90
108 126
144 162
180 0
)
Fig. 8. CCRs for KFD Fig. 9. The modelled CCRs for KFD
6 Conclusions and Future Work
In this paper, we have presented an analysis of the effect of view angle variation on the performance of appearance-based gait recognition methods the proposed and mod- els. The novelty of our work is three-fold: first, it is a systematic study on multi-view gait recognition; secondly, it investigates how the performance is affected by view an- gle changes with a useful mathematical model depicting the relationship between view angle and performance(despite the current simplicity); last but obviously not least, it an- swers two open questions: why the side view is more suitable to recognize human gaits, and how view angle variation impacts the gait recognition performance. Our future work will be focused on view-invariant gait feature extraction, a better mathematical model taking into account the effect of θg, θp and features on σ, and the establishment of a multi-view gait database in an outdoor environment.
Acknowledgement
This work is partly supported by National Natural Science Foundation of China (Grant No. 60335010), National Basic Research Program of China (Grant No. 2004CB318100) and International Cooperation Program of Ministry of Science and Technology of China (Grant No. 2004DFA06900).
References
1. Murray, M.P.: Gait as a total pattern of movement. American Journal of Physical Medicine 46 (1967) 290–332
2. Sarkar, S., Phillips, P.J., Liu, Z., Vega, I.R., Grother, P., Bowyer, K.W.: The humanid gait chal- lenge problem: Data sets, performance and analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 27 (2005) 162–177
3. Johnson, A.Y., Bobick, A.F.: A multi-view method for gait recognition using static body parameters. In: Proc. of 3rd International Conference on Audio and Video Based Biometric Person Authentication, Halmstad, Sweden (2001) 301–311
816 S. Yu, D. Tan, and T. Tan
4. Kale, A., Chowdhury, A.K.R., Chellappa, R.: Towards a view invariant gait recognition algo- rithm. In: Proc. of IEEE Conference on Advanced Video and Signal Based Surveillance, FL, USA (2003) 143–150
5. Shutler, J.D., Grant, M.G., Nixon, M.S., Carter, J.N.: On a large sequence-based human gait database. In: Proc. of the 4th International Conference on Recent Advances in Soft Computing, Nottingham, UK (2002) 66–72
6. : (Center for biometrics and security research, casia. http://www.cbsr.ia.ac.cn) 7. Han, J., Bhanu, B.: Statistical feature fusion for gait-based human recognition. In: Proc. of
the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington DC, USA (2004) II842–II847
8. Yu, S., Wang, L., Hu, W., Tan, T.: Gait analysis for human identification in frequency domain. In: Proc. of the 3rd International Conference on Image and Graphics, Hong Kong, China (2004) 282–285
9. Wang, L., Tan, T., Ning, H., Hu, W.: Silhouette analysis-based gait recognition for human identification. IEEE Transactions on Pattern Analysis and Machine Intelligence 25 (2003) 1505–1518
Introduction