Real Time 3D Face Pose Tracking From an Uncalibrated Camera

Real Time 3D Face Pose Tracking From an Uncalibrated Camera

Zhiwei ZhuDepartment of ECSE

Rensselaer Polytechnic Institute, Troy, [email protected]

Qiang JiDepartment of ECSE

Rensselaer Polytechnic Institute, Troy, [email protected]

1 Abstract

We propose a new near-real time technique for 3D facepose tracking from a monocular image sequence obtainedfrom an uncalibrated camera. The basic idea behind ourapproach is that instead of treating 2D face detectionand 3D face pose estimation separately, we perform si-multaneous 2D face detection and 3D face pose tracking.Specifically, 3D face pose at a time instant is constrainedby the face dynamics using Kalman Filtering and by theface appearance in the image. The use of Kalman Filter-ing limits possible 3D face poses to a small range whilethe best matching between the actual face image andthe projected face image allows to pinpoint the exact 3Dface pose. Face matching is formulated as an optimiza-tion problem so that the exact face location and 3D facepose can be estimated efficiently. Another major featureof our approach lies in the use of active IR illumination,which allows to robustly detect eyes. The detected eyescan in turn constrain the face in the image and regular-ize the 3D face pose, therefore the tracking drift issuecan be avoided and the processing can speedup. Finally,the face model is dynamically updated to account forvariations in face appearances caused by face pose, faceexpression, illumination and the combination of them.

Compared with the existing 3D face pose trackingtechniques, our technique has the following benefits.First, our technique can track face and face pose simul-taneously in real time, which has been implemented as areal time working system. Second, only one uncalibratedcamera is needed for our technique, which will make oursystem very easy to set up. Third, our technique canhandle facial expression change, face occlusion and illu-mination change, which will make our system work underreal life conditions.

2 Introduction

Face pose tracking is very important in vision basedapplications such as HCI, face recognition, and virtual re-ality. Many techniques have been proposed for face poseestimation. Basically, face pose estimation techniquescan be classified into two main categories: appearance-

based approaches [1, 2, 3, 4, 5] and model-based ap-proaches [6, 7, 8, 9]. Appearance-based approaches at-tempt to use holistic facial appearance, where face istreated as a two-dimensional pattern of intensity vari-ations. They assume that there exists a mapping rela-tionship between 3D face pose and certain properties ofthe facial image, which is constructed based on a largenumber of training images. The appearance-based ap-proaches usually only provides a sparse set of face posesrather than continuous-valued face poses. In summary,the main benefit of these approaches is their simplicity,but there are several significant hurdles. First, despiteefforts such as the use of wavelet transform to minimizethe effect of some factors, the mapping function is still afunction of many other factors including face poses, suchas illuminations, camera parameters and etc. Second,these techniques often require a face detector first. Theyoften rely on others’ techniques to detect faces or cropthe face regions by hand. Finally, face pose is estimatedon the detected face region. In these techniques, facedetection and face pose estimation is done separately,ignoring any dependence between them.

Model-based (or features-based) approaches usuallyassume a 3D model of the face and recover the face posebased on the assumed model. First, a set of 2D − 3Dfeature correspondences are established. Then the facepose is estimated by using the conventional pose estima-tion techniques [7, 8, 6]. Ji et al. [9] propose a 3D facepose estimation by modelling the face as an ellipse andby using anthropometric statistics of the face. Face posesare recovered based on the distortion of the face images.So, for most of these approaches, pose estimation is doneafter face feature tracking. Though simple to implement,robust and accurate facial feature tracking is often a sig-nificant challenge due to face occlusion, face movement,facial expression change and illumination change.

We can conclude that most existing methods for facepose estimation follow the strategy of face detection inthe image and face pose estimation from the detectedface images. The main problem with all these approachesis that face detection and face pose estimation are carried

0-7695-2158-4/04 $20.00 (C) 2004 IEEEProceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’04) 1063-6919/04 $ 20.00 IEEE

out independently. There is no input from each other.But in reality, these two steps are very interrelated, andbecause the face location in the image is caused by facepose in 3D, they must be consistent with each other.Therefore, we propose to take full advantage of the inter-dependent relationship between face image and 3D facepose and perform face detection and face pose estimationsimultaneously.

Several similar techniques [10, 11, 12, 13, 14] have beenproposed recently to track the face in 3D space. SumitBasu et al [13] presented a 3D head tracker by modellingthe head as an ellipsoid. Experiments show that thealgorithm is stable to extract 3D head information accu-rately, but it is sensitive to the motion being observedin the scene due to the use of optical flow regularization.Recently, La Cascia et al. [15] present a 3D head track-ing system that is robust to varying illuminations. Intheir technique, the head is modeled as a texture mappedcylinder and head tracking is formulated as registeringthe face image with the cylinder’s texture mapped im-ages under different face poses. In other words, trackingis based on the minimization of the sum of squared dif-ferences between the incoming texture and a referencetexture. Therefore, it is sensitive to the changes in facialexpressions, illuminations and self-occlusion. Althoughan illumination corrector is used to avoid the illumina-tion effect under the assumption of a Lambertian sur-face in the absence of self-shadowing , it can not workvery well under the indoor environment because the facesurface is not truly Lambertian nor is there an absenceof self-shadowing. Alternatively, in this paper, we aregoing to introduce a face model updating strategy toaccount for these changes, which can improve the accu-racy of the SSD based tracker dramatically under thesechanges. Also, because human is not truly cylindrical,the accuracy of their system, as explored by Brown [16],degenerates under some conditions, such as large rota-tions around the vertical axis and large frame to framepose changes. Even, La Cascia’s approach is unable todistinguish rotations around the horizontal axis and ver-tical translations or similar rotations around the verticalaxis and the horizontal translations. In order to improvethe accuracy of the head tracker, Brown [16] presentsmethods to overcome these problems. First, adaptivemotion templates are used to handle the variable frame-to-frame motion changes. Second, 2D positional infor-mation of the neckline is utilized to constrain the 3D po-sitional estimates. Third, additional motion templatesare used to compensate for the large head rotations. Butthere are still several issues. Neckline is difficult to trackrobustly due to possible occlusions, and the strategy touse additional templates is not convincing. Even, theskin tone based face detector will fail under some situa-tions such as poor illuminations. More recently, Thomas

Vetter et al. [17] proposed an algorithm to fit the 2Dface images with 3D Morphable Models such that theface pose can be estimated. Although the face pose canbe estimated accurately, the average processing time foreach frame is around 30 seconds, which is too slow for areal time face pose estimation system.

In this paper, we describe a new technique to per-form the 2D face tracking and 3D pose estimation syn-chronously. In our method, 3D face pose is tracked us-ing Kalman Filtering. The initial estimated 3D pose isthen used to guide face tracking in the image, which issubsequently used to refine the 3D face pose estimation.Face detection and face pose estimation work togetherand benefit from each other. Weak perspective projec-tion model is assumed so that face can be approximatedas a planar object with facial features, such as eyes, noseand mouth, located symmetrically on the plane. Figure1 summarizes our approach.

Frontal-parallel Face Model Acquisition

Predict Next 3D Face Pose UsingKalman Filter

Refine 3D Face Pose via Model BasedFace Image Matching

Update 3D Face Pose and Face Model

Model Based 3DFace Pose Tracking

Frame K+1

Frame K

Figure 1. The Flowchart of Face Pose Tracking

First, we automatically detect a frontal-parallel faceview image based on the detected eyes and some sim-ple anthropometric statistics. The detected face regionis used as the initial 3D planar face model. The 3D facepose is then tracked starting from the frontal-parallel facepose. During tracking, the 3D face model is updated dy-namically to account for the face appearance changes in-troduced by the illumination, facial expression, face pose,or the combinations of them, and the face detection andface pose estimation are synchronized and kept consistentwith each other. Also, the use of the 2D eye location isused to constrain the face location in the image, whichcan avoid the tracking drift issue and speedup the pro-cessing. If the tracker fails, it will recover automaticallyby the robust recovery technique.

Because of these improvements, our technique can suc-cessfully track face pose under large out-of-plane facerotations. It is more tolerable to illumination and fa-cial expression changes. Furthermore, face pose can be


robustly tracked even under rapid head movements. Fi-nally, automatic face detection is performed to initialize3D face model.

3 Weak Perspective Model for Face PoseEstimation

We employ an object coordinate system affixed touser’s face, with face normal (face pose) being the Zaxis of the object frame. Without loss of generality, faceis assumed to be planar. Let X = (x, y, 0)t be the co-ordinate of a 3D point on the face relative to the objectcoordinate frame, and p = (c, r)t the coordinate of thecorresponding projection image point in the row-columnframe.

For each pixel (c1, r1) in a given image frame I1 andthe corresponding image point (c2, r2) in another imageI2, using basic projection equation of weak perspectivecamera model for planar 3D object points [18] yields

(c2 − cc2

r2 − rc2

)= M2 ∗ M1

−1

(c1 − cc1

r1 − rc1

)(1)

where M1 and M2 are the projection matrices for imageI1 and I2 respectively, and (cc1, rc1) and (cc2, rc2) are theprojection points of the same reference point (xc, yc, zc)in the image I1 and image I2. Equation 1 is the funda-mental weak perspective homographic projection equa-tion that relates image projections of the same 3D pointsin two images with different face poses. The homographicmatrix P = M2 ∗ M1

−1 characterizes the relative orien-tation between the two face poses.

From equation 1, if we have one face image and itscorresponding pose matrix, we can theoretically recon-struct all the other different face view images, which willhave different pose matrices.

The face pose can be characterized by a rotation ma-trix R resulted from successive Euler rotations of thecamera frame around its X axis by ω, its once rotated Yaxis by φ, and its twice rotated Z axis by κ [18]. Elementsof the projection matrix is related with these three an-gles and a scale factor λ, representing the distance fromface to camera. The 3D pose of a face can therefore becharacterized by the three Euler angles and the scalerλ. While the three angles determine face orientation, λdetermines the distance from face to camera. Face poseestimation can be expressed as determination of thesefour parameters (ω, φ, κ, λ).

4 Simultaneous 3D Face Pose Determi-nation and 2D Face Tracking

We propose a novel technique to track 3D face poseand 2D face in the image synchronously as follows.

4.1 Automatic 3D Face Model and PoseInitialization

In our algorithm, we should have a fronto-parallel faceto represent the initial face model. This initializationis automatically accomplished by using the eye trackingtechnique we have developed. Specifically, the subjectstarts in fronto-parallel face pose position with the facefacing directly to the camera as shown in Figure 2. Theeye tracking technique is then activated to detect eyes.After detecting the eyes, the first step is to compute thedistance deyes between two eyes. Then, the distance be-tween the detected eyes, eyes locations and the anthro-pometric proportions are used to estimate the scope andthe location of the face in the image automatically. Ex-periments show that our face detection method workswell for all the faces we tested. Example of the detectedfrontal face region is shown in Figure 2. Once the face re-gion is decided, we will treat it as our initial face model,whose pose parameters are used as initial face pose pa-rameters.

C

rX

Y

Z

Crop the face viewand treat it as the

initial 3D face model

Figure 2. The initial face model

The face pose parameters (ω, φ, κ, λ) for the initial facemodel are (0◦, 0◦, 0◦, 1), where λ, without loss of gener-ality, is normalized to 1. The corresponding projectionmatrix M is:

M =(

1 00 1

)(2)

We use the center of the 3D face model as the referencepoint, which is represented as (xc, yc, zc). Furthermore,we assume its corresponding projection image point is(cc, rc) in the row-column image plane.

Compared with the existing frontal face detectionmethods, ours takes full advantage of the detected eyesto guide the detection of the frontal face, and it is sim-ple, robust and automatic. In section 5, we will alsodemonstrate the tolerance of our face initialization toslight deviations from the frontal-parallel face pose andto perturbations of initial positions of the face.

4.2 Face Pose Tracking Algorithm4.2.1 Kalman Filter with Eye ConstraintsGiven the initial face image and its pose in the first frame,the task of finding the face location and the face pose insubsequent frames can be implemented as simultaneous3D face pose tracking and face detection.


Given the 2D face model obtained from the ini-tialization, the current face pose parameters Xt =(ωt, φt, κt, λt) and the current face image location(xt, yt), the Kalman Filtering based face pose trackingconsists of the following steps.

1. Combined Prediction

Let the state vector at time t be represented as Xt =(ωt φt κt λt xt yt)t. According to the theory ofKalman Filtering [19], the system can therefore bemodelled as

Xt+1 = ΦXt + wt (3)

where Φ is the state transition matrix, and wt sys-tem perturbation.

Given the system model, X−t+1, the state vector at

t + 1, can be predicted by

X−t+1 = ΦXt + wt (4)

along with its covariance matrix Σ−t+1 to characterize

its uncertainty.

The prediction based on Kalman Filtering assumessmooth face movement. The prediction will be offsignificantly if head undergoes a sudden rapid move-ment. In dealing with the problem, we propose toapproximate the face movement with eyes movementsince eyes can be reliably detected in each frame. Letthe predicted face pose vector at t+1 based on eyesmotion be Xp

t+1. Then the final predicted face poseshould be based on combining the one from Kalmanwith the one from eyes, i.e.,

X∗t+1 = X−

t+1 + Σ−1t+1(X

−t+1 − Xp

t+1) (5)

The simultaneous use of Kalman Filtering and eyesmotion allows to perform accurate face pose predic-tion even under significant and rapid head move-ments. We can then derive a new covariance matrixΣ∗

t+1 for X∗t+1 using the above equation to charac-

terize its uncertainty.

2. Detection

Given the predicted state vector X∗t+1 at time t + 1

and the prediction uncertainty Σ∗t+1, we can perform

a face pose estimation via face detection verification.Specifically, X∗

t+1 and Σ∗t+1 form a local pose space

at time t+1. The pose search in this local pose spacewill lead to the measured pose zt+1. The search canbe formulated as a minimization problem to be de-tailed in section 4.2.2. Let the measurement modelin the form needed by the Kalman filtering be

zt+1 = HXt+1 + mt+1 (6)

where mt+1 represents measurement uncertainty,normally distributed as mt+1 ∼ N(0, S), where S

is measurement error covariance matrix. The ma-trix H relates the state Xt+1 to the measurementzt+1.

3. Face Pose Updating

Given the predicted face pose X∗t+1, its covariance

matrix Σ∗t+1 and the measured face pose zt+1, face

pose updating can be performed to derive the finalpose Xt+1

Xt+1 = X∗t+1 + Kt+1(zt+1 − HX∗

t+1) (7)

where Kt+1 is the Kalman gain matrix.

4. Update Face Model

If current face pose aspect is significantly differentfrom the face model aspect, the face model should beupdated. The face model for frame t + 2 is updatedbased on successful face pose estimation for framet + 1. Our study shows that it is important to up-date the face model dynamically to account for thesignificant aspect changes under different face ori-entations or facial expressions. Details on updatingstrategy will be discussed in Section 4.2.4.

4.2.2 Face Detection and Pose Estimation viaMatching Optimization

The combined prediction from Kalman Filtering providesthe predicted face position, 3D face pose, and the asso-ciated uncertainty Σ∗

t+1 for the next frame t + 1. Σ∗t+1

can be used to limit the search area for the face locationand face pose at time t+1. Face detection and face poseestimation are to search for a face position and 3D facepose within the scope determined by Σ∗

t+1 such that thedetected face view image can best match the projectedface view image under the given face pose.

Mathematically, this is formulated as follows. Findthe state vector zt+1 within the scope determined byΣ∗

t+1 from X∗t+1 such that the detected face image best

matches the projected face image. We formulate thematching criterion as Sum of Squared Difference errorsover all the image pixels within the region of interest:

Ematching =N∑

i=1

(Ip(i) − Ic(i))2 (8)

where, Ip(i) is the pixel value of ith pixel in the recon-structed face view image Ip, Ic(i) the pixel value of theface image in the current image frame, and N the to-tal pixel number of the reconstructed face view image.Ip(i) is projected from the reference image Ir(c, r) via amapping function f

Ip = f(Ir, α)

where α = (ω, φ, κ, λ, x, y), which consists of the facepose parameters. We need to find the locally optimal


face pose parameter set α∗n, which results in the pro-

jected face image that best matches the face in the cur-rent image frame

α∗n = arg min

αEmatching

Ematching is minimized to solve for the 6 pose parame-ters, where x∗

t+1 serves as the initial value of α.

4.2.3 Regularization of Minimization Criteriaby Eyes Positions

Since different sets of pose parameters may produce thesame image, yielding the same Ematching, the minimiza-tion procedure may converge to a wrong place if it is leftunconstrained. This problem is further exacerbated bythe susceptibility of SSD to drifting. To overcome this, apenalty term is imposed to each SSD error correspondingto each set of pose parameters. Since we can accuratelyand independently detect the eyes position, the detectedeyes positions can be used to constrain the 3D face poseand the 2D face image.

For each pair of the projected eyes in the projectedface image and the detected eyes in the detected faceimage, the distance Eeyes between the detected eyes andthe projected eyes can be expressed as

Eeyes = ELeft + ERight (9)

where ELeft is the Euclidean distance between the de-tected left eye and the projected left eye, and ERight isthe Euclidean distance between the detected right eyeand the projected right eye. The correct face pose andface position should simultaneously minimize the resultsfrom equations 8 and 9. Therefore, the criteria for theimage matching minimization can be expressed as thesum of both terms

E = βEmatching + (1 − β)Eeyes (10)

where β is a scale factor determining the relative impor-tance of two terms. It is determined empirically.

4.2.4 Dynamic Face Model UpdatingIn practice, during tracking, the human face usually un-dergoes different kinds of variations, most of which comefrom the pose, expression, illumination and the combi-nation of them. In order for face templates to maintainadequate tracking performance while tracking face withtime-varying appearance, it is necessary to dynamicallyadapt the face model to keep them consistent with thechanging appearance of the face.

Specifically, our strategy for updating the face tem-plate is to include in each new face template with a por-tion of the initial template, which is assumed to havebeen chosen correctly, and thus contain the desired tar-get. For example, for the kth frame, once the best

match is found, with the matching confidence measure-ment above a certain threshold, the new face templateIr(k + 1) for frame k + 1 is updated to be the combina-tion of the initial face template Ir(0) and the face modelIg(k) generated from the image region Ic(k) in the newframe that best matches the current template as follows:

Ir(k + 1) = ρIr(0) + (1 − ρ)Ig(k)

where Ig(k) = g(Ic(k), α), and α is the detected facepose parameters related with frame k and function g isthe inverse projection function of equation 9.

Furthermore, the constant ρ ∈ [0, 1] determines thecontribution of the initial template to the new template.ρ = 0 is the case of fully updated templates and ρ = 1gives standard templates.

By the dynamic face model updating and physicaleye locations constraints, our tracker greatly improve thetracking accuracy.

4.3 Tracking Failure Detection and Re-covery

Human eyes are the most significant features of theface, therefore, the location of the human’s face can bedetermined once two eyes are detected [20]. Further, eyesfrom the same face are likely to blink at the same timeand with the same frequency. They also move rigidlywith the head, and the expected size of the face can beestimated from the distance between two eyes and thelocation of the face can be estimated from the eyes posi-tions. Therefore, the eye’s motion and position informa-tion can be combined easily to tell whether the face posetracker fails or not. Once the face’s motion and the eye’smotion or the face’s position and the eye’s position areinconsistent with each other, we can conclude that thetracker fails.

Once the face pose tracker fails, a backup techniqueis proposed to recover automatically. First, based on theeye locations in the image, the new face center is es-timated according to some anthropometric proportions.Second, 22 fiducial facial feature points are easily de-tected via the eye constrained facial feature tracker pro-posed in [21]. Based on the detected facial features, aset of rough face pose parameters can be estimated viathe feature-based technique [6]. Finally, based on thesenew roughly estimated face pose parameters, they arefurther refined by a local search over the face pose pa-rameter space to minimize the matching criteria 10.

Based on the above techniques, our face pose trackercan automatically detect the tracking failures and reini-tialize the tracker when the tracker is lost.

5 Experimental Results

A series of experiments involving real image sequencesare conducted to characterize the performance of our face


pose estimation technique. First, the accuracy of the es-timated face pose is analyzed. Then, we study the sensi-tivity of our algorithm to perturbations with initial facepose and placement. Further we demonstrate the effec-tiveness of face model updating for accurate face posetracking.

5.1 Accuracy of the Face Pose Tracker

In order to measure the accuracy of the estimated facepose, several image sequences were collected. The groundtruth for these sequences was simultaneously collectedvia ”Flock of Birds” magnetic tracker.

In Figure 3, two estimated face pose angles, Pitch andYaw, are shown together with the ground truth. Theestimation error for Yaw is 2.9260 degrees and the esti-mation error for Pitch is 3.6174 degrees.

Figure 3. Comparison between the estimated facepose and the ground truth. In each graph, the dashedline depicts the ground truth and the solid line depictsthe estimated face pose.

5.2 Sensitivity of the Face Pose TrackerTwo experiments are conducted to test sensitivity of

the tracker to the initial face size and the initial facepose for the face model separately. One sequence thatcontains a person rotating his face naturally in frontof the camera is chosen for this experiment. First, weperturb the size of face, which is represented by theface width, by ±5 and ±10 percent of the estimated

face width. Figure 4 shows some tracking results corre-sponding to the different variations of different face sizes.We can see the tracking trajectories basically follow thesame trend though there are some small deviations atcertain frames. Second, we simulate the perturbationsto the frontal-parallel pose of the face model by choos-ing the initial face model from the image frame, whichcontains the face under the non-frontal-parallel pose.Then the tracking is started from that image frame. Wechoose frames 1, 8, 12 and 14 as the starting framefor tracking respectively. They roughly correspond tothe following face poses: (0◦, 0◦, 0◦), (−0.4◦,−0.5◦,−1◦),(−8.8◦,−10.9◦,−2◦) and (−11.7◦,−12.8◦,−2.7◦). Fig-ure 4 shows some tracking results.

We can conclude that the tracker is not very sensitiveto slight variations of initial face poses.

5.3 Face Model Updating

The initial face model is obtained from a frontal-parallel face image. When there are significant face ap-pearance changes, either caused by face rotations, light-ing changes, face expression, or the combinations ofthem, the frontal-parallel face model is not suitable formeasuring the similarity between these images and theface model any more. Figure 5 shows that the face posetracker will fail to track the face pose due to the signif-icant face appearance changes when no face model up-dating is involved.

The face model is updated when significant aspectchange has occurred between the face model and currentface pose. Face model updating allows to successfullytrack face poses, which the tracker without face modelupdating will fail previously as demonstrated in Figure5.

5.4 Convergence and SpeedIn our implementation, tracking speed averages about

10 frames a second using the Powell minimizationmethod in a computer with an Intel Pentium 4 pro-cessor and 512M memory. This performance num-ber includes the time needed to extract images fromthe video sequence and save them back into the harddisk. Further speed gain can be obtained with afaster computer and with additional optimization of thecodes. Tracking results of our system are shown in Fig-ure 6. Video demos of our system may be found athttp://www.ecse.rpi.edu/∼cvrl/Demo/demo.html.

6 Advantages

Unlike conventional face pose estimation techniques,which often perform face detection and face pose esti-mation separately, our technique performs both concur-rently, allowing one to complement another. Tracking3D pose using Kalman Filtering effectively converts the


Pitch rotation Yaw rotation Roll rotation X translation Y translation Scale factor λ

Figure 4. (a) First row: sensitivity of the tracker to errors in estimating size of the initial face model. The face size isperturbed by ±5 and ±10 percent off the estimated face width. (b) Second row: sensitivity of the tracker to errors in theinitial face pose of face model. In each graph, the curves correspond to image frames 1, 8, 12 and 14.

Frame 64 Frame 68 Frame 72 Frame 74 Frame 78

Figure 5. When significant face aspects occur, the face pose tracker without model updating will fail as shown in toprow images. The proposed pose tracker still can track the face poses successfully under these circumstances, as shown inthe images in the bottom row. In all images, white lines represent the face norms, as determined by the estimated poses.The two small circles represent the detected eyes, and the solid circles represent the projected eyes from the face modelusing the estimated pose parameters.

conventional inverse face pose estimation problem to aforward problem, therefore leading to a unique solution.Our method does not require tracking facial features. Itperforms face pose tracking from a monocular uncali-brated camera.

The pure planar tracker can accurately estimate theposition of head, but tends to lose the target as soonas there is some significant out of plane rotations. Thisdrift issue can be successfully handled by the detectedeyes information. Our eyes detection and tracking com-bine active IR with mean-shift, therefore, it is very robustunder various face rotations and illuminations. The de-tected locations of eyes can therefore constrain the facelocation in the image successfully. Therefore, our tech-nique overcomes problems with the existing face poseestimation techniques based on planar models by allow-ing significantly out-of-plane rotations. In the meantime,our technique preserves its simplicity as contrasted withsome other more complex face models.

Our face pose tracker is based on minimizing the sum-of-squared (SSD) between the projected face view imageand the detected face view image in the input image. Asreported in [22, 23, 24, 25], pure SSD based region track-ing is sensitive to the appearance changes. Therefore, wehave to account for the face appearance changes causedby face pose, facial expression, illumination and the com-binations of them. The need to update face model dy-namically is to ensure that the current face model is com-patible with current face aspect. This is another criticalpoint to the success of our system, which leads to ro-bust and accurate tracking even under significant facialexpression changes, large out-of-plane rotation and illu-mination changes.

Further, the use of IR sensing does not limit the mo-bility of the person. Most existing 3D face pose trackersare for stationary people (e.g. people in front of the com-puter) instead of a mobile person. So, in this regard, oursystem is applicable and is more robust because of IR


Figure 6. Face and face pose tracking result images taken from second video sequence from experiments, which arerandomly selected. The white rectangle indicates the tracked face region and the white line is the norm of the face planewhich is drawn according to those three angles.

sensing.Simply, the benefits of our approach include: 1) work-

ing with a single uncalibrated camera; 2) allowing signif-icant out-of-plane face rotations; 3) no need to track anyfacial features, 4) automatic face detection; 5) not verysensitive to face appearance changes introduced by pose,expression, illumination and the combination of them; 6)automatic failure recovery.

7 Conclusions

In this paper,we present a technique for simultaneous3D face pose tracking and face detection under differ-ent face orientations, facial expressions and illuminationchanges in real time. Experimental results show howour technique greatly improves the standard pure planarface tracker, but still enjoys its simplicity. Based on theproposed techniques, we have built a real time workingsystem that will start to track the face and estimate the3D face pose as soon as the user is sitting in front ofthe camera. Our system is robust and accurate enoughto track the face and estimate face pose, and it is suit-able for the vision based applications such as HCI, facerecognition and virtual reality.

References

[1] S. McKenna and S. Gong, “Real-time face pose estimation,”Int. J. Real Time Imaging, Special Issue on Visual Monitor-ing and Inspection, vol. 4, pp. 333–347, 1998.

[2] M. Motwani and Q. Ji, “3d face pose discrimination usingwavelets,” in ICIP, October 7-10,2001.

[3] S. Nayar, H. Murase, and S. Nene, “Parametric appearancerepesentation,” in In Early Viual Learning. Oxford UniversityPress, 1996.

[4] J. Huang, X. Shao, and H. Wechsler, “Face pose discrimina-tion using support vector machines (svm),” in ICPR, 1998.

[5] N. Kruger, M. Potzsch, , and C. v.d. Malsburg, “Determina-tion of face positions and pose with a learned representationbased on labeled graphs,” Image and Vision Computing, Au-gust 1997.

[6] P. Yao, G. Evans, and A. Calway, “Using affine correspon-dence to estimate 3-d facial pose,” in ICIP, 2001, pp. 919–922.

[7] A. Gee and R. Cipolla, “Determining the gaze of faces inimages,” Image and Vision Computing, vol. 12, pp. 639–948,1994.

[8] T. Horprasert, Y. Yacoob, and L.S. Davis, “Computing 3-d head orientation from a monocular image sequence,” inInternational Conference on AFGR, 1996.

[9] Q. Ji, “3d face pose estimation and tracking from a monocularcamera,” Image and vision computing, 2002.

[10] D. DeCarlo and D. Metaxas, “The integration of optical flowand deformable models with applications to human face shapeand motion estimation,” CVPR, 1996.

[11] T. S. Jebara and A. Pentland, “Parameterized structure frommotion for 3d adaptive feedback tracking of faces,” CVPR,1997.

[12] A. Schdl, A. Haro, and I. Essa, “Head tracking using a tex-tured polygonal model,” Proceedings Workshop on Perc. UserInterfaces, 1998.

[13] S. Basu, I. Essa, and A. Pentland, “Motion regularization formodel-based head tracking,” ICPR, 1996.

[14] A. Azarbayejani, T. Starner, B. Horowitz, and A.Pentland,“Visually controlled graphics,” PAMI, 1993.

[15] M. La Cascia, S. Sclaroff, and V. Athitsos, “Fast, reliable headtracking under varying illumination: An approach based onrobust registration of texture-mapped 3d models,” PAMI,2000.

[16] L. Brown, “3d head tracking using motion adaptive texture-mapping,” CVPR, 2001.

[17] Sami Romdhani and Thomas Vetter, “Efficient, robust andaccurate fitting of a 3d morphable model,” International Con-ference in Computer Vision, 2003.

[18] E. Trucco and A. Verri, “Introductory techniques for 3-dcomputer vision,” Prentice-Hall, New Jersey, 1998.

[19] Kalman, Rudolph, and Emil, “A new approach to linear fil-tering and prediction problems,” Transactions of the ASME–Journal of Basic Engineering, vol. 82, no. Series D, pp. 35–45,1960.

[20] Carlos H. Morimoto and Myron Flicker, “Real-time multipleface detection using active illumination,” IEEE InternationalConference on AFGR, Grenoble, France, March 2000.

[21] Haisong Gu, Qiang Ji, and Zhiwei Zhu, “Active facial trackingfor fatigue detection,” 2002.

[22] Gregory D. Hager and Peter N. Belhumeur, “Efficient regiontracking with parametric models of geometry and illumina-tion,” PAMI, Vol. 20, No. 10, October 1998.

[23] J. Shi and C. Tomisito, “Good features to track,” CVPR,1994.

[24] N. P. Papanikolopoulos, “selection of features and evaluationof visual measurements during robotic visual servoing tasks,”Journal of Intelligent and Robotic System, 13(3), 279-304,1995.

[25] P. Anandan, “A computational framework and an algorithmfor the measurement of visual motion,” IJCV, 2(3), 283-310,1989.


Documents

Real Time 3D Face Pose Tracking From an Uncalibrated Camera