Recognition of Blurred Faces Using Local Phase Quantization

Timo Ahonen, Esa Rahtu, Ville Ojansivu, Janne HeikkilaMachine Vision Group, University of Oulu, PL 4500, FI-90014 Oulun yliopisto, FINLAND

{tahonen,erahtu,vpo,jth}@ee.oulu.fi

Abstract

In this paper, recognition of blurred faces using therecently introduced Local Phase Quantization (LPQ)operator is proposed. LPQ is based on quantizing theFourier transform phase in local neighborhoods. Thephase can be shown to be a blur invariant property un-der certain commonly fulfilled conditions. In face imageanalysis, histograms of LPQ labels computed within lo-cal regions are used as a face descriptor similarly tothe widely used Local Binary Pattern (LBP) methodol-ogy for face image description. The experimental re-sults on CMU PIE and FRGC 1.0.4 datasets show thatthe LPQ descriptor is highly tolerant to blur but stillvery descriptive outperforming LBP both with blurredand sharp images.

1. Introduction

In the last years, different problems in face imageanalysis, such as face detection, face recognition andfacial expression recognition have received very muchattention in computer vision research. These problemsare interesting from the viewpoints of basic researchaiming to efficient descriptors for facial images and ofapplications such as surveillance and human-computerinteraction [2].

A key issue in face analysis is finding efficient de-scriptors for face appearance. Given the low inter-person variation in face images, ideal descriptors shouldbe very discriminative. Still, they should be robust todifferent perturbations and changes such as illuminationand pose changes, aging of the subjects, etc.

Despite the extensive research efforts towards facedescriptors robust to the aforementioned disturbances,the problems caused by blur often present in the real-world face images have been mostly overlooked. Blurmay be present in face images due to motion of the sub-ject or the camera during the exposure, camera not be-ing in focus, or low quality of the imaging device such

as analog web camera. For example, in the Face Recog-nition Grand Challenge dataset [5], many of the “uncon-trolled still” images appear blurred because the auto-focus of the digital pocket camera did not focus into theface area.

Until now, there has been little work on blur invari-ant face descriptors. Facial image deblurring for recog-nition has been addressed in a few publications, see [8]and references therein, but to the best of authors’ knowl-edge, explicitly constructing blur-invariant descriptorsfor face recognition has not been proposed before.

In this work we address the challenges caused byblur by applying the recently proposed blur tolerant Lo-cal Phase Quantization method to face description.

2. Face description using local phase quan-tization

The Local Phase Quantization operator was origi-nally proposed by Ojansivu and Heikkila for texture de-scription [4]. The operator was shown to be robust toblur and outperform the Local Binary Pattern operator[3] in texture classification. In this work, we proposeconstructing a face descriptor using the LPQ operator.

2.1. Local phase quantization

The spatial blurring is represented by a convolutionbetween the image intensity and a point spread func-tion (PSF). In the frequency domain, this results in amultiplication G = F · H , where G, F , and H are theFourier transforms of the blurred image, the original im-age, and the PSF respectively. Further considering onlythe phase of the spectrum, the relation turns into a sum� G = � F + � H .

If the PSF is centrally symmetric, the transform Hbecomes real valued and the phase angle � H must equal0 or π. Furthermore, the shape of H for a regular PSFis close to a Gaussian or a sinc-function, which makesat least the low frequency values of H to be positive.At these frequencies, � H = 0 causing � F to be a blur

978-1-4244-2175-6/08/$25.00 ©2008 IEEE

invariant property. This phenomenon is the basis of thelocal phase quantization (LPQ) method described in thefollowing.

In LPQ the phase is examined in local M -by-Mneighborhoods Nx at each pixel position x of the imagef(x). These local spectra are computed using a short-term Fourier transform defined by

F (u,x) =∑

y∈Nx

f(x − y)e−j2πuT y. (1)

The transform (1) is efficiently evaluated for all imagepositions x ∈ {x1,x2, . . . ,xN} using simply 1-D con-volutions for the rows and columns successively.

The local Fourier coefficients are computed at fourfrequency points u1 = [a, 0]T , u2 = [0, a]T , u3 =[a, a]T , and u4 = [a,−a]T , where a is a sufficientlysmall scalar to satisfy H(ui) > 0. For each pixel posi-tion this results in a vector

Fx = [F (u1,x), F (u2,x), F (u3,x), F (u4,x)].

The phase information in the Fourier coefficients isrecorded by observing the signs of the real and imagi-nary parts of each component in Fx. This is done byusing a simple scalar quantizer

qj(x) ={

1 , if gj(x) ≥ 00, otherwise , (2)

where gj(x) is the jth component of the vector Gx =[Re{Fx}, Im{Fx}].

The resulting eight binary coefficients qj(x) are rep-resented as integer values between 0-255 using binarycoding

fLPQ(x) =8∑

j=1

qj(x)2j−1.

As a result, we get the label image fLPQ whose valuesare the blur invariant LPQ labels.

It can be shown that in quantization the informationis maximally preserved if the samples to be quantizedare statistically independent. Now in the case of real im-ages the neighboring pixels are highly correlated, whichwill result in dependency between the Fourier coeffi-cients gj quantized in LPQ. We can however improvethe situation by adding a simple decorrelation to LPQ.

Assume that the image function f(x) is a result of afirst-order Markov process, where the correlation coef-ficient between adjacent pixel values is ρ, and the vari-ance of each sample is 1. As a result the covariancebetween positions xi and xj becomes σij = ρ||xi−xj ||,where || · || denotes L2 norm. With this information onecan easily construct the covariance matrix C of all M2

samples in the neighborhood Nx.

Rewrite the expression (1) as F (u,x) = wTu fx,

where wu is a vector containing the complex exponen-tials at frequency u, and fx is another vector containingthe values of f in Nx. Stacking the vectors wui

into amatrix we have

W = [Re{wu1 ,wu2 ,wu3 ,wu4},Im{wu1 ,wu2 ,wu3 ,wu4}]T ,

so that Gx = Wfx. Hence the covariance matrix D ofGx can be obtained from D = WCWT .

The covariance matrix D can be now used to derivea whitening transform Ex = VT Gx, where V is an or-thogonal matrix derived from a singular value decom-position as D = UΣVT .

The values in Ex are uncorrelated, which assumingGaussian distributions is equal to independence. Us-ing Ex in the quantization (2) instead of Gx resultsin more information to be stored in most cases, eventhough the correlation model or the Gaussian assump-tion would not hold exactly. It can be also shown thatsince decorrelation is purely a rotation of the vector Gx,it does not affect to the blur invariance. Note also thatV can be solved in advance for a fixed ρ.

2.2. LPQ face descriptor

To use local phase quantization for face description,we apply the procedure of [1]. The face image is firstlabeled with the LPQ operator. Then, the label imageis divided into non-overlapping rectangular regions ofequal size and a histogram of labels is computed inde-pendently within each region. Finally, the histogramsfrom different regions are concatenated to build a globaldescription of the face.

3. Experiments

The efficiency of the proposed face descriptor in theface recognition scenario was tested using two datasets,the CMU PIE (Pose, Illumination, and Expression)dataset [6] and the Face Recognition Grand Challenge(FRGC) experiment 1.0.4 [5].

With both datasets, the images were first converted togray-scale and then registered using given ground trutheye positions. In the CMU PIE experiment, the imageswere cropped to size 128× 128 pixels and in the FRGCexperiment the image size was 130 × 150 pixels.

In both the experiments, the local phase quantiza-tion operator was compared to LBP, which is one of thestate-of-the-art methods in facial image analysis. Moreprecisely, the LBPu2

8,2 operator [3, 1] was used as a con-trol method. The parameters for the LPQ operator wereM = 7, a = 1/7 and ρ = 0.9.

Figure 1. Example images from the CMUPIE database with increasing artificialblur. From left: no blur, σ = 1, σ = 2

.

The LBP and LPQ histograms were extracted in-dependently from the non-overlapping rectangular re-gions of size of 8 × 8 pixels and then concatenated tobuild a global description of the face. Recognition wasperformed with a nearest neighbor classifier with Chisquare distance using the obtained histograms as fea-ture vectors.

In the first experiment, the performance of the LPQface descriptor was tested on images with artificiallycaused blur. The widely used CMU PIE database [6]was selected to serve as test material in this experi-ment. The images in the PIE database contain system-atic lighting variation so it is possible to observe thejoint effect of lighting changes and blur. For our exper-iments, we selected a set of 23 images of each of the 68subjects in the database. Two of these are taken with theroom lights on and the remaining 21 each with a flashat varying positions.

One sharp image per person was used as a galleryimage, and the remaining 22 images were artificiallyblurred by convolving them with a Gaussian blur maskwith σ = {0, 0.25, . . . , 2}. These 68 × 22 blurred im-ages were then used as probes for testing the methods.Examples of original and blurred images from the PIEdataset are shown in Fig. 1. The experiment was re-peated with 10000 random partitions into gallery andprobe data and the mean and standard deviations overthe tests were used to assess the performance.

The mean recognition rates with standard deviationsfor LPQ and LBPu2

8,2 are plotted in Fig. 2. As it canbe seen in the results, LPQ produces better recognitionresult than LBP even with no blur (99.2 % vs. 92.7%). The LBP descriptor tolerates slight blur very wellbut as blur increases from σ = 0.5, the recognition rate

0 0.5 1 1.5 20.65

0.7

0.75

0.8

0.85

0.9

0.95

1

σ of the Gaussian blur

Rec

ogni

tion

rate

LPQLBP

Figure 2. Recognition rates on the CMUPIE images with increasing Gaussian blur.

drops rapidly. At σ = 2.0, the achieved recognitionrate is 70.1 %. Local phase quantization, on the otherhand, tolerates increasing blur significantly better. Atσ = 1.0, the recognition rate is 98.6 %, after which therate starts to decrease slightly faster, but even at σ = 2.0the recognition rate still is 93.5 %, higher than usingLBP on images with no blur.

In the second experiment we compared the descrip-tors inn the FRGC experiment 1.0.4. That dataset is di-vided into training, probe and gallery sets which haveno overlap. The probe and gallery images represent152 subjects, and there are 1 gallery image and 2–7probes per subject, totaling 152 images in the galleryset and 608 images in the probe set. The gallery imageswere taken with good quality camera under controlledconditions whereas the probe images were taken witha pocket digital camera under uncontrolled conditions.Examples of the gallery and probe images are shownin Fig. 3. As can be seen in the sample images, theuncontrolled probes contain variation of lighting, facialexpression, and blur. That the probes contain significantout-of-focus blur makes this dataset very interesting forcomparing LPQ and LBP.

To reduce the effect of lighting variations in facialimages, Tan and Triggs proposed an illumination nor-malization procedure consisting of gamma correction,difference of Gaussian filtering, and contrast equaliza-tion [7]. In their experiments, they showed that thismethod significantly increases the recognition results ofLBP based face description on the FRGC dataset. Theyalso proposed a variant of LBP named Local TernaryPatterns and a Distance Transform to replace the his-togram based approach.

Figure 3. Example images from the FRGC1.0.4 dataset. From left: gallery image andtwo probe images.

Table 1. Recognition rates on the FRGC1.0.4 dataset.

Without WithMethod preproc. preproc.LTP+DT [7] 68.4LBPu2

8,2 32.6 64.3LPQ 45.9 74.5

In our experiments on the FRGC 1.0.4 dataset we ap-plied the illumination normalization preprocessing pro-posed by Tan and Triggs. Moreover, we report the re-sults obtained with Local Ternary Patterns and DistanceTransform as another control method. The preprocess-ing and control results were computed using implemen-tations made available by Tan and Triggs. Note thatthese figures differ from those reported in [7]. This isdue to the different experimental setup that was utilizedin [7]. In that work several gallery images per subjectwere used.

The recognition results in the FRGC experiment1.0.4 are reported in Table 1. The results without pre-processing show that the LPQ operator handles the blurand other variation in the probe set better than LBP.The lighting changes still pose a challenge also to LPQ.However, as illumination normalizing preprocessing isintroduced, the recognition rate of both the methods in-crease notably. With preprocessing, local phase quanti-zation reaches clearly higher recognition rate than localbinary pattern or local ternary pattern.

4 Discussion and conclusions

In this paper we proposed local phase quantizationoperator for face recognition. Prior to this work, the ef-fectiveness of this blur insensitive operator was experi-mentally shown on texture images without blur or withartificial blur. Here we analyzed the applicability of the

operator for the very challenging task of face recogni-tion and showed that it reaches higher recognition ratesthan the widely used local binary pattern operator.

The tests on the CMU PIE database with artificiallycaused Gaussian blur showed that the LPQ operator isvery tolerant to blur compared to LBP. Most interest-ingly, its recognition rate even with no blur was bet-ter than that of LBP. In the FRGC Experiment 1.0.4,the LPQ operator clearly outperformed both the controlmethods, LBP and LTP. This shows that it is very ro-bust not only to blur but also to other challenges such aslighting and facial expression variations present in real-world images. The operator is also simple to implementand fast to compute requiring only image convolutionswith small separable kernels and vector rotations. De-spite its simpleness and robustness, LPQ was shown tobe highly discriminative producing very good recogni-tion results.

In this work, the LPQ descriptor was applied to theface recognition problem. However, we believe that thissame methodology is applicable to other face descrip-tion tasks as well as to local image description in otherproblem domains.Acknowledgement. This work has been performed

with support from the MOBIO project, which is a7th Framework Research Programme of the EuropeanUnion.

References

[1] T. Ahonen, A. Hadid, and M. Pietikainen. Face descrip-tion with local binary patterns: Application to face recog-nition. IEEE T-PAMI, 28(12):2037–2041, 2006.

[2] S. Z. Li and A. K. Jain, editors. Handbook of Face Recog-nition. Springer, 2005.

[3] T. Ojala, M. Pietikainen, and T. Maenpaa. Multireso-lution gray-scale and rotation invariant texture classifica-tion with local binary patterns. IEEE T-PAMI, 24(7):971–987, Jul 2002.

[4] V. Ojansivu and J. Heikkila. Blur insensitive texture clas-sification using local phase quantization. In ICISP 2008.

[5] P. J. Phillips, P. J. Flynn, T. Scruggs, K. W. Bowyer,J. Chang, K. Hoffman, J. Marques, J. Min, and W. Worek.Overview of the face recognition grand challenge. InCVPR 05, pages I: 947–954, 2005.

[6] T. Sim, S. Baker, and M. Bsat. The CMU pose, il-lumination, and expression database. IEEE T-PAMI,25(12):1615–1618, 2003.

[7] X. Tan and B. Triggs. Enhanced local texture feature setsfor face recognition under difficult lighting conditions. InAMFG 2007.

[8] Y. Yao, B. R. Abidi, N. D. Kalka, N. A. Schmid, andM. A. Abidi. Improving long range and high magnifica-tion face recognition: Database acquisition, evaluation,and enhancement. Computer Vision and Image Under-standing, 111(2):111–125, 2008.