Learning Race from Face: A Survey

0162-8828 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TPAMI.2014.2321570, IEEE Transactions on Pattern Analysis and Machine Intelligence

1

Race Classification from Face: A SurveySiyao Fu, Member, IEEE, Haibo He, Senior Memeber, IEEE, and Zeng-Guang Hou, Senior Member, IEEE

Abstract—Faces convey a wealth of social signals, including race, expression, identity, age and gender, all of which have attracted increasingattention from multi-disciplinary research, such as psychology, neuroscience, computer science, to name a few. Derived from rapid advancesin computer vision and machine intelligence, computationally intelligent race analysis via face has been particularly prevalent recently becauseof its explosively emerging real-world applications, such as security and defense, surveillance, human computer interface (HCI). These studiesraise an important problem: How implicit, non-declarative racial category can be conceptually modeled and quantitatively inferred from the face?Nevertheless, race classification is challenging due to its ambiguity and complexity depending on context and criteria. To address this issue, recently,significant efforts have been reported toward race detection and categorization in the community. This survey provides a comprehensive and criticalreview of the state-of-the-art advances in face-race perception, principles, algorithms, and applications. We first discuss race perception problemformulation and motivation, while highlighting the conceptual potentials of racial face processing. Next, taxonomy of feature representational models,algorithms, performance and racial databases are presented with systematic discussions within the unified learning scenario. Finally, in order tostimulate future research in this field, we also highlight the major opportunities and challenges, as well as potential important cross-cutting themesand research directions for the issue of learning race from face.

Index Terms—Face recognition, race classification, image categorization, data clustering, face databases, machine learning.

F

1 INTRODUCTION

FACE explicitly provides the most direct and quickestway for evaluating implicit critical social information.

For instance, face could convey a wide range of semanticinformation, such as race1, gender, age, expressions, andidentity, to support decision making process at different levels.Behavior research in psychology also shows that encounteringa new individual, or facing a stimulus of human face normallyactivates three ”primitive” conscious neural evaluations: race,gender, and age, which have consequential effects for theperceiver and perceived [1], [2], [3], [4], [5], [6] (see Fig. 1).Among which, race is arguably the most prominent and dom-inant personal trait, which can be demonstrated empiricallyby its omnirelevance with a series of social cognitive andperceptual tasks (attitude, biased view, stereotype, emotion,belief, etc.). Furthermore, it yields deep insights into how toconceptualize culture and socialization in relation to individ-

• Siyao Fu is with the School of Information and Engineering, Minzu Universityof China, Beijing 100081, China. This work was finished when the author waswith the the University of Rhode Island. (E-mail: [email protected])

• Haibo He is with the Department of Electrical, Computer, and BiomedicalEngineering, University of Rhode Island. RI 02881, US. (Email:[email protected])

• Zeng-Guang Hou is with the State Key Laboratory of Management and Controlfor Complex Systems, Institute of Automation, Chinese Academy of Sciences,Beijing 100190, China. (Email:[email protected])

1. In general English the term “race” and “ethnicity” are often used asthough they were synonymous. However, they are related to biologicaland sociological factors respectively. Generally, race refers to a person’sphysical appearance, while ethnicity is more viewed as a culture concept,relating to nationality, rituals or cultural heritages. For example, detecting anEastern Asian from Caucasian crowds is race recognition task, while visuallydifferentiating a German and a French certainty belongs to ethnic categoryand thus requires extra ethnographically discriminative cues including dress,manner, gait, dialect, among others. Therefore, considering the soft biometriccharacteristics of race, we prefer to use “race” as more suitable categoryterminology in this article. Nevertheless, for some of the existing papers inliterature which have already used the term “ethnicity” but were indeedaddressing related “race” issues, we have also cited and discussed thosepapers as well, in order to provide a comprehensive and complete surveyon this topic.

Fig. 1: Illustration of face-race perception. The quick glimpse of thepicture will activate three “primitive” conscious evaluations of aperson: a young white female, although this figure was a computer-generated “artificial” face with a mix of several races. (Photo source:The New Face of America, TIME Magazine, November 18, 1993.)

ual appearance traits, including social categorization [7], [8],[9], association [2] and communication [10]. Therefore, theestimation of racial variance by descriptive approaches forpractical purposes is indeed indispensable in both social andcomputer science. However, while race demarcation drivesthe intrinsically genetic variation structure of essential facialregions to gather more explicit appearance information, thecore question emerges as the computational mechanism un-derlying this extraordinary complexity. This raises the follow-ing fundamental multi-disciplinary conundrum: How does acomputer model and categorize a racial face?

To answer this fundamental question, numerous researchconsortium and scholars have developed intensive investiga-



2tions from different angles. For example, psychologists havestudied behavior correlates of race perception such as other-race-effect (ORE) and attention model (e.g. [1], [7], [11], [12],[13]), which show existence of racially-discriminative facialfeatures such as eye corners or nose tip (further anthropomet-ric survey confirmed those areas help to discriminate racialgroups [14], [15], [16], [17], [18], [19]). Neurophysiologistshave shown how race perception influences and regulatescognitive processes such as affection [20], [21], [22], [23] andstereotype[24], [25]. Computational neuroscientist have builtmodels to simulate and explain race perceptions (e.g. [2],[26], [27], [28], [29]). Other cognitive experiments in [8], [9],[30] have also indicated the existence of racial features as asalient visual factor. Among the general public, the validity ofracial categories is often taken for granted. Converging linesof investigations provide a sterling evidence of perceptualdiscrimination relationship (albeit a rather qualitative one)between the implicit racial category model and the explicitface representation. Following these quantitative analysis,computer vision scholars have been motivated to tackle theinherent problem of racial demarcation to build computation-ally intelligent systems capable of categorizing races.

Yet, intuitively straightforward as it might seem to be, theimplicit underlying algorithm implementation tends to becomplicated and diversified. First, the nomenclature of raceis a perplexing picture, as being defined qualitatively ratherthan quantitatively, painted by diverse perspectives ratherthan unified aspects. The ambiguity of the definition leadsto the uncertainty of both problem formulation and develop-ment of methods, which differs it from other facial informa-tion analysis, such as face recognition (a specific one-to-onematching problem); facial expression (six universal emotionalstatus [31]); and gender (only two classes). What’s worse, thesensitivity to culture stereotype, racism and prejudice fuelsmuch of the troubles, which in turn makes corresponding datacollection and analysis difficult to carry on systematically. Thecontroversy raised by definition ambiguity and the invaliditycaused by data scarcity make computational classificationapproaches seldom work, which partially explains the scarcityof successful race learning methods and comprehensive per-spectives compared to other facial data processing facets.

In spite of this, derived by burgeoning advances in ap-plications such as security surveillance, human computerinteraction, bioinformatics, an increasing number of effortshave been reported toward race detection and categorizationin the community. However, despite overwhelming theoreti-cal principles, algorithms, applications and empirical results,there are relatively few crucial analysis that runs throughthese repertoire. Indeed, over the past few decades there havebeen several important work in face-based race perceptionavailable. Such as [32], [33] in psychology, and [34], [35] inneuroscience. However, facing fast development of cutting-edge techniques and algorithms, reviewing efforts towardracial feature extraction and analysis can not rely solely onthese existing works due to following reasons: First, subjectivecognitive experiments for testing human capability in racerecognition can not compare with automatic race detectionmethods. Second, it is well known that automatic race cat-egorization needs to be trained and generalized in a large-

scale database while subjects in traditional experiments couldonly face limited data displayed on screen. Third, emerginglatest 3D scan technology makes 3D facial fiducial data farmore convenient and available for computational recognitionapproaches rather than human perception, suggesting thatexisting survey papers may not be suitable for directing newcomputational research trends. Overall, the increasing contra-diction between overwhelming social/scientific demographicdata and scarce race mining methods calls for a comprehen-sive survey on computational race recognition and generaldiscussion to cover the state-of-the-art techniques and to givescholars a perspective of future advances.

Aimed at developing a unified, data-driven framework ofrace/ethnicity classification, the goal of this survey is to pro-vide a critical and comprehensive review of the major recentadvances in the research on race recognition. Since “race”is a rather loosely defined and ambiguous term linked withmultidisciplinary research areas, such a survey work wouldunavoidably be involved with a combination of both sole com-puter vision based analysis and psychological-physiologicalbehavior experimental observation results. Note that whilethe former is back bone of this survey, the empirical analysisand experimental support, however, come from the latter.In contrast to most previously published surveys in relatedfields, we focus mainly on the approaches that are capableof handling both computer vision and cognitive behaviorsanalysis. It is also noteworthy to be pointed out that forall face related research areas, there already exist severalrepresentative reviews and surveys, such as face recognition[36], [37], [38], expression recognition [1], [39], age estimation[40], [41], [42], and gender recognition [43], [42], even surveyof face databases [36], [44]. However, to our best knowledge,there is no such comprehensive survey on race recognition,which also motivates us to present a review for completingthe gap and enriching the research forefront.

The rest of this survey is organized as follows: Section 2 de-scribes race perception from multi-disciplinary perspectives,which serves motivations and foundations for race recog-nition. With these sketches of theoretical and applicationalopinions, analytical feature representation techniques of iden-tifying a racial face which enables us to design discriminativeclassification system are detailed in Section 3, which providesan overview of racially distinctive feature representation ap-proaches, such as chromatic feature, local and global featureextractions. Section 4 presents a comprehensive review of thestate-of-the-art research on race recognition approaches, in-cluding single-model/multi-model race recognition, 3D racialface analysis, and intra-ethnic recognition. In conjunction withreview of algorithms, we proceed to provide a summary andanalysis of major available racial face databases in Section5, specifically, we also provide several representative anthro-pometry survey data and racial face generation softwares. InSection 6, we highlight several representative applications ofrace recognition to demonstrate its critical potentials in manydomains, such as video surveillance, public safety, criminaljudgement and forensic art, and human computer interaction.In order to motivate future research along the topic, we furtherprovide insights on the challenges and opportunities of racerecognition in Section 7. Section 8 concludes the survey.



32 HUMAN RACE PERCEPTION FORMULATION

Given its both cognitive and practical purposes, race percep-tion is inherently a multidisciplinary issue involving differentresearch fields, including psychology, cognitive neuroscience,computer vision, among others. Therefore, the progress dur-ing the investigations is undoubtedly contingent on theprogress of the research in each of those fields. Our surveythus starts from the fundamental and analytical understand-ing of race based on interdisciplinary expertise.

2.1 The conceptual description of race

We begin by briefly introducing basic interpretation andconceptualizations of race which are crucial for providinginformation about the racial features of which recognitionalgorithms or systems are based on, thus, “What is race”?From the definition of Wiki encyclopedia, the term “race”is defined as follows [45]: “Race is a classification system,used to categorize humans into large and distinct populationsor groups by heritable, phenotypic characteristics, geographicancestry, physical appearance, ethnicity, and social status.”

Apparently, that definition is very “fuzzy”, so apt to per-petuate confusion and engender discord for developing racerecognition systems2. For the research purposes, it is worthhighlighting several important points of consensus that haveemerged from psychological-physiological research traditions:

Perhaps the most prominent, yet wrongly understood as-pect of race is the common trap of confusing race with skincolor. Although classification by using skin tones will simplifythe problem greatly (several attempts have used color toperform race classifications, such as [47], [48], [49]. In somesocieties and among some anthropologists, color terminologyhas been used to label races). However, skin color is sucha variable visual feature within any given race that it isactually one of the least important factors in distinguishingbetween races and consequently, should be taken with care inapplications [50], [51], [52], [53] (see Fig. 2).

Secondly, physical characteristics such as hairshaft mor-phologic characteristics and craniofacial measurements areviewed as significant indicators of ethnic belongings. How-ever, for computer vision methods, during preprocessing nor-mally a face mask will be applied to cut them off, leavingonly face region standing out, making those racial cues littleassistance in applications. While in visual surveillance, thosenon-face visual appearances could be employed as apparentvisual cues and provide salient features for tracking andrecognition [54].

Thirdly, in an effort to define a clear pattern classificationsystem, such aforementioned visible physical (phenotypic)variant features must be associated with large, geographicallyseparable populations (groups). Therefore, it is commonly

2. We would like to note that although most people think of races as simplephysically distinct populations, recent advances from bioinformatics, on theother hand, demonstrated that human physical variations do not fit an exactracial model, thus, there are “no genes that can identify distinct groups thataccord with the conventional race categories” [46]. In a rough nut shell, theconcept of race has no biological validity, which is a theoretically correctviewpoint that is conclusive but actually unpersuasive for computer visionbased classification systems. This survey focuses mainly on physical traitswhich represents basic racial groups rather than genetic ones.

Fig. 2: Illustrative example showing that skin tone is not the majordeterminant of perceived racial identity: observers look at a seriesof facial photos where a central face (racially different from thesurroundings) appeared lightened or darkened. The behavior resultshowed that there was no difference in ratings of race. (Figure source:[50])

acceptable that, in a rough real-world application sense, clas-sification systems designate these continental aggregates asrace labels, such as the European/Caucasian race, and theAsian/Mongolian race [55], [56], which is also the commonlyperceived categories of race. Note, however, that such groupscan range from 3 to more than 200, judged by the specificcriterion. However, for practical computer vision based raceclassification system we suggest that seven most commonlyencountered and accepted racial groups (African American,South Asian, East Asian, Caucasian, Indian, Arabian, andLatino, all these cover more than 85% world population) willbe enough. (see Fig. 3 for an ilustration)

Fig. 3: Illustrative genetic variance distribution of human races,while in practice it is often accepted that 3- to 7-races classificationsystem would be enough for regular applications. (Figure source:http://www.faceresearch.org/)

Motivated by the above plethora of aspects, the definition of



4race category in computer vision can be expressed as follows:Consider a dataset P of all possible facial images Ii, whichare assumed to belong to several pre-defined racial groupsG1 to Gn. A group Gi is called racialized if its memberscan be agglomerated by either explicit or implicit patterns(physical observations, such as skin tones or morphology, orfeature vectors, etc.) to be appropriate evidence of ancestrallinks to a certain geographical region3. Accordingly, the raceclassification/categorization issue can be expressed as an ex-ploratory image analysis from a statistical pattern recognitionperspective, that is, “to design a classification system whichis capable of determining or inferring these racial identitiesfrom given facial images with minimum ambiguity”.

2.2 Race and Other-Race Effect (ORE)Principally related with the race perceptual model is the socalled other-race-effect (ORE, also called “own race bias”,“cross-race effect”). It means given a face recognition task,human observers tend to perform significantly better withtheir own-race as compared to other-race faces (Fig. 4), aconcept with more than 40 years’ history which has beendemonstrated repeatedly in numerous meta-analysis [32],[33]. Though varied explanations have been proposed toaccount for ORE by ethographers, cognitive anthropologists,sociologists, and even policy and law enforcement officer[32], [58], the fundamental reason still remains elusive [59].Nevertheless, it is interesting to point that there lies somesubtle yet complicate connections between ORE and theirinfluence on automatic race categorization approaches. Forexample, O’Toole et al investigated the ORE effect for facerecognition algorithms [60]. They found that Western algo-rithms recognized Caucasian faces more accurately than EastAsian faces and the vice versa, indicating face recognitionalgorithms favor the “majority” race data in the database,just like human subjects show the standard ORE for face.Dailey et al also found the existence of in-group advantagefor seeking evidence and a computational explanation ofcultural differences in facial expression recognition [61]. Thetransduced conclusion is that ORE affects the training ofrecognition algorithm (similar to human socialization), thus abalanced database could be essential in the race categorizationin order to counter the biased results.

2.3 Race and physical anthropometryThe definition of race emphasizes its conceptualism in anphysical anthropology, which dates back even to Greek neo-classical canons (c. 450 B.C.) era. The born of anthropometry

3. Note that physical morphology and geography not only play key rolesas basis to justify the treatment in problem formulation, they also makethe definition flexible, which is in accordance with the fact that race is notrigidly defined as other traits such as gender or emotion. For example, giventhis definition, we can say a facial image Ii is of the White race if Whiteis a racialized group G1 in P , and Ii is a member. As Haslanger states:“Whether a group is racialized, and so how and whether is raced, is notan absolute fact, but will depend on hierarchical context”[57]. On this view,American Indians can be racialized and labeled both anthropometrically andalgorithmically insofar as there exists enough representative and annotatedimages in dataset, otherwise they could only be demarcated as Asian dueto subordinated hierarchy relationship. Nevertheless, they still belongs to adifferent “racial” group because of their distinctive features, even the exactdefinitions lack.

Fig. 4: Different attention regions for Western and Eastern subjects(eye tracking data) [62], [63], which can be grouped into nose-centric versus eye-centric gazing patterns. Biased attention may causeambiguity in affective perception.

implies that race could possibly be measured and classifiedbased on predefined standard criterion, with which a numberof methods have been devised for calculating racial types[64]. Among which, L. Farkas was the known pioneer forestablishing an index of 42 discriminative anthropometricfacial points (ranked by mean and standard deviation) froma consolidation list of 155 cranio-facial anthropometric pro-portions [65], [66]. More systematic review of inter-ethnicvariability in facial dimensions [67] and recent survey resultsin American [14], [15], [16] and Chinese demographic data[17], [18], [19] also confirmed statistically significant variancesin facial anthropometric dimensions between all race groups.The highlights of these variations in inter-ethnic phenotypicdifferences pave the way of anthropometry based automaticrace recognition.

However, the drawbacks of traditional anthropometric sur-vey sampling methods (incapable of digitalization) and data(usually univariate, unable to accurately describe 3D shape)prevent it from direct application. Fortunately, the availabilityof latest 3D scan techniques (laser scanners, CT scanner, Ultrasonic scanner, PET) is a welcome stimulus to this challengingtask, offering reliable and accurate digital head and face data.Nevertheless, a fundamental question is how to incorporatethose anthropometric data into computer vision based racerecognition approaches because anthropometric meaningfullandmarks vanish in 2D frontal images. The survey resultsin [68] implied that the combination of 2D and 3D imagesmight provide a feasible solution (Detailed in Section 4 and5).

2.4 Race and cognitive neuroscienceHow we perceive and categorize race and how social cate-gories of race are processed, evaluated have long been a cen-tral topic of cognitive neuroscientists. With the developmentof analysis tools (fMRI, ERP, EEG) [1], [11], [12], [13], [7],[34], [69], [70], [71], [72] and theoretical cortex models em-bedded with computationally intelligent algorithms [73], [74],[75], the research on race perception by fusing those multi-modality physiological data have been greatly expanded inrecent decades. For example, temporal ERP observation re-sults in [21], [76], [77] have shown that race based mod-ulation was activated as early as around 100ms, and raceeffects were reported repeatedly among many essential ERP



5

Fig. 5: Illustrative time course of face perception in human brain,recorded by ERP potential peaking. Race modulation was observedin N100 (mean peak latency around 120ms) and P200. Gendermodulation occurred in the P200 as well (mean peak latency around180 ms), N170 component indicates structural encoding of face (v.snon-face). P300s concern social category membership differed fromthe preceding individuals.

components, such as N1704, P200, N200 and P300 (see Fig. 5for illustration). The fast response to race label implies thatrace categorization might occurs at the very early stages ofperceptual face encoding and thus can significantly influenceface processing. Studies using fMRI and ERP to investigatethe neural system sensitive to race consistently report ac-tivation in an interconnected regions, including the amyg-dala [78], [79], [80], [81], [82], anterior/posterior cingulatecortex (ACC/PCC)[83], [77], ventral/dorsolateral/Medial pre-frontal cortex (VLPFC/DLPFC/MPFC), and fusiform gyrus(FFA)[84], [85], [25], [86]. Fig. 6 presents an illustrative exam-ple of hypothetical race processing cortical areas. These fMRIresults, coupled with the ERP data, confirm the key role raceplays in a series of individual and social processing, such asstereotype [25], [24], prejudice [24], [87], [88], emotional andaffective regulation [89], [90], [91], [63], [92], personality [93],and other behavior and decision making applications [20],[21], [22], [23]. The immediate implication of these studiesare: (1) Race encoding and categorization are subconsciousand mandatory; (2) There exists a specific race processing unitinvolved with other cortical processing (similar to face pro-cessing). Apparently, these behavior findings may shed lighton developing cortex-like computational intelligent systemsfor race recognition.

As we could see from above, the past decades have wit-nessed a surge in understanding of how our brain perceptand recognize racial face information, and the richness ofperceptual experience also highlights the pre-eminence of racein automatic categorization. It can be generally concludedthat much of current research achievements have been guidedby two following distinct yet closely connected approaches:one is driven primarily by implicit neurophysiological obser-vation data, and the other is governed by explicit physical

4. For N170 component (modulation for face and non-face stimuli, thus,reflect structural face encoding), conflicting observations were reported con-cerning its sensitivity to racial faces. The reason might be due to variationof devices, subjects, and experiment conditions, the modulations may differin response to various face stimuli, causing observation inconsistent or evenfails among studies. Therefore, those results should be taken with care.

Fig. 6: Illustration of race processing regions [34].

appearance and psychophysics. This suggests that, by meld-ing neuro-computing models and physical discriminative fea-tures, the computer vision approach could establish a unifiedframework for race classification. However, simple combina-tion will not facilitate definition of algorithms, rather, it leavescrucial questions. Indeed, as much as we understand whereracial face encoding and decoding process are produced cog-nitively, very little is known on how they are representedand interpreted by the human visual system. Without propercomputer vision models and decision making systems, thescientific studies summarized above, the design of intelligentbioinformatics verification systems, as well as efficient HCIplatforms will continue to elude us. It is also important to notethat rather than viewing computer vision based race classifi-cation as some sort of mid-point on a spectrum of theories,models, and techniques with human cognitive model at oneend and physical appearance cues at the other, it turns outto be more helpful to view computer vision based race recog-nition system as a particular (vision-oriented) projection ofthe more general space of cognitive systems that embraces allperceptual modalities. This boils down to two key questions.First, which parts of the face are of critical importance for agiven racial face classification task by either arbitrary dividingthe face into several anatomically meaningful regions (e.g.,periocular regions, nose, and mouth), or simple holistic face?Second, how to extract meaningful features from these faceregions to train a corresponding classifier? Aiming to answerthese questions, significant efforts have been made during thelast decade for developing reliable appearance descriptors,compact representation methods, as well as statistical discrim-inative classifiers, as will be discussed in following sections.

3 RACIAL FEATURE REPRESENTATION

3.1 OverviewTo simulate how human visual system works, the first step isto find meaningful representation of the racial face one wouldlike to recognize. Early works involve derivation of color,texture, and shape based algorithms, also called appearance-based approaches, in which color, texture, geometric featureswere extracted from a racial face to build a model. This modelwas then fitted to the image for comparison. Advances inpattern recognition and machine learning have made this the



6preferred approach in the last two decades. Although thesemethods can achieve competitive performance efficiently froman engineering point of view, they do not necessary followthe face encoding mechanism used by human visual system,which has been widely agreed to be feature based categoriza-tion [94]. Furthermore, appearance-based approaches wouldperform poorly when facing image manipulation such as scaleand illumination variation. Recent approaches followed byfeature based method with the consideration of both configu-ral and shape features, making these algorithms more robust,efficient, and accurate. To this end, in this section, we startwith a brief review of several standard techniques for racialfeature representation.

3.2 Racial Feature Representation

Again, racially distinctive information is encoded by thevisual appearance of face. As the result, racial informationprocessing system begins with preprocessing and extractionof race-discriminative features. The goal is to find a specificrepresentation of the face that can highlight relevant racedistinctive information. According to the influential multi-level encoding model proposed by Bruce-Young [3], [4],human faces are distinguished by their characteristic cues-ranging from the basic global/holistic level (also called first-order relations), to the detailed feature analysis (coined assecond-order relations), till the final configural perceptionsof both levels (for specific individual identification) [1], [95].The diagnostic race categorization requires knowledge fromsecond order level, which means the main work of the currentresearches shall consequently be focused on race sensitivefeature representation and extraction techniques. In the fol-lowing we make distinction in the those representative featureextraction methods and sort them qualitatively as opposed toquantitatively:

• Chromatic representation Two dichotomies commonlyused for comparison in race classification are that of Asianversus Non-Asian, or Caucasian versus Non-Caucasian. Skintones have long been employed as the primary featuresto perform such rough race classifications [48], [96], [49],and the results seemed to be satisfactory. However, as il-lustrated previously, there exist several serious drawbacksfor this rudimentary feature: First, there are too much peo-ple from different racial groups who share same skin color.For instance, if judged by skin color, then all Southern In-dian/Austrilian/Melanesia/Africans would be clustered to-gether for dark-skin tones, though they apparently belong todifferent racial identities. What’s worse, skin color is highlysensitive in uncontrolled illumination environment and directemployment may cause severe errors [50], [51], [52], [53] (seeFig. 2 for illustration). To summarize, the strong claims putforward by sole skin color based race recognition can besafely refuted, not only on the basis of illumination variantcharacteristic of color appearance, but more compellinglybecause of the conclusive evidence from psychology studies[50], which demonstrates that skin color bears virtually norelationship to race perception. Nevertheless, the contributionof those researches lies in that visual appliances of chromaticattributes still remain essential since fusion of skin tone and

facial metrics can by all means boost the performance on racialprototypicality judgement (see Section 4).

• Global feature representation Holistic representationis arguably the most typical technique to be used in racerecognition for its capability to preserve configural (e.g. theinterrelations between facial regions) information, which isessential for discriminating race identity. For example, PCAis generally preferred to reliably extract and categorize facialcues to race (see Fig. 7) [4]. O’Toole et al have investigatedPCA-based methods in [97], [98], [99], [100], [60], [101] withimages from Japanese and Caucasian subjects, and their con-clusion comfired that the race could be predicted with rela-tively good accuracy (≥ 80%) with PCA. PCA has also beensuccessfully used in [102] for Myanmar and Non-Myanmarrace classification, in [103] for Arabic, Asian, and Caucasianclassification, and in [48] for Asian, African-American, andCaucasian classification, even in neural visual processing forrace categorization [101]. Those studies, together with exhaus-tive work [104],[105], provide evidence that the eigenvectorswith large eigenvalues are referred to as visually derivedsemantic information about racial faces. In short, PCA is stillone of the most frequently used statistical feature extractionmethods in racial feature representation.

Fig. 7: Illustrative visualization representing the relationship betweeneigenface and facial information, (b) and (c) are multiplied by 3σ,the square root of eigenvalue. Clearly the facial properties such asrace, pose, illumination and gender are controlled by the differenteigenfaces [104].

• Local feature descriptor representation Compared withglobal representation, local feature descriptor has a numberof advantages in the context of unconstrained racial facerecognition. For instance, being viewed as reliable and robustappearance feature descriptors, Gabor wavelet representationis undoubtedly suitable for race categorization applicationssuch as in [106], [107], [108], [109]. However, Gabor fea-ture vector resides in a space of very high dimensionality,so dimension reduction techniques are needed to acquirea more sparse, decorrelated and discriminative subset. Forexample, Adaboost [108], optimal decision making rule [110]and QuadTree clustering [111] have been employed to se-lect a compact subset yet preserving the discriminativeness.Other local descriptors have also been investigated by schol-ars. Fu et al embedded topographic independent componentanalysis (TICA) to form a hierarchical multi-level cortex-likemechanism model to recognize facial expression for different



7races [112]. Experiments show that such an ICA-based systemachieves a classification rate of 82.5%. Weber local descriptors,wavelet, and local binary patterns have been investigatedrespectively in [113], [114], [115], classification results on fiverace groups from the FERET database showed their effective-ness and superior performance over holistic PCA. Other low-level based features such as gradient direction histograms,multiple convolution network generated features (Fig. 8), andwavelet features were also discussed in [116], [117], [118],[119].

Fig. 8: Discriminative race feature representation by multiple layerconvolution neural networks (CNN). (a): supervised CNN filters, (b):CNN with transfer learning filters [116].

• Other representations Keeping in view of the abovediscussion it is hard to define the optimal representation5

way of racial features in real-world applications, thereforeseveral scholars have tried other alternative ways to preservethe configural ethnic information of the facial parts eitherimplicitly or explicitly by compromising all those aforemen-tioned representation methods to form a hybrid representationscheme. Such case can be exemplified by Ding et al’s effortof boosting local texture and global shape descriptor together[120]. By doing so a holistic representation can be obtainedfor several local regions of face and similarly a local compactdescription can still be obtained by concatenating severallocally processed regions of the face into one global vector.Those methods are still characterized as local since they useseveral local patches of the face, but they are simultaneouslyholistic in nature. Some of the typical combinations are fusingskin color, Gabor feature, local facial parts, and PCA together.As illustrated in the following section, this concept yieldsseveral unique approaches for race recognition tasks withsatisfactory performance.

To conclude this section, it is very interesting to discusshow human using different salient facial features for raceclassification. Ongoing efforts within cognitive neuroscience,pattern recognition, and advanced human-machine systemshave indicated the hypothesis that racial face processing isactually a trade-off between selectivity and invariance forefficiency and accuracy, with increasing complexity to facialappearance (rotation, scale, lighting, etc.). In other words, theracial face representation is rather dynamic and adaptive, cor-responding to the recognition distance and image resolution.

5. Although several other feature extraction techniques are available. Forexample, anthropometric measurements can statistically reveal the mostimportant facial feature areas (e.g. mouth, eyes and eyebrows), they areexcluded because they belongs to directive statistical analyzing manner ratherthan learning. However, the detailed discussion of these techniques will bepresented in next section.

For degraded facial images due to distance, the racial faceclassification is basically treated as a whole object, in whichskin color and holistic face information would be employed,which is fast and insensitive to local distortion and partialocclusion (glasses, hair, etc.). For dealing with high resolutionimage or close range, geometrical method and local repre-sentation will get involved for effective visualization, whichis computationally expensive yet more robust to physicalvariation (e.g., pose, lighting). Also, close range discriminationprocess would activate algorithms focusing on racially salientface features such as eyes, mouth, chin, and nose, and extracttheir correspondent geometrical points. To this send, patternrecognition techniques are critical for such feature based raceclassification task, which will be discussed in detail in thefollowing section.

4 RACE CLASSIFICATION: STATE-OF-THE-ART

While it is well recognized that face perception in nature ispiecemeal integration rather than holistic [121]. Facial com-ponents for race classification are nonetheless processed in aholistical and comprehensive way. Eye tracking results clearlyreveal that there exists a ranking list for those racially sensitivefacial components, implying a configural relationship amongthese features/regions. For example. the eyes and nostril parthave been consistently reported to be the most robust partsof the face verification, they will naturally provide essentialinformation that has to be taken into account of race classifi-cation. Below, we exhibit those feature’s contributions from afine-to-coarse roadmap, which corresponds to the recognitionaccuracy from the intrusive, close range in which single modelwill be sufficient, to the distant, non-intrusive range, whichrequires cooperation from multiple modality features (see Fig.9).

Fig. 9: Illustrative example showing racial sensitive regions.

4.1 Single-model race recognitionBehavior observation data have shown that explicit racialcategorization task is closely related to a comprehension work,in terms of both overall physical appearance featured (such asskin tones) or local discriminative regions (such as eye socketand nose). Consequently, it is reasonable that multimodalityapproaches shall outperform either mode alone. However, re-stricted by the dataset and computational burden, the majorityof research has been focused on single model based race classi-fication. Below we present several characteristic single modelrace classification approaches using various facial parts.



84.1.1 Iris TextureIris is arguably the mostly exploited biometrics in all kinds ofsubject verification and identification applications. Statisticallysignificant differences in retinal geometric characteristics havebeen reported in several behavioral studies[122], [123]. Thestudy of employing iris on race categorization initiated from[124], which had used Gabor feature and Adaboost selec-tion combination on a combined iris dataset for two-racesclassification. Their conclusion was that iris is race-related, aresult which has been further confirmed by [125], [126], [127],[128], [129], [130], all of which fit well with race recognitions(Asian/Non-Asian, Asian/Black/Cucasian). Specifically, Qiuet al. [125] studied the correlation between race and iris textureand concluded that race information is illustrated in iristexture features, with best classification rate of 88.3% by SVM.Similar results were obtained by Lagree et al [128] with 90.58%using sequential minimal optimization algorithm. Zarei et al.[126] applied multilayer perception (MLP) neural networksto achieve the corrected recognition rate of 93.3%. The topresult is 96.7% from [127] with the combination of super-vised codebook optimization and locality-constrained linearcodeine (LLC). However, remind that in typical real-worldapplications race is always viewed as a crucial and coarsesoft-biometric cue, thus normally required being identifiedfrom the distance or low quality video in a non-intrusiveand computationally efficient manner. Therefore, iris-basedrace recognition, as an intrusive and complicated acquisitionprocedure, is more theoretically feasible rather than practicallyapplicable. Iris texture categories that are closely correlatedwith race may be of value in database retrieval issues.

4.1.2 Periocular regionCompared with iris, periocular region (including eyelid, eye-lash, canthus or epicanthal, among others) is more easilyacquirable, rich in texture, and more quantified as useful bio-metric area (e.g., canthus is essential for differing Caucasianfrom other races because Caucasians tend to have a distinctcusp, see Fig. 10 for example) [121]. The idea that periocularregion contains discriminative racial information has receivedincreasingly empirical confirmation recently, even in infraredimages [131]. Among which, Lyle et al. [132] have usedperiocular region to perform race identification as Asian/non-Asian. Local binary patterns (LBP) is one of their main featureextractors. In addition, grayscale pixel intensity has beenused to evaluate their proposed system on the FRGC facedataset and a baseline accuracy of 91% was achieved. Liet al. [133] have investigated distinguishing features aroundthe specific eyelash region, they located eyelash region byusing ASM to model eyelid boundary and extracted 9 localpatches around it. The goal was to segment eyelash andto generate global eyelash direction distribution descriptorwith which the nearest-neighbor classifier was performed.Their experimental results showed 93% accuracy for East-Asian/Causians classification. Xie et al. [48] have used kernelclass-dependent feature analysis (KCFA) combined with facialcolor based features to tackle the problem of ethnicity clas-sification. Focusing on the periorbital region, the facial colorbased features were employed to incorporate with the filteredresponses to extract the stable ethnicity features. Compared to

previous approaches, their proposed method achieves the bestaccuracy of ethnicity classification on large industrial basedface databases and the standard MBGC face database.

Fig. 10: Illustrative example showing different periocular region,from left to right: Asian, African-American, Caucasian, Indian. Com-pared with iris image, periocular region could provide more richsemantic information for race classification (Source: Google Images).

4.1.3 Holistic face

Up to now, most existing approaches have been focusedon holistic, frontal faces (see Fig. 11 for illustration). Theearly work originated from Gutta et al [134], [135], [136]who have reported systematic investigation of using RBFneural network with decision tree for race classification onFERET database. Their best performance was 94%. Guo etal. [137] have adopted biologically-inspired features in theirethnicity classification system. It showed that the proposedethnicity classification algorithm’s accuracy can be as highas 98% within the same gender group. They further investi-gated the canonical correlation analysis (CCA) in solving thejoint estimation problem of three facial cues (race, gender,age)simultaneously [138]. Their results showed that CCAcould derive an extremely low dimensionality in estimatingthese demographic information, indicating the potential forpractical applications since CCA is very fast and compu-tationally convenient. Lu’s work [139] can be viewed as aclassical example for performing race classification on holisticfacial images, which employed LDA in an ensemble frame-work of multiscale analysis for Asian/non-Asian classificationtask. The experimental results on a manually mixed databaseacquired 96.3% accuracy overall (best 97.7%). A more so-phisticated probabilistic graphical model based method hasbeen proposed in patent [119], which constructed a filterpool for facial images and chose ethnicity-representative filtergroups for given ethnicity class to build an ethnic class-dependent probabilistic graphical model. By computing thelikelihood score from responses to each models of test imagethe classification result can be inferred. However, the racialgroup numbers and the algorithm’s performances were notstated.

As mentioned above, compared with cropped face, extrafrontal facial regions (such as hair) and their combinationswith specific facial components (such as eyes and nose) havealso been investigated to enhance the overall recognitionperformance. Li [140] has acclaimed a patent for ethnicityclassification, in which a combination of features, namelyblock intensity and texture feature (BITF), and a LDA clas-sifier were proposed to four racial groups (Asian, Caucasian,African and others). The contribution of this method is that theinventor took soft biometric features (hair color, eye color andeyebrow-to-eye metrics) into consideration, further boostingthe system’s performance. Lei et al. [141] have proposed



9

Fig. 11: Framework of typical race recognition system. The preprocessing includes the detection of facial regions, illumination normalization,edge detection. The second level functions like Gabor filter, sending output to the perceptual level for extracting features robust for selectivityand variance, after being grouped and classified, the category level gives the output [112].

another comprehensive approach using four face components(whole face, eyes, nose and mouth) and four low-level feature(HoG, color moments, Gabor, and LBP), a 16 combination(e.g. <eye, Gabor>) sets were formed. A two-level learningstrategy (SVM and Adaboost) was applied to find the optimalrelevance among the face region and feature (e.g. <whole face,color> is effective for African attribute). Then sparse codinghas been adopted for face retrieve framework. The highlightof their work was the experiment on a considerably large-scale Flickr dataset of more than 200k face images, achievinga significant improvement of hit rate@100 (from 0.036 to 0.42).

Particular attention should be paid on an interestingtemplate-based ethnicity classification work by Manesh et al.[110], who have employed optimum decision making rule onthe confidence level of automatically separated face regionsusing a modified Golden ratio mask. Instead of performingGabor feature extraction and classification directly, extractedGabor feature vector of each patch was treated as a singlepattern classification problem and recognition accuracy ofthese classifications indicated the confidence level of eachpatch for ethnicity classification, which were fused togetherto acquire final classification results. In a combined databasefrom FERET and CAS-PEAL for Asian/Non-Asian classifica-tion, they obtained recognition results as high as 98%.

The fusion of both global and local facial features for raceclassification has also been exploited by scholars recently. Forexample, Salah et al. [115] proposed a fusion scheme whichused block-based uniform local binary patterns and Haarwavelet transform. K-Nearest Neighbors (KNN) classifier onEGA database for three races (European, Oriental, African)classification obtained average results of 96.9%.

4.1.4 3D faces2D based racial face classifications normally encounter thedifficulty when facing geometric and illumination variation,to which 3D model based approach is rather insensitive.Recently several scholars have began their research for 3Dracial face classification without the associated texture or pho-tographic information. Therefore facial geometrical structureis explored for potential discriminativeness. Lu et al. [142] hasproposed an integration scheme of using both registered range

and intensity images for ethnicity identifications. Todericiet al. [143] proposed a framework for both ethnicity andgender based subject retrieval. Using pure facial-structure-based metric function measured from Harr wavelet and CW-SSIM coefficients, they evaluated four types of classificationmethods: KNN, kernelized KNN, multi-dimensional scaling(MDS) and learning based on wavelet coefficient. They re-ported high level of classification performance on FRGC v2.0:99% mean accuracy for MDS. Their results indicated that it ispossible to recognize race information with only 3D mesh ofhuman face. Similar results have been reported by Oceguedaet al. [144], who also investigated finding the most discrim-inative regions of the face for 3D racial face classification.Using Gauss-Markov posterior marginals (GMPM) for com-puting discriminative map of subjects from BU-3DFE database(Asian/White), they performed cross validation on FRGC v2.0for the aim of comparing with Toderci’s work. The resultsturned out to be very competitive with a much simpler linearclassifier. However, their work was based on the assumptionof smooth distribution of facial discriminative information,i.e., it is unlikely that a small isolated facial region containhigh discriminative information while its neighbor regionsdo not. From a computational point of view, there is noguarantee that this assumption can be generalized to otherraces. Zhong et al. incorporated Gabor features with QuadTreeclustering to form a visual codebook for both western andeastern subjects [111] by using Max distance function andfuzzy membership function, they proposed a fuzzy 3D facecategorization approach, which reached performance as highas 80.16% for eastern and 89.65% for westerners on FGRC2.0 3D face database. However, a rough 2-class categorizationlimits its further analysis and it remains unknown whethertheir method is suitable for 2D face problem. Ding et al.[120] proposed a combination method of boosted local textureand global shape descriptor extracted from 3D face models.Oriented gradient maps (OGMs) were used to highlight eth-nicity sensitive geometry and texture feature sets. Experimentscarried out on FRGC v2.0 datasets obtained performanceup to 98.3% to distinguish Asians/non-Asians. In total, allrepresentative achievements have been carried on FGRC forAsian and non-Asian, it is of interest if the generalization can



10be extent to multiple classes retrieval problem.

Other than traditional approaches, recent studies on phys-ical anthropometry have led to the investigation of using3D anthropometric statistics for race categorization and re-lated face analysis. From Section 2.4 we know that physicaldiscriminative facial traits vary cross different racial groups,most of which are located in flexible areas such as mouth,nose, eyes and eyebrow. Correspondingly, building a 3D high-definition head-face model (such as the one used in [145], seeFig. 12) is a pre-stage work for reliable and accurate detectionof those landmarks. An anthropometric face model has beenbuilt in [146], in which crucial facial feature region wasidentified by using eye-nose triangle relationship and 18 mostdiscriminative facial feature points were detected separatelyby using histogram and contour following algorithm. Anethnicity specific generic elastic model from single 2D imagewas proposed in [109] for further synthesis and recognition(Fig. 13). The whole anthropometric discriminative structureof race and location of facial fiducial landmarks could beperfectly revealed by those models.

Fig. 12: The anatomically anthropometric landmarks used in [145]. Itshould be noted that those landmarks are highly redundant, there-fore, how to chose a distinct subset according to some predefinedcriterion is essential.

Fig. 13: 3D generic models and depth image of different races [109]and anatomical landmarks of Gupta’s work [145].

Other from 3D race models, how to explore the featureselection during race information processing is also essential.Berretti et al. [147], [148] have investigated the individualrelevance of variation of local facial regions and differentethnic groups in 3D face recognition. The aim was to identifythe most relevant features for different ethnic groups. Experi-mental results on the FRGC v1.0 dataset showed that the most

relevant information for the purpose of discriminating tworacial groups (Asian and Caucasian) is captured by the spa-tial arrangement between (3,7) and (4,5) pair, respectively. Itrepresents a viable approach to improve recognition accuracy,by enabling training of more accurate classifiers on specificethnic groups (See Fig. 14). Gupta et al. [145], [149] haveextracted a highly compact anthropometric feature feature setincluding 25 facial fiducial points associated with highly vari-able anthropometric facial proportions. Sukumar et al. [150]have used facial anthropometric data to learn structurallydiscriminant features in 3D face on a racially diversifieddatabase. Their experimental results are two-fold: (1) a sparsefacial fiducial distance set (e.g., the vertical profile, jaw-jawdistance, depth of nose, and depth at the eye and chin to neckdistance features) contributes most for race discrimination;and (2) the anthropometric features have great potential forrace categorization/classification.

Fig. 14: The anatomical landmark regions used in [148], showing thefive most relevant iso-geodesic stripe pairs for Asian and Caucasiansubjects.

In summary, robust automatic location of facial fiduciallandmarks (makes algorithm reliable) and effective selectionof facial anthropometric proportion (makes algorithm com-putationally efficient) are two essential steps for guaranteeof these methods’ success. In fact, in vast majority of theseresearches, including those conducted by the authors of thissurvey, great care should be taken to tune the preprocessingparts, including alligation, marking, measurement, in orderto minimize any difference between race categories, in termsof the low-level geometrically physical attribute of individualsubjects. Therefore how to facilitate those pre-processing stepsis still open question for further research exploration.

4.1.5 Dynamic face

Current methods are trained on the still images sets andthus could only apply off-line. Shakhnarovich et al. [151]have proposed a unified learning framework which couldclassify ethnic groups with a real-time manner. Their real-time demographic (race and gender) classifier was built onfast face detection algorithm. Harr wavelet function was usedfor ethnicity feature extraction which is further filtered byAdaboost. The key concept lies in evaluation of the classifiercross time by temporal integration of a decision criterion D(t)

D(t) =1

T

T∑i=0

e−αiV (f(xt−i))Q(xt−i) (1)



11which allows for combining classifier output at each timeframe throughout a video sequence. Experimental results onboth still image sets and video clips verified their approach.Although the dataset was biased (non-Asians outnumberAsian) and classifier was simple binary, it offers a directionfor on-line potential applications on low quality video surveil-lance in future.

4.2 Multi-model race recognition

Up to now, lots of effort on the race classification havebeen focused on using single modality, which may causeuncertainty when facing degraded images. This uncertaintyarises from both the physical nature of racial face (for example,the variance of skin color even within same race group) andthe transformation of these physical cues into meaningful cat-egories. Statistically, a simple way to reduce such uncertaintyis to combine data from multiple measurements, both withinand across modalities. Recently a few studies have investi-gated feasibility of using multiple modalities integration forthe task. Zhang et al. explored the ethnic discriminabilityof both 2D and 3D face features by using MM-LBP (Multi-scale Multi-ratio LBP) multi-modal method [152]. Anotherpromising approach on multi-modality based race recognitionis the fusion of face and gait. Empirical results from thepreliminary investigation [153] suggested significant role ofgait biometrics in conjunction with face biometrics as potentialcandidates for more advanced race identification techniques.It should be pointed out that Zhang’s team has been veryactive in multi-modality fusion based ethnicity classificationproblem. Recently they have further extended the fusionresearch by introducing a cascaded multi-modal biometricssystem involving a fusion of frontal face and lateral gait traitsin [154]. Their conclusion was that the combination of gaitcues in long distance and frontal Gabor facial features inshort distance can perform ethnicity recognition effectivelyand thus significantly improved the overall classification ac-curacy. For example, over 99.5% classification accuracy wasreported on the FRGC v2.0 database in [152]. In summary,research on multi-modality race classification has witnessedsignificant progress in the last few years. The proliferationof literature in this direction also indicates that more variousmultimodal data should be applied, such as affective cuesor audio cues. Correspondingly more multimodal data fusionmethods should be investigated, such as HMM-based fusionor NN-based fusion.

4.3 Intra-ethnic recognition

As far as race categorization is concerned, most of the exitingefforts studied the basic race groups (African Americans, Cau-casians, Asians, in some cases with Indians) due to their rela-tive discriminativeness, their marked representation, and theavailability of relevant training and testing material prepara-tion. There are a few tentative efforts to detect non-universal,sub-ethnic groups, such as Koreans, Japanese and Chinese.Though being supported by the physical anthropometricalevidences, performing those intra-race group categorizationstill needs much more accurate classifiers and support fromdeliberately displayed, large scale, racially diversified facial

datasets, including both neutral and emotional faces. Recentlythe researches have switched from widely separated popula-tions, both geographically and culturally, to more closely re-lated but ethnically distinct groups, such as sub-ethnic groupsin Mongoloid (Zhuang, Han, Tibetan). Duan et al. [107], [155]have suggested distinct differences maybe exist even in thesame ethnic group geographically, this has been supportedby astrometric characteristics among three Chinese ethnicgroups. With respect to these geometrical variations, threeelastic templates of all ethnic groups were built. Using featuresextracted by Gabor wavelet and KNN classifier, performanceof the approach was verified on an ethnic dataset. Theirresults indicate that it would be of interest to sort and detectall those essential facial characteristics among intra-ethnicgroups. However, since the definition of subdivisions of racialgroups lack concurrence, the generalization of such methodsrely on the concept from physical anthropology heavily.

To summarize this section, Table 1 provides an overview ofthe currently existing exemplar systems for race categorizationwith respect to the utilized facial features, classifier, andperformance. We also mention a number of relevant aspects,including the following:

• Type of the utilized data (subject background),• Type of the approaches applied (feature representing and

classifier),• Databases used (race category and number, other infor-

mation).Considering the benchmark results of so many datasets, we

could see a clear trend in increase performance over the yearsas methods have gotten better and training datasets havebecome larger. Apparently, more feature combination andmore sophisticate algorithms would allow to further improvequality. However, we already notice that existing datasetsreach saturation (most results are in the range 95%-98% of theperfect solution). We believe it is thus time to move towardseven larger datasets, and to include subsets recorded underreal-world conditions (more race categories, illumination andview variant, etc). This is going to be discussed in next section.

5 RACIAL FACE DATABASES

It is well known that for any facial information analyzingsystem, if the training data is not representative of the testdata which an algorithm relies on, the performance of thealgorithm could deteriorate. This has been repeatedly con-firmed by many large-scale face recognition tournaments,such as 2002 NIST face recognition vendor test (FRVT) [26],2006 FRVT [27], [28], and 2010 NIST multi-biometric evalu-ation [29]. Therefore, one can find that the majority of raceface recognition researches, correspondingly, were based onthose commonly accepted representative databases, such asFERET[163]. Although not intentionally designed for racialface processing, these databases’s large-scale and relativelycomprehensive race composition characteristics provide amore or less significant contribution for most early work onracial face recognition.

However, researchers have gradually realized that the per-formance could not always be guaranteed on traditional, nonrace-specific face databases. Indeed, many face databases em-ployed are actually race ill-balanced, sometimes scholars have



12

Table 1. Commonly used methods for race face recognition

Author Approaches Accuracy Database Details(feature+classifier) (%) (name and numbers) (Specific information)

Lyle et al. [132] Grayscale and LBP 94 FRGC Asian v.s. Non-Asian(4232 faces, 404 subjects)

Guo et al. [137] BiFs(Biologically 98.3 MORPH-II African American v.s. CaucasianInspired Features) (55000 faces) (only in the same gender)

Tariq et al. [96] Shape Context matching 80.37 Silhouetted images(441 faces) East vs. Southeastern Asian

Zhang et al. [156] Gait Energy Image 84 Walking people East Asian v.s. South Americanfrom 7 cameras

Zhang et al. [152] LBP based Gait Fusion 97 FRGC 2.0 (1917) Asian v.s.CaucasianZhang et al. [154] LBP based Gait Fusion 95 Collected online(24) East-Asian vs South-American

Xie et al. [48] KCFA and color features 98 Mugshot DB(50000 Caucasian, Caucasian, Asian50000 African, 4000 Asian) and African American

Xie et al. [48] KCFA and color features 98 MBGC DB(20000 Caucasian, Caucasian, Asian10000 African, 10000 Asian) and African American

Qiu et al. [124] Iris texture 85.9 CASIA, UPOL and UBIRIS Asian v.s. Non-AsianGabor+Adaboost (3982 iris images)

Qiu et al. [124] Iris texture 85.9 CASIA, BioSecure DB Asian vs. Non-AsianGabor+SVM 91 (2400 iris images)

Zarei et al. [126] Iris texture 93.3 ND-IRIS-0405 DB Asian vs. CaucasianFilter + MLP neural network 1200 iris images (120 subjects)

Lagree et al. Iris texture 90.58 ND-IRIS-0405 DB Asian vs. Caucasian[128], [130] SMO+SVM 1200 iris images (120 subjects)

Zhang et al. [127] Iris texture 96.7 CASIA+UPOL Iris DB Asian vs. CaucasianCodebook+SVM 11320(2066 eyes)

Li et al. [133] Eyelash region 93 CMU-PIER+UBIRIS Asian vs. CaucasianASM+KNN 214 iris images

Lu et al. [142] Range and Intensity 96.8 UND and MSU Asian vs. Non-AsianSVM (276 and 100 subjects)

Lu et al. [139] Gaussian+LDA 97.7 AsianPF01, NLPR Asian vs. Non-AsianAR and Yale

Zhong et al. [111] Gabor + K-means 89.65 FRGC 2.0 Eastern vs. WesternAkbari et al. [54] Fuzzy Moments, Complex 96 Ethnic DB Arabian Muslims

Geometric, Legendre (100 subjects)Klare et al. [157]] LBP+Gabor 91 Operation DB White, Black and Hispanic

PCA+LDA (102,942 faces)Gutta et al. RBF+Decision Tree 94 FERET DB Caucasian, Asian

[134], [135], [136] 3006 faces (1009 subjects) Oriental, AfricanDuan et al. [107] Gabor+LDA 90.95 Ethnic DB Tibetan, Uygur, Zhuang

Lin et al. [108] Gabor+Adaboost+SVM 95 FERET African American, Caucasian,(N/A) Eastern Asian

Ou et al. [105] PCA+ICA+SVM 82.5 FERET Asian v.s. Non-AsianToderici et al. [143] Harr Wavelets, kNN 99.5 FRGC 2.0 Eastern vs. Western

Kernel kNN, MDS, (3375 faces) Asian vs. WhiteShakhnarovich et al. Harr Wavelets+ Adaboost 81 Mugshot from Web Asian v.s. Non-Asian

[151] (3500 images +30 video clips)Tin et al. [102] PCA+ KNN 96 Mugshot from Web Myanmar vs. non-Myanmar

(250 faces) Kachin, Kayah, originsBellustin et al. [158] Skin color + Adaboost 96 Indian face DB Indian and European

910 Whites/646 IndiansWu et al. [159] Haar+Adaboost 96.1 FERET DB Asian, Caucasian,

(3105 faces) African AmeircanHosoi et al. [160] Gabor+SVM 96 HOIP DB Asian, Caucasian, African

Muhammad et al. LBP,WLD+KNN 96 FERET DB Eastern/Middle Asian, Hispanic[113], [114] (2368 faces) Caucasian, African American

Kumar et al. [117] Color histogram 94.6 PubFig DB Asian, CaucasianSVM (58797 faces) Indian, African American

Ding et al. [120] Shape+ Local texture 98.3/97.8 FRGC 2.0/BU-3DFE Asian vs. Non-AsianAdaboost+Decision Tree 4007(466)/2500(100)

Manesh et al. [110] Facial part templates 98 FERET+PEAL Asian vs. Non-AsianGabor+SVM (1691 faces)

Salah et al. [115] LBP+Harr Wavelet 98.7 EGA DB Asian, CaucasianPCA+KNN (746 faces) African American

Ahmed et al. [116] CNN+ Transfer learning 93.9 FRGC 2.0 DB Asian, Caucasian(14714 faces) Other

Roomi et al. [49] Skin color+ Adaboost 91.6 Yale + FERET Caucasian, Asian,(250 subjects) African American

Demirkus et al. [161] Skin color + Hair Color 94 Online Caucasian, Asian,SVM (600 subjects) African American

Notes on the table: Not all datasets give training data and testing data, therefore they are not listed. Accuracy means the best results for all possible racegroups. N/A means the information is not given in original paper.



13

Fig. 15: Diagram showing the whole configuration of the CUN face database (with varying poses, expressions, accessories, illumination,photographic room and a multi-camera system [112], [162].

to combine several databases together to perform multi-raceclassification [106], [110], [115], [164]. Therefore, a sufficientlylarge and fairly representative racially diverse face database,though resource-intensive, is by all means necessary andimportant promise for assessment and evaluation of currentlydeveloped algorithms. The merits of designing such a raciallydiverse face database are listed as follows:• To be able to obtain pre-annotated, preliminary race

categorization for retrieval or verification tasks while otherdatabases are incapable of acquiring.• To provide uniform representative racial information for

discovering the deep cognitive mechanism underlying faceperception.• To pave the way for a through and systematic study

of how certain factors such as race, gender, age, emotioninteractively influence the automatic recognition of faces.

Due to the aforementioned merits, recently, several scholarsand research institutions have began their exploration in thisarea by setting up racially diverse face databases. In this sec-tion we begin our discussion by briefly reviewing the currentpublicly available face databases6 that are of demonstrateduses to scholars.

6. Note that many databases are actually common ones in face recognitionfield. However, considering that many early race recognition approaches havebeen evaluated on these datasets. We still list them for the aim of presentinga comprehensive review. On the other hand, for those datasets not frequentlyused but may have potential contribution to the field. We also list them forresearcher’s convenience.

5.1 Major representative face databases

Table 2 presents an overview of these noteworthy databasesthat have been ever reported in literature. For each database,we provide following information,

(1) Subject background (ethnic groups, gender, demo-graphic information),

(2) Capture environment (i.e., indoor/outdoor, multi-illumination, pose, accessaries),

(3) Affection information (i.e., whether the facial expressionsare captured and/or categorized),

(4) Sample size (the number of subjects and available datasamples. Such as photos or videos, whether being sampledaccording to some statistics or randomly).

Based on Table 2, it is apparent that most databases haveconsidered many real world situations and have their ownunique characteristics. We highlight several representative andcommonly used databases with their major characteristics asfollows:

(1) Texas 3D ethnic database. This database includes 3Dimages from the major ethnic groups of Caucasians, Africans,Asians, East Indians, and Hispanics. This database have al-ready been testified by some anthropometric based algorithms[145], [149].

(2) The CAS-PEAL (Pose, Expression, Accessory, Lighting)database. This database has been considered as a major Chi-nese face database and is now mainly used for face recognitionand related applications. Consequently, it is often been treatedas an Asian subset to be combined with other races to form



14Table 2. Commonly used racial face databases

Name Size of subjects Recording conditions Race/Ethnicity BackgroundCAS-PEAL [165] 1040 (30,900) Expressions (6), illumination (9), occlusion (2), Chinese subjects.

time (1-5), poses (21), background (4).IFDB [166] 616 (3,600) Facial expressions (2), illumination (1), Iranian subjects

age (2-85), poses (4). (Middle east).Texas 3DFRD [145], [149] 105 (1,149) 2D/3D data stereo device. Caucasians, Indians, Asian.

Anthropometric facial fiducial points.KFDB [167], [168] 1000 (52,000) Facial expressions (5), poses (7). Korean subjects.

illumination (16).JAFEE [169] 216(10) Subjects watch emotion-inducing photos and Japanese subjects.

videos, displaying 7 typical expressions.CAFE [170] 10 posers for each of the Caucasians, East Asians,

display 7 typical expressions and Pacific Region.FGRC 2.0 [171] 4007(466) Facial expressions (6), 3D (range and texture) , Latin, Caucasian.

time and illumination variations (N/A), age (18-28). Indian, African American.CMU DB [172] 1.5 million Frontal or near frontal, Caucasian, African American,

ground truth ethnicity labels provided. Asian, Hispanic.BU-3DFE [173] 2400(100) 3D range data using 3DMD digitizer, White, Black, Middle-east Asian,

6 basic emotions, 4 levels of intensity East-Asian, Indian, Hispanic LatinoFERET [163] 14126 Illumination (2). Facial expressions (2). Caucasian, Asian,

Poses (9 - 20), time (2). Oriental AfricanEthnic DB [107] N/A Frontal images of ethnic minority students. Chinese Ethic Minorities.

Cohn-Kanade [174], [175] 210 Grayscale values, facial Expressions (6), African American, Asian, Latino.(480 videos) and AUs (action unit).

Asian PF01 [176] 103 Front face (1), illuminations (4), Korean subjects.poses (8), facial expressions (4).

Hajj & Umrah [177] N/A Poses, facial expressions (4), Saudi Arabia subjects.(HUDA) illuminations, backgrounds, accessaries (N/A) (from 25 countries)

JACFEE [178] 56 Poses, facial expressions (7), Japanese, American subjects.Illuminations, backgrounds, accessaries (from 2 countries)

CUN [112] 100,000 Poses, facial expressions (7), Chinese ethnic subjects.Illuminations (27), backgrounds (4), accessaries (4) (from China)

Indian DB [179] 40(710) Pose, backgrounds, facial expressions (5), Indian subjects.FEI [180] 2,800 Poses, facial expressions, accessaries, Latin ethnic subjects

(100) facial landmark pre-annotated. (from Brazil)EGA [164] 469 Poses, facial expressions, accessaries, 5 main racial groups.

age, illumination, gender. (Mixed from other 6 databases)

Notes on the table: The first term in second collum means means the number of subjects, the second term means the actual number of captured imagines.Some information on the recording conditions are not available from the original paper.

a balanced test benchmark for race recognition [110], [115],[164].

(3) CUN database. This database covers variations in race,facial expression, illumination, background, pose, accessory,among others. It includes 56 Chinese “nationalities” or ethnicgroups for race recognition (See Fig. 15).

We also note that there are some specific race-orienteddatabases such as Iranian face database (IFDB) [166], Hajj andUmrah database (HUDA) with middle-east Asian database[177], Indian face database [179], as well as FEI with Latinethnicity database [180]. It is therefore convenient to integratethese single race dataset to create a more heterogeneous andrepresentative database for race recognition. For example, theEGA face database can be viewed as a typical example ofsuch kind [164]. Also, BU-3DFE database [173] would be quiteuseful in 3D race recognition and racially diversified facialexpression recognition.

We would also like to note that in addition to these afore-mentioned face databases, there are two additional categoriesof data sources which are of particular interest to the race-facelearning: anthropometric demographic survey data and raceface generation software. In the following two subsections,we also briefly review these two types of data to provide acomplete survey of the ethnic face databases.

5.2 Anthropometric demographic survey data

As mentioned in section 2, anthropometric surveys havealways been an valuable resource for carrying out race catego-rization. We list several representative anthropometric surveysconcerned with facial anthropometric information.

• CAESAR, also called Civilian American and EuropeanSurface Anthropometry Resource [181]. The CAESAR projectwas the first anthropometric survey to provide 3D humanmodels with spatial landmarks. Data were gathered in NorthAmerica, Netherlands, and Italy (13,000 3-D scans from 4,431subjects). CAESAR has been used by Afzal Godil [38] forface recognition using both 3D facial shape and color mapinformation, satisfactory results were obtained.

• NIOSH, National Institute of Occupational Safety andHealth (NIOSH) head and face database, the main body ofdata consisted of 947 data files in the format of a Unix-based,3D package called INTEGRATE [14]. Each file contained 3-Dcoordinate locations of anatomical landmarks for one indi-vidual with demographic information including gender, age,race, and traditional anthropometric measures were collected.

• SizeChina being viewed as the first 3D head and facescan survey for the adult Chinese population [17]. It has col-lected high resolution 3D data from over 2,000 adult Chinesesubjects. Thirty-one facial landmarks were used for statisticalanalysis and to create a high definition 3D anthropometrichead and face model. SizeChina has been used for revealing



15some subtle facial dimension variations between Chinese andCaucasians in [18], [19].

• USF short for the University of South Florida (USF)Human ID 3D database [182], sponsored by the US defenseadvanced research projects agency (DARPA). It contains 100laser scanners aligned to a 3D reference model of 8,895facial points. USF database has been used for several facerecognition applications.

5.3 Racial face generation software

Apart from those aforementioned standard databases, theemergence of artificial face generation software should alsobe viewed as a great supplement. They offer the systematicparameters of specific features of facial structure and moregeneral properties of the facial demographic information (age,ethnicity, gender). For example, FaceGen Modeller [183], acommercial tool originally designed for face generation invideo games (Singular Inversions Inc), has been widely usedfor creating and handling realistic 3D facial stimuli in over150 ways, including race, age, facial expression and gen-der (see Fig. 16). Another noteworthy software is FACSGen[184], [185], a synthesized lib which can import any faceexported from FaceGen Modeller. With a friendly UI, userscould manipulate up to 150 high level morphological topologyof the face. Though being argued on the effectiveness andvalidity of application (generated face models in this waylacks detailed texture and scale), those softwares have alreadybeen widely used for social cognition, cognitive neuroscienceand phycological research areas related to face perception andrecognition [186], such as face sensitive perception [93], other-race effect [92]. It is reasonable to believe that in the nearfuture more racial face related softwares will emerge with theproliferation of racial face needed research areas.

Fig. 16: 3D facial stimuli generated by FaceGen Modeller [183]

6 RACE RECOGNITION: REAL-WORLD APPLICA-TIONS

As an essential facial morphological trait, race perceptionkeeps affecting our life. With the changing of views towardsthe race issue due to globalization, the ever-increasing facialdata for both social and scientific research purposes, newemergence of devices and support from government agencies,

increasing applications and related projects have been ex-plored. Salutary examples of race-sensitive application includethe following:• Biometrics based individual identification Racially di-

verse structure is inherent within facial appearance and isechoed in both human descriptions and biometric represen-tation. Therefore, race could be used to classify an individ-ual in coarse and broad categories, helping to refine thefurther discriminative recognition and identification tasks.For example, in a racial/ethnic database where women withHijab, or men with beard or mustache are crucial featuresin Muslim community such as Arabian countries [54]. Racecould be especially useful for being incorporated into videosurveillance system [161], [187], [188], [117] for security, publicsafety, offender identification and forensic evidence collection.Increasingly terrorists’ threats call for close integration of reli-able development of such race/ethnicity sensitive informationextraction and correspondingly intelligent video surveillancesystem, which is capable of providing meaningful analysisand extracting categorical information (such as gender andrace) from poor quality video without need to recognize oridentify the individual. Notable examples include Demirkus’sprototype system using soft biometric features (skin tone, haircolor, age and ethnicity) to tag people in multiple camerasnetwork [161], and Bellustin’s instant human attributes clas-sification system, with embedded classifiers for age, race andgender recognition [158]. Indeed, in certain specific applica-tions of video surveillance where a face image is occludedor is captured in off-frontal, illumination-challenging pose,race information can offer even more valuable clues for facematching or retrieval. In short, the employment of racial traitrecognition is highly expected to improve the face analy-sis performance when appropriately combined with a facematcher [157], [106].• Video security surveillance and public safety Race iden-

tification, along with other soft-biometric information (suchas hair color, eye shape, gait, gender, age, among others), canbe embedded into a high-end video surveillance analyzingsystem. Such an intelligent analytic solution can be appliedin airports and other critical infrastructures to prevent crimeby comparing images detected by the system against a presetblacklist database. Also note that fast racial group identifica-tion helps to facilitate database retrieval by discarding subjectsnot belonging to the specific race category. Such a video-basedpredictive profiling strategy have already been proven to bequite useful in aviation, maritime, mass transportation, andother open environments, such as large retail malls, privateand governmental office buildings, recreational centers andstadiums (see Fig. 17). On one side, in limited and definedcircumstances, race sensitive identifier may be appropriateto protect public safety, and assist in the investigation ofpotential hazard activity. On the other hand, the extractionof race sensitive information also helps to build more intelli-gent surveillance system for privacy protection, which couldgenerate a video stream with all privacy-intrusive informationencrypted or scrambled, such as PrivacyCam [189].• Criminal judgment and forensic art It is well known

that race can be considered as critical ancillary informationthat is useful for forensic experts to give testimony in court of



16

Fig. 17: Systematic illustration of a computationally intelligent biometric video surveillance system. From the top: the first level dividesthe input data into two branches: physical body tracking and face tracking. With interaction of databases, in the person session the systemfocuses on the overall biometrics such as height, behavior and trajectory following; while in the face session the system mainly concernsabout detailed facial information, such as facial expressions, gender, race, identity, etc., in a coarse-to-fine order. Weighted outputs are sentto fusion center where the final decision making is performed. The system can be accomplished in a multi-camera networks. The extractionof those sensitive information also helps to build intelligent CCTV for ensuring privacy.

law where they are expected to conclusively identify suspects.For example, eyewitness identification, as one of the most im-portant methods in the criminal scenery investigation, appre-hending criminals and direct evidence of guilt, can be volatileby facial recognition deficit due to the cross-race effect or ORE(see section 2 for ORE). It has been arguably speculated forcontributing to unequal treatment or wrongful conviction ofinnocent ethnic minority group members [190]. Research onstereotyping in the United States also reveals that persistentracial prejudice influences the criminal justice system heavily[191], [192], [193]. The dilemma among law enforce officersand witnesses in process of identity parades addresses thenecessariness of automatic race/ethnicity analysis techniques.For instance, intelligent video cameras installed in criminalscenery could offer persuasive evidence of objective raceor soft-biometric information identification, helpful for crossvalidation, thus preventing false-positive identification of sus-pects. Also, computer-based race synthesis can significantlyenhance the efficiency of the forensic artist while providingphotorealistic effects needed [194]. Even forensic databases inthe Unite Sates are typically organized according to racial cat-

egories. By considering separate databases for different ethnicgroups, forensic scientists minimize the probability of unfairdecisions against members of minorities. Enlightened by thesocial and scientific significance, it is important to considerhow analyzing racial identities impact policy making, lawenforcement, and criminal judgement procedures. In short,computationally intelligent race recognition algorithm couldcertainly provide quantitative and descriptive evidence thatcan be used by forensic examiner in the courts.

• Human computer interface (HCI) Racial cues are perhapsbest exemplified by their potential to the computer-consumerinteraction factor. By identifying the user’s ethnicity belong-ings or corresponding culture background, ethnicity homo-phyllic services (or virtual agents with sociodemographiccharacteristics) can offer customers ethnically congruent ser-vices, prohibiting the potential hazards of being offended bycultural/ethnical taboos and undesirable effects[195], [196],[197]. The next generation of robot should respond respect-fully and effectively to people of all ethnicity and race be-longings in a manner that recognizes, affirms, and communi-cates in cross-race situations. For example, in an appropriate



17settings to increase the quality of services, such techniquescan be quite useful in public service environments, such ashospital, health care center, museum, a smart HCI system oran avatar [198], [199], [200] can detect subtle, appearance-based or behavior-based racial (even culture specific) cues,then offer services such as the choice of speaking English ver-sus Arabic or other alternatives. These systems will ultimatelychange ways in which we interact with computer vision basedprograms.

7 RACE RECOGNITION: CHALLENGES AND OP-PORTUNITIES

7.1 What have we learned so far?We have reviewed five key topics on race recognition so far:problem motivation and formulation, racial feature represen-tation procedure, models and methods, race face databases,and applications. Based on which, we briefly summarize whatwe have learned so far from these intensive, multidisciplinaryresearch over the past decades.

7.1.1 Categorizing and classifying racesThe nearly 99% accuracy of Toderici et al’s work [143] indi-cates that common approaches such as similarity metric modelcould achieve satisfactory result without using anthropomet-ric feature points, which paves the way of using traditionalface recognition like method to perform race recognition, andthis has been confirmed by a bunch of papers. However, itshould be noted that these satisfactory performances are basedon high definition 3D mesh models and on rather simple andeasily separated subject groups such as Asians/Caucasians.When those two assumptions do not hold, performance ofsuch system cannot be guaranteed.

As a cue sharing both holistic attributes and localized subtlevariations, the categorization of race can be accomplished bothby holistic approaches and local feature extraction methods,which depends on granularity of the problem. Thus, theaccuracy and importance of the racial diversity identificationdiffers across application background. Available methods re-flect these differences, but only in a rather rough way (suchas Asian/Caucasian), while failing to account more diversifiedclasses or clusters. This will usually require the developmentof more detailed and accurate models linking discriminativefeature extraction methods, multi-level racially distinct in-formation fusion logics, and comprehensive decision makingunits. With the establishment of large-scale, racially diversifiedface databases and the development of more computationallyintelligent, cortex-mechanism like approaches, more specificmodeling and categorization methods will soon be tractableand feasible.

7.1.2 Race categorization by anthropometriesThe next achievement is the comprehensive framework ofunderstanding and categorizing race in ways of physicalanthropometric parameter and ad-hoc assumptions, in termsof both 2D/3D face datasets. Being viewed as a visually basedstimulus category defined by the statistical variability of facesfrom within and across demographic categories, statisticallysignificant differences are observed among the race/ethnicity

groups. The detection and extraction of facial features relevantto those physical differences are practically essential for build-ing reliable race categorization based on anthropometric mea-surements. We list several noteworthy contributions in thisfield. Nevertheless, the inborn inconvenience of differencesin traditional anthropometrical measurements which couldbe cumbersome in study design, measurement protocols, andstatistical analysis, which prevents further robust analysis ofthe measurements, and the problems of robust and accuratelocation of those anthropometrically distinct facial fiducial fea-ture landmarks. However, with fast development of 3D digitalphotogrammetry and scan technology with which accuracy,reliability, precision could be available [17], future improve-ments would be directed to the further detailed analysis ofdiscriminative feature landmarks for race classification.

7.1.3 Racially diversified face databasesWe have listed all the representative databases in this researchfield, with detailed analysis in terms of parameters. Moreimportantly, we also note the successful launching of severalrepresentative 3D human body databases, a natural wealthwarehouse for carrying on anthropometric based race classifi-cation, which provides an opportunity to improve the currentresearch standards by offering anthropometric measurements.It is reasonable to presume more joint research will be carriedon both physical anthropometric data and facial image datain future. Current 3D anthropometric surveys created to datefocus largely on Western populations, therefore the basic 3Dhead shape of a significant portion of the world’s populationremains unknown, which should be seen as a future researchdirection.

7.2 What could we do next?Based on our current understanding of this challenging topicfrom multiple disciplines and existing technologies, there areseveral important future directions to further improve raceclassification. We highlight a few of which to motivate futureresearch and development in this critical field.

7.2.1 Real-world learningThe real challenges in race classification, as mentioned in thissurvey, are how to get satisfactory classification performancein both real-world scenarios and in a massive scale. Tractionalrace classification is usually carried on the carefully designedface database, which provides a clean and cropped frontalface image. The generalization to real world application en-counters the problem of complicated, low-definition, variedillumination video cameras installed at airport, metro systems,shopping malls, etc. Extra challenge comes from the fact thattraditional frontal view based methods may fail when dealingwith non-frontal, multi-view face images. For example, skintone or color attributes have long been considered as an essen-tial part in most majority of racial face recognition. However,it must be pointed out that most crime cases empirically hap-pen at night, in which the night-vision security camera sys-tems could only provide inferred photo/video without color.In such cases, more accurate identification algorithms arerequired, promising directions includes anthropometry andcombination with other soft-biometrics, such as gait. A recent



18emerging direction is to use silhouette methodology (shapecontext based matching)(See Fig. 18). Empirical results frompsychology [201] and computer vision [96], [202] have bothconfirmed silhouette’s potential in race categorization. How-ever, silhouettes lack texture and color information, whichare also known to contribute to recognition and perceptionof race. Therefore, the best way would be a multi-modalityfusion framework addressed as follows.

Fig. 18: Silhouette profiles across races, from left to right: African-American (Male/Female), Asian (Male/Female), Caucasian Ameri-can(Male/Female). Figure source: [201].

7.2.2 Manifold learningFace image lies in a high-dimensional space, but a class ofimages generated by latent variables (race, facial expressions,gender, pose, illumination, etc.) lies on a smooth manifold inthis space [203]. Therefore, a key issue in race categorizationis through projecting the face images into low-dimensionalcoordinates [204]. Many manifold learning models such asLocally Linear Embedding (LLE), ISOmetric feature MAPping(ISOMAP), Laplacian Eigenmaps, Reimanian manifold learn-ing are available. Though none of them has been appliedto the issue of race classification, the methodology appliesquite naturally. Generally, if we consider a set of training facesequences of C race groups. The procedure of race manifoldlearning and classification can be briefed as defining the facemanifold of the ethnic group from training data and testnew data by computing the predefined projection distance. Itshould be pointed out that not only race category information,but other demographic soft biometric information applies aswell, thus a multi-manifold learning framework could bebuilt. It is reasonable to see more literature on this researchdirection in the future.

7.2.3 Multi-modality frameworkThe current perspective is that human race categorizationinvolves multiple physical facial cues integration, each modal-ity encode compact feature which is specific to the repre-sentation of each module. In a variation of this view, oneracial face is subdivided into several racially discriminativecues, each of which is associated with particular reliability,availability, and their corresponding weights. This view toracial face classification conforms roughly to what we know ofneurophysiology and psychology. Therefore, extracting racialcues through either multi-sensor system or by multi-algorithmsystem would be the next logical step. One advantage ofsuch framework is that each physical trait can be assignedwith different weights concerning with the contribution tothe final decision level. For example, front face geometricfeature usually is more informative than skin color feature

while silhouette is even weaker, thus minor weights willbe assigned to the latter two traits, in contrast to the moreaccurate information. Such framework would by all meansreduce the false recognition rate. On the other hand, differentwith multi-modality system, the idea underlying a multi-algorithm system is that different features and/or matchingalgorithm will emphasize different aspects of the test subject,therefore their combination may give birth to an improvedperformance. Very recent empirical results from Scheirer etal [205] have confirmed the potential of Bayesian fusionframework by combining descriptive attributes.

7.3 New frontier: Emerging opportunities and challengesIn this section, we mainly focus on the future developmentof racial face recognition, referring to application potentialsthat come from different research directions. Moreover, wewill only attempt to shed light on one small corner of thisrapidly growing field; For instance, we will not cataloguethe many 3D racial-head-face model which might be helpfulto product design and comics, nor will we survey the in-triguing recent findings of racial face’s influence within sociallevel perception. Instead, we will build towards answeringthe following questions what are (what should be) the nextimportant research directions, and how do these topics giverise top performance in application tasks? These questionshave obvious relevance for our general understanding of boththeory-driven study of racial face recognition, and the useof racial cues to the future more comprehensive and robustcomputer vision systems in various applications.

7.3.1 From 2D to 3D facial dataWe acknowledge that current 2D racial face classificationsystems have already achieved ”satisfactory” performance (atsome extent) in constrained environments. However, theyface with difficulties when handling large amounts of facialvariations such as head pose, lighting conditions, and fa-cial expressions, especially when all these factors combinedtogether. For example, to our best knowledge, there is noresearch in current literature on race recognition with varyinglighting and viewpoint situations. Real world applications,such as video security surveillance system, may require theability of classifying the racial category of subject of interestcaptured in uncooperative environment by using PTZ camera,which also poses challenges to 2D based racial face recognitionmethods. 3D based system, on the other hand, are theoreticallyinsensitive to illumination and pose changes, therefore canbe viewed as potentially perfect way to further improvethe classification results. Furthermore, it has been shownby several demographic surveys and statistical studies thatintegrating 3D face information could lead to an improvedperformance of race categorization. 3D face data (there arealready several head-face demographic survey results basedon both high definition 3D scanners and cameras) can makesufficient use of the comprehensive physical structure of headand face, thus offering far more reliable information and beingrobust to those aforementioned drawbacks. However, theycould not be simply applied directly to the algorithms, featureextraction methods based on various criterion and proper reg-istration with mapping issues further call for a comprehensive



19

Fig. 19: Computational illustration for general race processing regions and a rough two level working scheme hierarchy: Primary level(R1) includes ventral visual pathway (from retina to IT/TEO, for object recognition and understanding), PCC and FFA which controlthe detection, categorization and automatic evaluation of race, this level is bottom-up. Higher level (R2) includes VLPFC/DLPFC/MPFC,amygdala, ACC/PCC, it controls social interaction and application of race perception, which might exert top-down regulation to the R1 level.Among which, amygdala controls emotion regulation with attitudes, belief, and social decision-making. ACC/DLPC/VLPFC are responsiblefor behavior control, such as race-biased response or stereotype, and prejudice. MPFC deals with inference with interaction among racegroups. Selective attention exerts top-down administration. At present, most computational models have been focused on ventral pathway,and detailed knowledge about the precise role of cortical regions involved in racial cortex part is still lacking.

framework linking physical anthropology and geometric ap-pearance based computer vision methods together, providingsupervised guidelines for these algorithms. What’s more, evenif 3D face models are theoretically insensitive to illuminationchanges, they still need to be properly registered before thematching step. Natural outdoor lighting has proven to be verydifficult to handle, not simply because of the strong shadowscast by a light source such as the sun, but also because subjectstend to distort their faces when illuminated by a strong source,which may fail the 3D model. Moreover, the issue of facialexpression inuences (for example, a surprised Asian femaleface) is even more difficult than that in 2D modality, because3D face models provide the exact shape information of facialsurfaces. There is still long way to go before any satisfactoryresults can be obtained at meaningful scale.

7.3.2 From single to comprehensive analysis

It must be pointed out that in an effort to study active anddynamic facial information, race, including other basic facialcomponent analysis, must be embedded into a more com-prehensive framework. In cognitive psychology, race effecthas been reported to affect facial expression and emotion byobservations that emotional faces are not universal and tendto be culture dependent instead. A series of investigationson the impact of recognition accuracy cross training andmatching on cross races in [157], [97], [61], [98], [99], [100],[206], [207] further explored and confirmed the existence ofinherent bias phenomenon on face recognition. In particu-lar, experimental results in [26], [27], [28], [29], [58], [208],[209], [210], [157] have shown that both commercial andnon-trainable face recognition algorithms consistently tend toperform lower on the same cohorts (females, blacks, age group18-30), which indicates that it is possible to improve overallface recognition accuracy by either separately fusing multiple

weak recognition algorithms (e.g. race sensitive algorithmstrained on different demographic cohorts), or by setting upa comprehensive, balanced face database. Gender perceptiondiffers across race, indicating the existence of culture-specificstereotypes and concepts of gender [211] (But see [212]).Strong “other-race effect” was found in age estimation [213].While in turn, all those facial cues will interact and modulatethe race perception as well. For instance, Guo et al [137] foundthat race classification accuracies tend to be reduced up to6%-8% in average when female faces are used for trainingwhile males for testing. We could also notice that to ourbest knowledge, there is no attempt in the existing literatureto recognize race under emotional status. These dynamicsimply the potential of extending accounts of face analysis asa single model categorization, in that multi-modality analysisare not simply addictive and uncorrelated, but interactivelydetermine one person’s individual identification and socialcommunication status, which definitely shed lights on thedesign of more intelligent face processing systems. Unfortu-nately, seldom studies have examined the interactive effectamong those discriminative cues. It therefore makes perfectsense that more detailed investigations and research will beexpected in near future.

7.3.3 From computational intelligence to computational neu-roscience

The inspiration of this subsection comes directly from thefact that recent advances in computational modeling of theprimate visual system has show deep insights of potentialrelevance to some of the challenges that computer vision com-munity is facing, such as face recognition/categorization incomplex environment, motion detection and human behavioranalysis. The problems of race perception and classificationis closely related to cognitive inference and pattern recog-



20nition on these behavior data. While previous contributionsare appreciated, the field of race recognition still lacks com-putational guidance from high level cognitive supervision.Quantitative psychophysical and physiological experimentalevidences support the theory that the visual information pro-cessing in cortex can be modeled as a hierarchy of increasinglysophisticated, sparsely coded representations, along the visualpathway. Therefore, characterizing and modeling the func-tions along the hierarchy, from early or intermediate stagessuch as LGN, V1, are necessary steps for systematic studiesin higher level, more cognitive tasks of race recognition. Thereare several cortex-like models proposed in recent decades[73], [74], [75], [214], [215], [216], [217]. Combined with theillustration shown in Fig. 19. Indeed, the very recent pioneer-ing work of Ahmed et al [116] used hierarchial feed-forwardconvolutional neural network model with transfer learning toperform race classification. Being simple to implement andadaptable, the 3-class race classification on FRGC v2.0 datasetobtained performance up to 94%. It is therefore very likelythat near future will see the emergence of more computation-ally intelligent models emulating brain-like process for raceperception and processing.

Another noteworthy fact is that most existing algorithmshave focused on commonly used feature extraction methods.Very recent behavior experimental results [218], [219], [35],[220], [221] suggest that race category can be perceived byselective attention (see Fig. 19). A promising direction forfuture research, therefore, is the development of recognitionmodels that take visual attention models and eye trackingdata into account. However, there is no such principled com-putational understanding of race by explicit attention model,which should also be clarified in the future. In an effort toexplore the possibility of how computational neurosciencecan help us build better computer vision and machine learn-ing systems on race classification, the solutions go beyondthe scope of computer vision field and require collaborationfrom multi-disciplinary communities. In summary, our surveywork suggest a more nuance of understanding of how raceclassification functions both in perception and in computervision, which may inspire new approaches to higher-levelsocial categorization and perception.

7.3.4 From single race to mixed race

Last but not least, as worldwide racial mixing accelerates,any definitive identification of pure race based on physicalappearance will undoubtedly become more problematic andless explicit. The result is the emergence of more racially-mixed people who tend to be more average looking, inaesthetical way. However, categorization of such kind ofpeople usually leads to unnecessary obfuscation, makingthe truly accurate recognition task even more discreditedthan the illusive definition of race itself. That is why racerecognition is much more difficult than other facial traitrecognitions. Mixed-racial people, also sociologically termedas multiracial people, are referring people is made up of orrelating to people of many races. The most commonly mixed-racial people we may encounter are biracial which meansconsisting of, representing, or combining members of twoseparate races. The classification of mixed race has its unique

importance to the overall race classification framework. Forexample, this could be of help in next generation design ofhuman-computer-interface (HCI) system. A computer trainedby existing monoracial categories to categorize multiracialpeople may cause unnecessary trouble in such situation,which implies the importance of the capability of machineto make categorizations of multiracial people as multiracial.Therefore, simply assigning any logically complete, consistent,and determinable categorical labels to these mixed-racial faceswill fail to make any kind of objective and substantial sense.As the dichotomy vanishes gradually in these multiracialsubjects, the classification problem will be different fromregular race recognition, since traditional racial categoriza-tion is viewed as dichotomous, as an ”either/or” question(One subject could be either Asian, Latino, Caucasian, etc).This judgment is then made difficult by the ambiguity of amultiracial person confronting them. However, very recentreports [222], [223] indicate that multiracial people could beharder yet still recognizable. There are two possible ways towork along this direction to solve the problem. One is tofollow traditional face recognition approaches. A multiracialdataset could be established, including as many multiracialpeople as possible with specific labels such as Asian-white,Asian-Latino. Statistically discriminative features (LBP, HOG,etc.) are expected to be extracted by training enough data,then a classifier (could be a Decision Tree, kNN, neural net-work, Bayesian method, etc.) can be learned for testing whengiven new images, using fuzzy-based rules and membershipfunctions is also a promising technique to describe the roughcategorization of subjects. These categories could be includedinto traditional race datasets, and new trained classifier couldbe aware of other than single race output (Asian, Caucasian,etc.), there could be other biracial output as well; Anotherpossible way is to build specific 3D model of multiracialpeople using anthropometric data and landmarks. Althoughtill now there has been little to no theoretical development onthis direction. It is clear that either way would need strongbackup from sufficiently large-scale dataset, which is also whywe call for new comprehensive dataset in previous section.

8 CONCLUSIONS

Race is in the face. We notice it both from embedded ex-plicit physical appearances and implicit cognitive process.Thinking categorically about the social ethnical groups makesit possible to perform race categorization by using patternanalysis algorithms and computationally intelligent systems.Over the decades, tremendous progresses have been made inour understanding of the computational aspects of racial facelearning. This survey provides a comprehensive and criticalreview of the state-of-the-art advances in facial image-basedrace perception, synthesis and recognition topics. We beginthis review by describing the concept of race and severaldisciplines of relevance. Then we present recent works thathighlight important systems and algorithms closely associatedwith race recognition, before finally considering extensionssuggested by studies into more complex inferences and ap-plications. Advancement in this direction could greatly helpsolving other related challenging vision and cognition prob-lems.



21REFERENCES

[1] A. J. Calder, G. Rhodes, M. Johnson, and J. V. Haxby, “Oxford handbookof face perception,” 2011.

[2] R. Kurzban, J. Tooby, and L. Cosmides, “Can race be erased? coalitionalcomputation and social categorization,” Proceedings of the NationalAcademy of Sciences, vol. 98, no. 26, pp. 15 387–15 392, 2001.

[3] V. Bruce and A. W. Young, “Understanding face recogition,” BritishJournal of Psychology, vol. 77, pp. 305–327, 1986.

[4] A. J. Calder and A. W. Young, “Understanding the recognition of facialidentify and facial expression,” Nature Reviews Neuroscience, vol. 6, pp.641–651, 2005.

[5] C. N. Macrae and G. V. Bodenhausen, “Social cognition: Thinkingcategorically about others,” Annual review of psychology, vol. 51, no. 1,pp. 93–120, 2000.

[6] A. J. O’Toole, D. A. Roark, and H. Abdi, “Recognizing moving faces: Apsychological and neural synthesis,” Trends in cognitive sciences, vol. 6,no. 6, pp. 261–266, 2002.

[7] J. Freeman, N. Ambady, and P. Holcomb, “The face-sensitive n170encodes social category information,” NeuroReport, vol. 21, no. 1, pp.24–28, 2010.

[8] D. T. Levine and B. L. Angelone, “Categorical perception of race,”Perception, vol. 31, pp. 567–578, 2002.

[9] D. T. Levine, “Classifying faces by race: The structure of face cate-gories,” Journal of Experimental Psychology: Learning, Memory and Cogni-tion, vol. 22, pp. 1364 – 1382, 1996.

[10] M. Brewer and R. Brown, “Intergroup relations,” MacGraw-Hill Press,1998.

[11] J. L. Eberhardt, N. Dasgupta, and T. L. Banaszynski, “Believing is see-ing: The effects of racial labels and implicit beliefs on face perception,”Personality and Social Psychology Bulletin, vol. 29, pp. 360–412, 2003.

[12] S. Bentin and L. Y. Deouell, “Structural encoding and identificationin face processing: Erp evidence for separate mechanisms,” CognitiveNeuropsychology, vol. 17, pp. 35–54, 2000.

[13] B. Rossion and C. Jacques, “N170: Understanding the time course offace perception in the human brain,” International Journal of Neuroscience,vol. 112, pp. 1499–1512, 2002.

[14] B. Bradtmiller and M. Friess, “A head-and-face anthropometric surveyof us respirator users,” Anthrotech technical report for NIOSH/NPPTL,2004.

[15] Z. Zhuang and B. Bradtmiller, “Head-and-face anthropometric surveyof u.s. respirator users,” Journal of Occupational and Environmental Hy-giene, vol. 2, no. 11, pp. 567–576, 2005.

[16] Z. Zhuang, D. E. Slice, S. Benson, S. Lynch, and D. J. Viscusi, “Shapeanalysis of 3d head scan data for u.s. respirator users,” EURASIP Journalon Advances in Signal Processing, vol. 3, pp. 1–10, 2010.

[17] R. M. Ball and J. F. M. Molenbroek, “Measuring chinese heads andfaces,” Proceedings of the 9th international congress of physiological anthro-pology. Human diversity: design for life, pp. 150–155, 2008.

[18] R. Ball, C. Shu, P. Xi, M. Rioux, Y. Luximon, and J. Molenbroek,“A comparison between chinese and caucasian head shapes,” AppliedErgonomics, vol. 41, no. 6, pp. 832–839, 2010.

[19] Y. Luximon, R. Ball, and L. J. L, “The chinese face: A 3d anthropometricanalysis,” Proceedings of the TMCE2010 Symposium,, pp. 255–266, 2010.

[20] D. Schiller, J. B. Freeman, J. P. Mitchell, J. S. Uleman, and E. A. Phelps,“A neural mechanism of first impressions,” Nature Neuroscience, vol. 12,no. 4, pp. 508–515, 2009.

[21] H. Wiese, “The role of age and ethnic group in face recognitionmemory: Erp evidence from a combined own-age and own-race biasstudy,” Biological Psychology, vol. 89, no. 1, pp. 137–147, 2012.

[22] K. Hugenberg, S. G. Young, M. J. Bernstein, and D. F. Sacco, “Thecategorization-individuation model: an integrative account of the other-race recognition deficit,” Psychology Review, vol. 117, no. 4, pp. 1168–87,2010.

[23] K. B. Maddox, “Perspectives on racial phenotypicality bias,” PersonalitySocial Psychology Review, vol. 8, no. 4, pp. 383–401, 2004.

[24] R. S. Bigler and L. S. Liben, “A developmental intergroup theory ofsocial stereotypes and prejudice,” Advances in child development andbehavior, vol. 34, pp. 39–89, 2006.

[25] C. R. Jones and R. H. Fazio, “Person categorization and automatic racialstereotyping effects on weapon identification,” Personality and SocialPsychology Bulletin, vol. 36, no. 8, pp. 1073–85, 2010.

[26] P. Phillips, P. Flynn, T. Scruggs, K. Bowyer, J. Chang, K. Hoffman,J. Marques, J. Min, and W. Worek, “Overview of the face recognitiongrand challenge,” Proc. IEEE Conf. Computer Vision and Pattern Recogni-tion, pp. 947–954, 2005.

[27] P. Phillips, W. Scruggs, A. OToole, P. Flynn, K. Bowyer, C. Schott, andM. Sharpe, “Frvt 2006 and ice 2006 large-scale experimental results,”IEEE Transactions on Pattern Recognition and Machine Intelligence (PAMI),vol. 32, no. 5, pp. 831–846, 2010.

[28] K. Bowyer, K. Chang, and P. Flynn, “A survey of approaches andchallenges in 3d and multi-modal 3d+2d face recognition,” ComputerVision and Image Understanding, vol. 101, no. 1, pp. 1–15, 2006.

[29] K. Chang, K. Bowyer, and P. Flynn, “An evaluation of multi-modal2d+3d face biometrics,” IEEE Trans. Pattern Analysis and Machine Intel-ligence, vol. 27, no. 4, pp. 619–624, 2005.

[30] D. T. Levine, “Race as a visual feature: Using visual search and percep-tual discrimination tasks to understand face categories and the cross-race recognition deficit,” Journal of Experimental Psychology: General, vol.129, pp. 559 – 574, 2000.

[31] P. Ekman, E. R. Sorenson, and W. V. Friesen, “Pan-cultural elements infacial displays of emotion,” Science, vol. 164, no. 3875, pp. 86–88, 1969.

[32] C. A. Meissner and J. C. Brigham, “Thirty years of investigating theown-race bias in memory for faces: A meta-analytic review,” Psychology,Public Policy, and Law, vol. 7, pp. 3–35, 2001.

[33] S. G. Young, K. Hugenberg, M. J. Bernstein, and D. F. Sacco, “Perceptionand motivation in face recognition: a critical review of theories of thecross-race effect,” Personality and Social Psychology Review, vol. 16, no. 2,pp. 116–42, 2011.

[34] J. T. Kubota, M. R. Banaji, and E. A. Phelps, “The neuroscience of race,”Nature Neuroscience, vol. 15, no. 7, pp. 940–948, 2012.

[35] L. Cosmides, J. Tooby, and R. Kurzban, “Perceptions of race,” Trends incognitive sciences, vol. 7, no. 4, pp. 173–179, 2003.

[36] S. Z. Li and A. K. Jain, “Handbook of face recognition,” Springer Press,2004.

[37] A. F. Abate, M. Nappi, D. Riccio, and G. Sabatino, “2d and 3d facerecognition: A survey,” Pattern Recognition Letters, vol. 28, no. 14, pp.1885–1906, 2007.

[38] A. Godil, S. Ressler, and P. Grother, “Face recognition using 3d surfaceand color map information: Comparison and combination,” Symposiumon Biometrics Technology for Human Identification, 2004.

[39] Z. H. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, “A surveyof affect recognition methods: Audio, visual, and spontaneous expres-sions,” IEEE Transactions on Pattern Analysis and Machine Intelligence,vol. 31, no. 1, pp. 39–58, 2009.

[40] Y. Fu, G. D. Guo, and T. S. Huang, “Age synthesis and estimationvia faces: a survey,” IEEE Transactions on Pattern Analysis and MachineIntelligence, vol. 32, no. 11, pp. 1955–1976, 2010.

[41] G. D. Guo, “Age estimation and sex classification,” Video Analytics forBusiness Intelligence, pp. 101–131, 2012.

[42] N. Ramanathan, R. Chellappa, and S. Biswas, “Age progression inhuman faces: A survey,” Visual Languages and Computing, 2009.

[43] C. B. Ng, Y. H. Tay, and B. M. Goi, “Vision-based human genderrecognition: A survey,” arXiv preprint arXiv:1204.1611, 2012.

[44] C. Anitha, M. K. Venkatesha, and B. S. Adiga, “A survey on facialexpression databases,” International Journal of Engineering Science andTechnology, vol. 2, no. 10, pp. 5158–5174, 2010.

[45] Race Wiki, http://en.wikipedia.org/wiki/Race human classification.[46] E. Britannica, “Race,” Encyclopedia Britannica Ultimate Reference

Suite, Chicago: Encyclopedia Britannica, 2012.[47] H. Hill, V. Bruce, and S. Akamatsu, “Perceiving the sex and race of

faces: the role of shape and color,” Proceedings of the Royal Society ofLondon. Series B: Biological Sciences, vol. 261, no. 1362, pp. 367–373, 1995.

[48] Y. Xie, K. Luu, and M. Savvides, “A robust approach to facial ethnicityclassification on large scale face databases,” Proceedings of IEEE Interna-tional Conference on Biometrics: Theory, Applications and Systems (BTAS),pp. 143–149, 2012.

[49] S. M. M. Roomi, S. Virasundarii, S. Selvamegala, S. Jeevanandham,and D. Hariharasudhan, “Race classification based on facial features,”in National Conference of Computer Vision, Pattern Recognition, ImageProcessing and Graphics (NCVPRIPG), 2011, pp. 54–57.

[50] K. R. Brooks and O. S. Gwinn, “No role for lightness in the perceptionof black and white? simultaneous contrast affects perceived skin tone,but not perceived race,” Perception, vol. 39, no. 8, pp. 1142–1145, 2010.

[51] L. Yin, J. Loi, and J. Jia, “Towards race-related face identification: re-search on skin color transfer,” Proceedings of IEEE International Conferenceon Automatic Face and Gesture Recognition, 2004.

[52] G. Anzures, O. Pascalis, P. C. Quinn, A. M. Slater, and K. Lee, “Minimiz-ing skin color differences does not eliminate the own-race recognitionadvantage in infants,” Infancy, vol. 16, no. 6, pp. 640–654, 2011.

[53] M. A. Strom, L. A. Zebrowitz, S. Zhang, P. M. Bronstad, and H. K. Lee,“Skin and bones: The contribution of skin tone and facial structure toracial prototypicality ratings,” PLoS ONE, vol. 7, no. 7, pp. 1–8, 2012.



22[54] R. Akbari and S. Mozaffari, “Fuzzy moments method for face recog-

nition in an ethnic database,” International Journal of Signal Processing,Image Processing and Pattern Recognition, vol. 5, no. 1, pp. 85–97, 2012.

[55] C. Coon, in The Origins of Races. New York: Alfred A. Knopf, 1962.[56] G. Barbujani, “Human races: classifying people vs understanding di-

versity,” Current Genomics, vol. 6, no. 4, pp. 215–226, 2005.[57] S. Haslanger, “Gender and race:(what) are they?(what) do we want

them to be?” Nous, vol. 34, no. 1, pp. 31–55, 2000.[58] M. H. Papesh and S. D. Goldinger, “A multidimensional scaling anal-

ysis of own- and cross-race face spaces,” Cognition, vol. 116, no. 2, pp.283–288, 2010.

[59] A. J. O’Toole and V. Natu, “Computational perspectives on the other-race effect,” Visual Cognition, no. ahead-of-print, pp. 1–17, 2013.

[60] P. J. Phillips, F. Jiang, A. Narvekar, J. H. Ayyad, and A. J. O’Toole, “Another-race effect for face recognition algorithms,” ACM Transactions onApplied Perception, vol. 8, no. 2, pp. 1–11, 2011.

[61] M. N. Dailey, M. J. Lyons, H. Ishi, C. Joyce, M. Kamachi, J. Gyoba, andG. W. Cottrell, “Evidence and a computational explanation of culturaldifferences in facial expression recognition,” Emotion, vol. 10, no. 6, pp.874–893, 2010.

[62] R. E. Jack, O. G. B. Garrod, Y. Hui, R. Caldara, and P. G. Schyns, “Facialexpressions of emotion are not culturally universal,” Proceedings of theNational Academy of Sciences (PNAS), vol. 109, no. 19, pp. 7241–7244,2012.

[63] R. E. Jack, C. Blais, C. Scheepers, P. G. Schyns, and R. Caldara, “Culturalconfusions show that facial expressions are not universal,” CurrentBiology, vol. 19, pp. 1543–1548, 2009.

[64] D. Enlow and M. G. Hans, “Facial growth,” 3rd edition, Saunders, 1990.[65] L. G. Farkas, “Anthropometry of the head and face,” New York (USA):

Raven Press, 1994.[66] L. G. Farkas, M. J. Katic, and C. R. Forrest, “International anthropomet-

ric study of facial morphology in various ethnic groups/races,” Journalof Craniofac Surgery, vol. 16, no. 4, pp. 615–46, 2005.

[67] F. Fang, P. J. Clapham, and K. C. Chung, “A systematic review of inter-ethnic variability in facial dimensions,” Plastic and reconstructive surgery,vol. 127, no. 2, pp. 874–881, 2011.

[68] J. Y. Wong, A. K. Oh, E. Ohta, A. T. Hunt, G. F. Rogers, J. B. Mulliken,and C. K. Deutsch, “Validity and reliability of craniofacial anthropo-metric measurement of 3d digital photogrammetric images,” The CleftPalate-Craniofacial Journal, vol. 45, no. 3, pp. 232–239, 2008.

[69] T. A. Ito, E. Thompson, and J. T. Cacioppo, “Tracking the timecourseof social perception: The effects of racial cues on event-related brainpotentials,” Personality and Social Psychology Bulltein, vol. 30, no. 10, pp.1267–1280, 2004.

[70] R. H. Ofan, N. Rubin, and D. M. Amodio, “Seeing race: N170 responsesto race and their relation to automatic racial attitudes and controlledprocessing,” Journal of Cognitive Neuroscience, vol. 23, no. 10, pp. 3153–3161, 2011.

[71] J. L. Eberhardt, “Imaging race,” American Psychologist, vol. 60, no. 2, p.181, 2005.

[72] R. Love, “fmri used to study race effect on face recognition,” The Lancet,vol. 358, no. 9280, p. 476, 2001.

[73] T. Serre, “Learning a dictionary of shape-components in visual cortex:Comparison with neurons, humans and machines,” PhD Thesis, Mas-sachusetts Institute of Technology, 2006.

[74] S. Ullman, M. Vidal-Naquet, , and E. Sali, “Visual features of intermde-diate complexity and their use in classification,” Nature Neuroscience,vol. 5, no. 7, pp. 682–687, 2002.

[75] M. Riesenhuber and T. Poggio, “Hierarchical models of object recogni-tion in cortex,” Nature Neuroscience, vol. 2, pp. 1019 – 1025, 1999.

[76] T. A. Ito and B. D. Bartholow, “The neural correlates of race,” Trendson Cognitive Science, vol. 13, no. 12, pp. 524–31, 2009.

[77] J. L. Brebnera, “Determinants of the own-race bias and neural correlatesof own- and other-race face processing,” Ph.D dissertation, University ofAberdeen, 2011.

[78] E. A. Phelps, K. J. O’Connor, W. A. Cunningham, E. S. Funayama,J. C. Gatenby, J. C. Gore, and M. R. Banaji, “Performance on indirectmeasures of race evaluation predicts amygdala activation,” Journal ofCognitive Neuroscience, vol. 12, no. 5, pp. 729–738, 2000.

[79] E. A. Phelps, C. J. Cannistraci, and W. A. Cunningham, “Intact per-formance on an indirect measure of race bias following amygdaladamage,” Neuropsychologia, vol. 41, no. 2, pp. 203–208, 2003.

[80] M. D. Lieberman, A. Hariri, J. M. Jarcho, N. I. Eisenberger, and S. Y.Bookheimer, “An fmri investigation of race-related amygdala activityin african-american and caucasian-american individuals,” Nature Neu-roscience, vol. 8, no. 6, pp. 720–722, 2005.

[81] J. Ronquillo, T. F. Denson, B. Lickel, Z. L. Lu, A. Nandy, and K. B.Maddox, “The effects of skin tone on race-related amygdala activity: an

fmri investigation,” Social Cognitive Affective Neuroscience, vol. 2, no. 1,pp. 39–44, 2007.

[82] A. J. Hart, P. J. Whalen, L. M. Shin, S. C. McInerney, H. Fischer, andS. L. Rauch, “Differential response in the human amygdala to racialoutgroup vs ingroup face stimuli,” Neuroreport, vol. 3, no. 11, pp. 2351–2315, 2000.

[83] W. A. Cunningham, M. K. Johnson, C. L. Raye, J. C. G. ad J. C. Gore,and M. R. Banaji, “Separable neural components in the processing ofblack and white faces,” Psychological Science, vol. 15, no. 12, pp. 806–813,2004.

[84] I. Gauthier, M. J. Tarr, J. Moylan, P. Skudlarski, J. C. Gore, and A. W.Anderson, “The fusiform face area is part of a network that processsfaces at the individual level,” Journal of Cognitive Neuroscience, vol. 12,no. 3, pp. 495–504, 2000.

[85] A. J. Golbym, J. D. Gabrieli, J. Y. Chiao, and J. L. Eberhardt, “Differentialresponses in the fusiform region to same-race and other-race faces,”Nature Neuroscience, vol. 4, no. 8, pp. 845–850, 2001.

[86] C. Kaul, K. G. Ratner, and J. J. Van Bavel, “Dynamic representations ofrace: processing goals shape race decoding in the fusiform gyri,” Socialcognitive and affective neuroscience, p. nss138, 2012.

[87] R. S. Bigler and L. S. Liben, “Developmental intergroup theory: Explain-ing and reducing children’s social stereotyping and prejudice,” CurrentDirections in Psychological Science, vol. 16, pp. 162–166, 2007.

[88] D. A. Stanley, P. Sokol-Hessner, M. R. Banaji, and E. A. Phelps, “Implicitrace attitudes predict trustworthiness judgments and economic trustdecisions,” Proceedings of National Academy of Science (PNAS), vol. 108,no. 19, pp. 7710–7715, 2011.

[89] K. Hugenberg and D. F. Sacco, “Social categorization and stereotyping:How social categorization biases person perception and face memory,”Social and Personality Psychology Compass, vol. 2, no. 2, pp. 1052–1072,2008.

[90] K. J. Johnson and B. L. Fredrickson, “We all look the same to me,positive emotions eliminate the own-race bias in face recognition,”Psychological Science, vol. 16, no. 11, pp. 875–881, 2005.

[91] R. J. Dolan, “Emotion, cognition, and behavior,” Science, vol. 298, no.5596, pp. 1191–1194, 2002.

[92] O. Corneille, K. Hugenberg, and P. Timothy, “Applying the attractorfield model to social cognition: Perceptual discrimination is facilitated,but memory is impaired for faces displaying evaluatively congruentexpressions,” Journal of Personality and Social Psychology, vol. 93, no. 3,pp. 335–352, 2007.

[93] A. T. Hirsh, A. F. Alqudah, L. A. Stutts, and M. E. Robinson, “Virtualhuman technology: Capturing sex, race, and age influences in individ-ual pain decision policies,” Pain, vol. 140, pp. 231–238, 2009.

[94] D. A. Wilbraham, J. C. Christensen, A. M. Martinez, and J. T. Todd, “Canlow level image differences account for the ability of human observersto discriminate facial identity?” Journal of Vision, vol. 8, no. 15, p. 5,2008.

[95] G. Rhodes, “Looking at faces: First-order and second-order features asdeterminants of facial appearance,” Perception, vol. 17, pp. 43–63, 1988.

[96] U. Tariq, Y. Hu, and T. S. Huang, “Gender and ethnicity identificationfrom silhouetted face profiles,” Proceedings of IEEE International Confer-ence on Image Processing (ICIP), pp. 2441–2444, 2009.

[97] A. J. Otoole, H. Abdi, K. A. Deffenbacher, and J. C. Bartlett, “Classifyingfaces by race and sex using an autoassociative emory trained forrecognition,” Proceedings of the 13th Annual Conference ofthe CognitiveScience Society, pp. 847–851, 1991.

[98] A. J. Otoole, H. Abdi, K. A. Deffenbacher, and D. Valentin, “Low-dimensional representation offaces in higher dimensions of the facespace,” Journal of the Optical Society of America A, vol. 10, pp. 405–410,1993.

[99] A. J. Otoole, K. A. Deffenbacher, H. Abdi, and J. C. Bartlett, “Simulatingthe ‘other-race’ effect as a problem in perceptual learning,” Journal ofNeural Computing, Anificial Intelligence, and Cognitive Research, vol. 3,no. 2, pp. 163–178, 1991.

[100] ——, “Structural aspects of face recognition and the other-race effect,”Memory and Cognition, vol. 22, no. 2, pp. 208–224, 1994.

[101] V. Natu, D. Raboy, and A. J. OToole, “Neural correlates of own-andother-race face perception: Spatial and temporal response differences,”Neuroimage, vol. 54, no. 3, pp. 2547–2555, 2011.

[102] H. H. K. Tin and M. M. Sein, “Race identification for face images,”ACEEE International Journal on Information Technology, vol. 01, no. 02,2011.

[103] A. Ahmad, A. Ashraf, and A. S. Walid, “Arabic race classification offace images,” International Journal of Computers and Technology, vol. 4,no. 2, 2013.

[104] C. Liu, H. Y. Shum, and W. T. Freeman, “Face hallucination: Theoryand practice,” International Journal of Computer Vision, vol. 75, no. 1, pp.115–134, 2007.



23[105] Y. S. Ou, X. Y. Wu, H. H. Qian, and Y. S. Xu, “A real time race

classification system,” Proceedings of IEEE International Conference onInformation Acquisition, pp. 1–6, 2005.

[106] B. Klare and A. K. Jain, “On a taxonomy of facial features,” in 2010Fourth IEEE International Conference on Biometrics: Theory Applicationsand Systems (BTAS), 2010, pp. 1–8.

[107] X. D. Duan, C. R. Wang, X. D. Liu, Z. J. Li, J. Wu, and H. L. Zhang,“Ethnic features extraction and recognition of human faces,” Proceedingsof IEEE International Conference on Advanced Computer Control, vol. 2, pp.125–130, 2010.

[108] H. Lin, H. C. Lu, and L. H. Zhang, “A new automatic recognitionsystem of gender, age and ethnicity,” Proceedings of World Congress onIntelligent Control and Automation, 2006.

[109] J. Heo and S. Marios, “Gender and ethnicity specific generic elasticmodels from a single 2d image for novel 2d pose face synthesisand recognition,” IEEE Transactions on Pattern Analysis and MachineIntelligence, vol. 34, no. 12, pp. 2341–2350, 2012.

[110] F. S. Manesh, M. Ghahramani, , and Y. P. Tan, “Facial part displace-ment effect on template-based gender and ethnicity classification,”Proceedings of IEEE Conference on Control Automation Robotics and Vision(ICARCV), pp. 1644–1649, 2010.

[111] C. Zhong, Z. N. Sun, and T. N. Tan, “Fuzzy 3d face ethnicity cate-gorization,” Proceedings of IEEE International Conference on Advances inBiometrics, 2009.

[112] S. Y. Fu, G. S. Yang, and Z. G. Hou, “Spiking neural networks basedcortex like mechanism: A case study for facial expression recognition,”in Proceedings of IEEE International Joint Confernce of Neural Networks(IJCNN), pp. 1637–1642, 2011.

[113] G. Muhammad, M. Hussain, and F. Alenezy, “Race classification fromface images using local descriptors,” International Journal on ArtificialIntelligence Tools, vol. 21, no. 5, pp. 1–24, 2012.

[114] ——, “Race recognition from face images using weber local descriptor,”IEEE International Conference on Systems, Signals and Image Processing(IWSSIP), pp. 421–424, 2012.

[115] S. H. Salah, H. Du, and N. Al-Jawad, “Fusing local binary patternswith wavelet features for ethnicity identification,” Proceddings of IEEEInternational Conference on Signal and Image Processing, vol. 21, no. 5, pp.416–422, 2013.

[116] A. Ahmed, K. Yu, W. Xu, Y. Gong, and E. Xing, “Training hierarchicalfeed-forward visual recognition models using transfer learning frompseudo-tasks,” in Proceedings of European Conference of Computer Vision(ECCV). Springer, 2008, pp. 69–82.

[117] N. Kumar, A. Berg, P. Belhumeur, and S. Nayar, “Describable visualattributes for face verification and image search,” IEEE Trans. PatternAnalysis and Machine Intelligence, vol. 33, no. 10, pp. 1962–1977, 2011.

[118] T. Berg and P. N. Belhumeur, “Poof: Part-based one-vs-one featuresfor fine-grained categorization, face verification, and attribution esti-mation,” Proceedings of IEEE Conference of Computer Vision and PatternRecognition (CVPR), 2013.

[119] H. K. Moon, R. Sharma, and N. S. Jung, “Method and system for robusthuman ethnicity recognition using image feature-based probabilisticgraphical models,” US Patent, no. 8379937, p. B1, 2013.

[120] H. X. Ding, D. Huang, Y. H. Wang, and L. M. Chen, “Facial ethnicityclassification based on boosted local texture and shape descriptions,”Proceedings of IEEE International Conference and Workshops on AutomaticFace and Gesture Recognition, pp. 1–6, 2013.

[121] P. Sinha, B. Balas, Y. Ostrovsky, and R. Russell, “Face recognition byhumans: Nineteen results all computer vision researchers should knowabout,” Proceedings of the IEEE, vol. 94, no. 11, pp. 1948–1962, 2006.

[122] M. Wagner-Schuman, A. M. Dubis, R. N. Nordgren, Y. Lei, D. Odell,H. Chiao, E. Weh, W. Fischer, Y. Sulai, A. Dubra et al., “Race-and sex-related differences in retinal thickness and foveal pit morphology,”Investigative ophthalmology & visual science, vol. 52, no. 1, pp. 625–634,2011.

[123] X. Li, W. L. Wong, C. Y.-l. Cheung, C.-Y. Cheng, M. K. Ikram, J. Li, K. S.Chia, and T. Y. Wong, “Racial differences in retinal vessel geometriccharacteristics: A multiethnic study in healthy asians,” Investigativeophthalmology & visual science, vol. 54, no. 5, pp. 3650–3656, 2013.

[124] X. C. Qiu, Z. N. Sun, , and T. N. Tan, “Global texture analysis of irisimages for ethnic classification,” Advance in Biometrics, pp. 411–418,2005.

[125] X. C. Qiu, Z. N. Sun, and T. N. Tan, “Learning appearance primitivesof iris images for ethnic classification,” in Proceedings of IEEE onInternational Conference of Image Processing (ICIP), pp. 405–408, 2007.

[126] A. Zarei and D. Mou, “Artificial neural network for prediction ofethnicity based on iris texture,” in Proceedings of 11th InternationalConference on Machine Learning and Applications (ICMLA),, vol. 1, 2012,pp. 514–519.

[127] H. Zhang, Z. Sun, T. Tan, and J. Wang, “Ethnic classification based oniris images,” in Biometric Recognition, 2011, pp. 82–90.

[128] S. Lagree and K. W. Bowyer, “Ethnicity prediction based on iris texturefeatures,” in in Proceedings of 22th Midwest Artificial Intelligence andCognitive Science Conference, 2011, pp. 225–230.

[129] L. Stark, K. W. Bowyer, and S. Siena, “Human perceptual categorizationof iris texture patterns,” in Proceeding of 2010 Fourth IEEE InternationalConference on Biometrics: Theory Applications and Systems (BTAS), 2010,pp. 1–7.

[130] S. Lagree and K. W. Bowyer, “Predicting ethnicity and gender fromiris texture,” in 2011 IEEE International Conference on Technologies forHomeland Security (HST), 2011, pp. 440–445.

[131] K. Hollingsworth, K. W. Bowyer, and P. J. Flynn, “Useful features forhuman verification in near-infrared periocular images,” Image and VisionComputing, 2012.

[132] J. R. Lyle, M. P. E, S. J. Pundlik, and D. L. Woodard, “Soft biometricclassification using periocular region features,” Proceedings of IEEE onBiometrics: Theory Applications and Systems (BTAS), 2010.

[133] Y.-h. Li, M. Savvides, and T. Chen, “Investigating useful and distin-guishing features around the eyelash region,” in 37th IEEE AppliedImagery Pattern Recognition Workshop, 2008, pp. 1–6.

[134] S. Gutta, H. Wechsler, and P. Phillips, “Gender and ethnic classifica-tion of face images,” Proceesings of IEEE on International Conference onAutomatic Face and Gesture Recognition, pp. 194–199, 1998.

[135] S. Gutta and H. Wechsler, “Gender and ethnic classification of humanfaces using hybrid classifiers,” Proceesings of IEEE on International JointConfernce of Neural Networks (IJCNN), vol. 6, pp. 4084–4089, 1999.

[136] ——, “Mixture of experts for classification of gender, ethnic origin, andpose of human faces,” IEEE Transactions on Neural Networks, vol. 11,no. 4, pp. 948–960, 2000.

[137] G. D. Guo and G. W. Mu, “A study of large-scale ethnicity estimationwith gender and age variations,” Proceedings on IEEE Computer SocietyScience on Computer Vision and Pattern Recognition Workshops (CVPRW),pp. 79–86, 2010.

[138] G. Guo and G. Mu, “Joint estimation of age, gender and ethnicity: Ccavs. pls,” in Automatic Face and Gesture Recognition (FG), 2013 10th IEEEInternational Conference and Workshops on. IEEE, 2013, pp. 1–6.

[139] X. Lu and A. K. Jain, “Ethnicity identification from face images,”Proceedings of SPIE Defense and Security Symposium, 2004.

[140] J. Li, “Ethicity classification using multiple features,” United StatesPatent, Dec.11 2012, uS 8331 698 B2.

[141] Y. H. Lei, Y. Y. Chen, B. C. Chen, L. Iida, and W. H. Hsu, “Where iswho: large-scale photo retrieval by facial attributes and canvas layout,”Proceedings of the 35th international ACM SIGIR conference on Research anddevelopment in information retrieval, pp. 701–710, 2012.

[142] X. Lu, H. Chen, , and A. K. Jain, “Multimodal facial gender andethnicity identification,” Proceedings of IEEE International Conference onBiometrics, pp. 554–561, 2006.

[143] G. Toderici, S. M. O’Malley, G. Passalis, T. Theoharis, and I. A. Kaka-diaris, “Ethnicity- and gender-based subject retrieval using 3-d face-recognition techniques,” International Journal of Computer Vision, vol. 89,no. 2, pp. 382–391, 2010.

[144] O. Ocegueda, T. Fang, S. K. Shah, and I. A. Kakadiaris, “3d facediscriminant analysis using gauss-markov posterior marginals,” PatternAnalysis and Machine Intelligence, IEEE Transactions on, vol. 35, no. 3, pp.728–739, 2013.

[145] S. Gupta, K. R. Castleman, M. K. Markey, and A. C. Bovik, “Texas 3dface recognition database,” IEEE Southwest Symposium on Image Analysisand Interpretation, Austin, TX, pp. 97–100, 2010.

[146] Z. B. Azouz, M. Rioux, C. Shu, and R. Lepage, “Characterizing humanshape variation using 3d anthropometric data,” The Visual Computer,vol. 22, no. 5, pp. 302–314, 2006.

[147] S. Berretti, A. D. Bimbo, and P. Pala, “3d face recognition usingisogeodesic stripes,” IEEE Transactions on Pattern Analysis and MachineIntelligence, vol. 32, no. 12, pp. 2162–2177, 2010.

[148] ——, “Distinguishing facial features for ethnicity-based 3d face recog-nition,” ACM Transactions on Intelligent Systems and Technology, vol. 3,no. 3, p. 45, 2012.

[149] S. Gupta, M. K. Markey, and A. C. Bovik, “Anthropometric 3d facerecognition,” International Journal of Computer Vision, vol. 90, no. 3, pp.331–349, 2010.

[150] S. R. Sukumar, H. Bozdogan, D. L. Page, A. Koschan, and M. A. Abidi,“Learning structurally discriminant features in 3d faces,” Proceedings ofIEEE International Conference on Image Processing, pp. 1912–1915, 2008.

[151] G. Shakhnarovich, P. A. Viola, and B. Moghaddam, “A unified learningframework for real time face detection and classification,” Proceedings ofIEEE International Conference on Automatic Face and Gesture Recognition,2002.



24[152] D. Zhang, Y. H. Wang, and B. Bhanu, “Ethnicity classification based on

gait using multi-view fusion,” IEEE Conference of Computer Vision andPattern Recognition Workshops (CVPRW), pp. 108 – 115, 2010.

[153] Samangooei, Sina, Nixon, and M. S, “Performing content-based re-trieval of humans using gait biometrics,” Multimedia Tools and Appli-cations, vol. 49, no. 1, pp. 195–212, 2010.

[154] D. Zhang, Y. H. Wang, and Z. X. Zhang, “Ethnicity classification basedon a hierarchical fusion,” Biometric Recognition, vol. 7701, pp. 300–307,2012.

[155] L. Zedong, D. Xiaodong, and Z. Qingling, “A novel survey based onmultiethnic facial semantic web,” TELKOMNIKA Indonesian Journal ofElectrical Engineering, vol. 11, no. 9, pp. 5076–5083, 2013.

[156] D. Zhang and Y. H. Wang, “Gender recognition based on fusion onface and gait information,” Proceedings of IEEE International Conferenceon Machine Learning and Cybernetics, vol. 1, pp. 62–67, 2008.

[157] B. F. Klare, M. Burge, J. Klontz, R. W. V. Bruegge, and A. K. Jain,“Face recognition performance: Role of demographic information,”IEEE Transactions on Information Forensics and Security, vol. 7, no. 6, pp.1789–1801, 2012.

[158] N. Bellustin, A. T. Kovalchuck, O. Shemagina, V. Yakho, Y. Kalafati,A. Vaish, P. Sharma, and S. Verma, “Instant human face attributesrecognition system,” International Journal Advanced Computer Science andApplications, 2011.

[159] B. Wu, H. Z. Ai, and C. Huang, “Facial image retrieval based on de-mographic classification,” Proceedings of IEEE on Internation Conferenceon Parttern Recognition (ICPR), vol. 3, 2004.

[160] S. Hosoi, T. Erina, and K. Masato, “Ethnicity estimation with facialimages,” in Proceedings of IEEE on Automatic Face and Gesture Recognition,2004.

[161] M. Demirkus, K. Garg, and S. Guler, “Automated person categorizationfor video surveillance using soft biometrics,” in Proceedings of SPIE,Biometric Technology for Human Identification VII, 2010.

[162] S. Y. Fu, G. S. Yang, and X. K. Kuai, “A spiking neural network basedcortex-like mechanism and application to facial expression recogni-tion,” Computational intelligence and neuroscience, vol. 7, pp. 19–33, 2012.

[163] P. J. Phillips, P. J. Rauss, and S. Z. Der, “Feret (face recognitiontechnology) recognition algorithm development and test results,” ArmyResearch Lab technical report 995, 1996.

[164] D. Riccio, G. Tortora, M. Marsico, and H. Wechsler, “Ega, ethnicity, gen-der and age, a pre-annotated face database,” in 2012 IEEE Workshop onBiometric Measurements and Systems for Security and Medical Applications(BIOMS), 2012, pp. 1–8.

[165] W. Gao, B. Cao, S. G. Shan, X. L. Chen, D. L. Zhou, X. H. Zhang, andD. B. Zhao, “The cas-peal large-scale chinese face database and baselineevaluations,” IEEE Trans on Sytem, Man, Cybernatics, Part A: Systems andHumans, vol. 38, no. 1, pp. 149–161, 2008.

[166] A. Bastanfard, M. A. Nik, and M. M. Dehshibi, “Iranian face databasewith age, pose and expression,” Proceedings of IEEE International Con-fernce on Machine Vision, pp. 50–58, 2007.

[167] R. C. Roh and S. W. Lee, “Performance evaluation of face recognitionalgorithms on korean face database,” International Journal of PatternRecognition and Artificial Intelligence, vol. 21, no. 6, pp. 1017–1033, 2007.

[168] B. W. Hwang, H. Byun, M. Roh, and S. W. Lee, “Performance evaluationof face recognition algorithms on the asian face database. kfdb,” Audio-and Video-Based Biometric Person Authentication (AVBPA), pp. 557–565,2003.

[169] “JAFEE dataset,” http://www.kasrl.org/jaffe.html.[170] “Cafe dataset,” http://www.cse.ucsd.edu/gary.[171] “Frgc face dataset,” http://face.nist.gov/frgc/.[172] T. Sim, S. Baker, and M. Bsat, “The cmu pose, illumination, and

expression database,” IEEE Transactions on Pattern Analysis and MachineIntelligence, vol. 25, no. 12, pp. 1615–1618, 2003.

[173] “BUDFE database,” www.cs.binghamton.edu/lijun/Research/3DFE.html.[174] “Cohn-kanade database,” http://vasc.ri.cmu.edu/idb/html/face/.[175] “Cohn-Kanada dataset+,” http://vasc.ri.cmu.edu/idb/html/face/facialexpression.html.[176] “Asian face database PF01,” http://nova.postech.ac.kr/archives/imdb.html.[177] H. Zawbaa and S. A. Aly, “Hajj and umrah event recognition datasets,”

arXiv:1205.2345, 2012.[178] M. Biehl, D. Matsumoto, P. Ekman, V. Hearn, K. Heider, T. Kudoh, and

V. Ton, “Japanese and caucasian facial expressions of emotion (jacfee)reliability data and cross-national differences,” Journal of NonverbalBehavior, vol. 22, no. 1, pp. 3–21, 1997.

[179] “Indian face database,” http://vis-www.cs.umass.edu/∼vidit/IndianFaceDatabase/.[180] “Fei dataset,” http://www.fei.edu.br/cet/facedatabase.[181] K. Robinette, H. Daanen, and E. Paquet, “The caesar project: a 3-d sur-

face anthropometry survey,” Proceedings of IEEE international conferenceon 3D digital imaging and modeling, pp. 380–386, 1999.

[182] “USF DARPA Human-ID 3D Face Database,” University of SouthFlorida.

[183] “FaceGen Modeller.” [Online]. Available:www.facegen.com/modeller.html

[184] “FACSGen Software.” [Online]. Available: www.affective-sciences.org/facsgen-2010

[185] E. B. Roesch, L. Tamarit, L. R. amd D. Grandjean, D. Sander, and K. R.Scherer, “Facsgen: A tool to synthesize emotional facial expressionsthrough systematic manipulation of facial action units,” InternationalJournal of Nonverbal Behavior, vol. 35, no. 1, pp. 1–16, 2011.

[186] B. Bickel, M. Botsch, R. Angst, W. Matusik, M. Otaduy, and H. Pfister,“Multi-scale capture of facial geometry and motion,” ACM Transactionsin Graphics, vol. 26, no. 3, p. 33, 2007.

[187] Y. Fu, G. Guo, and T. S. Huang, “Soft biometrics for video surveillance,”In Intelligent Video Surveillance: Systems and Technology, 2009.

[188] A. Jain, S. Dass, and K. Nandakumar, “Can soft biometric traits assistuser recognition?” In Proceedings of International Society for Optics andPhotonics: Defense and Security (SPIE), pp. 561–572, 2004.

[189] A. Senior, S. Pankanti, and A. Hampapur, “Enabling video privacythrough computer vision,” IEEE Security and Privacy, vol. 3, no. 3, pp.50–57, 2005.

[190] S. M. Kassin, P. C. Ellsworth, and V. L. Smith, “The general accpetanceof psychological reserach on eyewitness testimony: A survey of theexperts,” Ameircan Psychologist, vol. 46, pp. 1089–1098, 1989.

[191] R. C. L. Lindsay and G. L. Wells, “What do we really know about cross-race eyewitness identificastion?” Evaluating witness evidence, Chich-ester:Wiley Press, 1983.

[192] S. J. Platz and H. M. Hosch, “Cross-racial/thnic eyewitness identifi-cation: A field study,” Journal of Applied Social Psychology, vol. 18, pp.972–984, 1988.

[193] D. Hamilton, S. Stroessner, and D. Driscoll, “Social cognition: Impacton social psychology,” Academic, San Diego, eds. P. Devine, D. Hamilton,T. Ostrom, pp. 291–321, 1994.

[194] C. J. Solomon, S. J. Gibson, and J. J. Mist, “Interactive evolutionarygeneration of facial composites for locating suspects in criminal inves-tigations,” Applied Soft Computing, vol. 13, no. 7, pp. 3298–3306, 2013.

[195] V. Evers, “Human-computer interfaces: Designing for culture,” Ph.D.dissertation, University of New South Wales Sydney, Australia, 1997.

[196] M. C. Mackal, “The effects of ethnicity and message content on affectiveoutcomes in a computer-based learning environment,” PhD dissertation,2009.

[197] J. C. Anacleto and A. F. P. de Carvalho, “Improving human-computerinteraction by developing culture-sensitive applications based on com-mon sense knowledge,” arXiv preprint arXiv:1001.0418, 2010.

[198] A. J. Cowell and K. M. Stanney, “Manipulation of non-verbal interac-tion style and demographic embodiment to increase anthropomorphiccomputer character credibility,” International journal of human-computerstudies, vol. 62, no. 2, pp. 281–306, 2005.

[199] J. A. Pratt, K. Hauser, Z. Ugray, and O. Patterson, “Looking at human–computer interface design: Effects of ethnicity in computer agents,”Interacting with Computers, vol. 19, no. 4, pp. 512–523, 2007.

[200] M. Allison and L. M. Kendrick, “Towards an expressive embodied con-versational agent utilizing multi-ethnicity to augment solution focusedtherapy,” in The Twenty-Sixth International FLAIRS Conference, 2013.

[201] N. Davidenko, “Silhouetted face profiles: A new methodology for faceperception research,” Journal of Vision, vol. 7, no. 4, pp. 1–17, 2007.

[202] U. Tariq, Y. Hu, and T. S. Huang, Gender and Race Identification by Manand Machine. Springer, 2011.

[203] H. S. Seung and D. D. Lee, “The manifold ways of perception,” Science,vol. 290, no. 5500, pp. 2268–2269, 2000.

[204] A. Hadid and M. Pietkainen, “Demographic classification from facevideos using manifold learning,” Neurocomputing, vol. 100, pp. 197–205,2011.

[205] W. J. Scheirer, N. Kumar, K. Ricanek, P. N. Belhumeur, and T. E. Boult,“Fusing with context: a bayesian approach to combining descriptiveattributes,” in IEEE International Joint Conference on Biometrics, 2011, pp.1–8.

[206] N. Furl, P. J. Phillips, and A. J. O’Toole, “Face recogniton algorithms andthe other-race effect: Computational mechanisms for a developmentalcontact hypothesis,” Cognitive Science, vol. 26, pp. 797–815, 2002.

[207] M. N. Dailey, G. W. Cottrell, C. Padgett, and R. Adolphs, “Empath: Aneural network that categorizes facial expressions,” Journal of cognitiveneuroscience, vol. 14, no. 8, pp. 1158–1173, 2002.

[208] A. Haque and G. W. Cottrell, “Modeling the Other-race Advantagewith PCA,” Proceedings of the 27th annual cognitive science conference, pp.899–904, 2005.

[209] R. Bruyer, S. Leclere, and P. Quinet, “Ethnic categorisation of faces isnot independent of face identity,” Perception, vol. 33, no. 2, pp. 169–180,2004.



25[210] M. De Marsico, M. Nappi, D. Riccio, and H. Wechsler, “Demographics

versus biometric automatic interoperability,” in in Proceedings of Inter-national Conference on Image Analysis and Processing, 2013, pp. 472–481.

[211] J. Wallies, O. V. Lipp, and E. J. Vanman, “Face age and sex modulatethe other race effect in face recognition,” Attention, Perception andPsychophysics, vol. 74, no. 8, pp. 1712–1721, 2012.

[212] G. Farinella and J. L. Dugelay, “Demographic classification: Do genderand ethnicity affect each other?” Proceedings of IEEE International Con-ference on Informatics, Electronics and Vision (ICIEV), pp. 383–390, 2012.

[213] H. Dehon and S. Bredart, “An ‘other-race effect’ in age estimation fromfaces,” Perception, vol. 30, pp. 1107–1113, 2001.

[214] K. Fukushima and S. Miyake, “Neocognitron, a self-organizing neuralnetwork model for a mechanism of visual pattern recognition,” LectureNotes in Biomathematics, Springer, pp. 267–285, 1982.

[215] Y. LeCun and Y. Bengio, “Convolutional networks for images, speech,and time-series. the handbook of brain theory and neural networks,”MIT Press, 1995.

[216] P. Janssen, S. Kalkan, M. Lappe, A. Leonardis, J. Piater, L. Wiskott et al.,“Deep hierarchies in the primate visual cortex: What can we learn forcomputer vision?” IEEE Transactions on Pattern Recognition and MachineIntelligence, vol. 35, no. 8, pp. 1847–1871, 2012.

[217] T. Poggio and T. Serre, “Models of visual cortex,” Scholarpedia, vol. 8,no. 4, p. 3516, 2013.

[218] S. D. Goldinger, Y. He, and P. M. H, “Deficits in cross-race face learning:Insights from eye movements and pupillometry,” Journal of ExperimentalPsychology: Learning, Memory, and Cognition, vol. 35, pp. 1105–1122, 2009.

[219] S. Trawalter, A. R. Todd, and A. A. Baird, “Attending to threat: Race-based patterns of selective attention,” Journal of experimental socialpsychology, vol. 44, no. 5, pp. 1322–1327, 2008.

[220] B. Rehder and A. B. Hoffman, “Eyetracking and selective attention incategory learning,” Cognitive psychology, vol. 51, no. 1, pp. 1–41, 2005.

[221] I. Bulthoff, “What gives a face its ethnicity?” Journal of Vision, vol. 12,no. 9, 2012.

[222] J. M. Chen and D. L. Hamilton, “Natural ambiguities: Racial categoriza-tion of multiracial individuals,” Journal of Experimental Social Psychology,vol. 48, no. 1, pp. 152–164, 2012.

[223] A. K. Ho, J. Sidanius, D. T. Levin, and M. R. Banaji, “Evidence forhypodescent and racial hierarchy in the categorization and perceptionof biracial individuals,” Journal of personality and social psychology, vol.100, no. 3, p. 492, 2011.

Siyao Fu (M’08) received his Ph.D. from the State KeyLaboratory of Management and Control for ComplexSystems, Institute of Automation, Chinese Academyof Sciences in 2008. From 2008 to 2012, he was anAssociate Professor in the Department of Electricaland Computer Engineering, Minzu University, Beijing,China. He is currently a Postdoctoral Research Fellowat the Department of Electrical, Computer, and Biomed-ical Engineering at the University of Rhode Island,Kingston, Rhode Island. His research interests includecomputer vision, intelligent robotic vision system, com-

putational intelligence, machine learning, and data mining.

Haibo He (SM’11) is the Robert Haas Endowed Pro-fessor in Electrical Engineering at the University ofRhode Island (Kingston, RI). He received the B.S. andM.S. degrees in electrical engineering from HuazhongUniversity of Science and Technology (HUST), Wuhan,China, in 1999 and 2002, respectively, and the Ph.D.degree in electrical engineering from Ohio University,Athens, in 2006. From 2006 to 2009, he was an as-sistant professor in the Department of Electrical andComputer Engineering, Stevens Institute of Technol-ogy, Hoboken, New Jersey.

His research interests include pattern analysis, machine intelligence, cybersecurity, smart grid, and hardware design for machine intelligence. He haspublished one research book (Wiley), edited 1 research book (Wiley-IEEE)and 6 conference proceedings (Springer), and authored and co-authored over130 peer-reviewed journal and conference papers, including Cover Page High-lighted Paper in the IEEE Transactions on Information Forensics and Security,and highly cited papers in the IEEE Transactions on Knowledge and DataEngineering, IEEE Transactions on Neural Networks. His researches havebeen covered by national and international media such as IEEE Smart GridNewsletter, The Wall Street Journal, Providence Business News, among oth-ers. He serves as the Program Co-Chair of the International Joint Conferenceon Neural Networks (IJCNN’14) and General Chair of the IEEE SymposiumSeries on Computational Intelligence (SSCI’14). Currently, he is an AssociateEditor of the IEEE Transactions on Neural Networks and Learning Systems,and IEEE Transactions on Smart Grid.

He was a recipient of the IEEE CIS Outstanding Early Career Award (2014),K. C. Wong Research Award, Chinese Academy of Sciences (2012), NationalScience Foundation (NSF) CAREER Award (2011), Providence BusinessNews (PBN) ”Rising Star Innovator” Award (2011), and Best Master ThesisAward of Hubei Province, China (2002).

Zeng-Guang Hou (M’05) received the B.E. and M.E.degrees in electrical engineering from Yanshan Univer-sity (formerly North-East Heavy Machinery Institute),Qinhuangdao, China, in 1991 and 1993, respectively,and the Ph.D. degree in electrical engineering fromBeijing Institute of Technology, Beijing, China, in 1997.

From May 1997 to June 1999, he was a Postdoc-toral Research Fellow at the Laboratory of Systemsand Control, Institute of Systems Science, ChineseAcademy of Sciences, Beijing. He was a ResearchAssistant at the Hong Kong Polytechnic University,

Hong Kong SAR, China, from May 2000 to January 2001. From July 1999to May 2004, he was an Associate Professor at the Institute of Automation,Chinese Academy of Sciences, and has been a Full Professor since June 2004.From September 2003 to October 2004, he was a Visiting Professor at theIntelligent Systems Research Laboratory, College of Engineering, University ofSaskatchewan, Saskatoon, SK, Canada. He is a Professor and Deputy Directorof the State Key Laboratory of Management and Control for Complex Systems,Institute of Automation, Chinese Academy of Sciences. His current researchinterests include neural networks, robotics, and intelligent control systems.

Dr. Hou was an Associate Editor of the IEEE Computational IntelligenceMagazine and IEEE Transactions on Neural Networks. He was/is programchair/member of several prestigious conferences. Currently, he is an AssociateEditor of Neural Networks, and ACTA Automatica Sinica, etc.

Documents

Learning Race from Face: A Survey