Reverse Correlation Methods for Social Cognition...Reverse Correlation Methods for Social Cognition Ron Dotsch Daniel Wigboldus Ad van Knippenberg Gent, March 9, 2010 Saturday, November

Reverse Correlation Methods for Social Cognition

Ron Dotsch

Daniel WigboldusAd van Knippenberg

Gent, March 9, 2010

Saturday, November 6, 2010

Interspecies differences in face categorization


Reverse Correlation

• Classification image technique (Ahumada, 1996)


Reverse Correlation


• Responses to stimuli in experiment contain information about how task is performed


Reverse Correlation



• Randomly (instead of systematically) vary stimuli on a lot of (instead of specific) dimensions


Reverse Correlation



• Randomly (instead of systematically) vary stimuli on a lot of (instead of specific) dimensions

• Use correlation between stimulus features and response pattern to infer decision criteria (classification image)


Reverse correlation image classification task(Dotsch, Wigboldus, Langner, & van Knippenberg, 2008; Mangini & Biederman, 2004)

Reverse correlation task









Base + noise Base - noise

A 6


Classification images

MoroccanChinese



MoroccanChinese



MoroccanChinese


y = sin x y = sin x at 90º

Noise generation


y = sin x y = sin x at 90º

Noise generation


Noise generation212 M.C. Mangini, I. Biederman / Cognitive Science 28 (2004) 209–226

Fig. 1. Creating the noise. (a) Six orientations and two phases (cosine upper row; sine lower row) of randomamplitude sinusoids are summed to create the 2 cycles/image noise. The same process is repeated four times foreach “tile” of the 4 cycles/image noise, 16 times for the 8 cycles/image noise, etc. (b) All five octaves are summed tocreate the noise pattern. To the right are shown the 2, 4, 8, 16, and 32 cycles/image noise patterns of all orientations.

visible in photographs. Image morphing software (Gryphon Morph 2.5) was used to morpheach model to, two arbitrarily chosen models of the same gender, resulting in a total of 200images. Taking the mathematical average of these 200 faces resulted in the base image.1 Forthe individuation task, a morph of images of John Travolta and Tom Cruise was used as thebase image. The base images are shown in the first column of Fig. 3.For each trial a random noise stimulus was generated. Although most experiments using the

response classification technique have used Gaussian white pixel noise, our noise was com-posed of truncated sinusoids that are nominally localized in space, frequency, orientation, andphase.2 Each sinusoid consisted of two cycles of a sine wave in a square envelope. Sinusoidalpatches at five octave scales (2, 4, 8, 16, and 32 cycles/image), six orientations (0, 30, 60, 90,120, and 150◦), and two phases (0 and π/2) were summed to create one noise pattern (Fig. 1).The resulting noise pattern was described by 4096 parameters that corresponded to the ampli-tudes of the sinusoids. The amplitude parameters were selected randomly from a zero meanuniform distribution. The separate scales were multiplied by constant factors that insured thatthe range of each scale did not exceed the dynamic range available for the noise. These factorsalso contributed to the frequency profile presented in Fig. 5. A justification of why this noisewas chosen is presented in Section 3.Image creation and presentation was performed on a Macintosh G4 computer using Matlab

with the Psychophysics and VideoToolbox extensions (Brainard, 1997; Pelli, 1997).

2.1.3. ProcedureSubjectswere assigned to one of three discrimination tasks: gender, expression (happyversus

unhappy), or identity (John Travolta versus Tom Cruise). At the beginning of the experimentthe subjects were told that they would be making a simple discrimination but that it wouldbe difficult to perform. For the Gender and Expression tasks subjects were told that a set of

4092 parameters (contrasts/amplitudes)Saturday, November 6, 2010

Mangini & Biederman (2004)M.C. Mangini, I. Biederman / Cognitive Science 28 (2004) 209–226 215

Fig. 3. The results from the three tasks based on the high confidence “probably” trials (a) Happy/unhappy, (b)male/female, (c) Tom Cruise/John Travolta. The base images were identical for the expression and gender tasks.The darkest and lightest areas of the classification images, which are from the data of all 36 subjects, indicate theareas that most influenced the subjects’ classifications. The addition of the classification image to the base faceresults in Class Image 1, which appears happy. The subtraction of the classification image results in Class Image2, which appears unhappy. The same addition and subtraction operations produce the class images for (b) femaleand male and (c) Cruise and Travolta, respectively. The rightmost two columns show the classification images forthe median subject calculated in terms of Euclidean pixel distance for a given subject’s classification image and theaverage classification image.

Gender in Columns 3 and 4, it becomes clear that while the eyes and mouth play a large role,the diffuse energy in the center of the face creates distinctive gender changes of the nose. Thisillustrates how this technique can discover the diffuse, subtle information employed by faceperceivers.A classification image can be constructed from only those sinusoidal components that dif-

fered significantly between the two categories on each task. Significance was tested withrepeated independent t tests for each of the 4092 components with adjustment for multiplecomparisons (Rom, 1990). For the Expression task, 187 components reached significance(p < .0005), for Gender, 85 components (p < .001), and for the celebrity identity task, 52components (p < .001). The images in Fig. 4 reveal that, indeed, relatively few components,in the order of 100, are adequate to recreate the class differences.Valentin et al. (1994) and Sergent (1989) have speculated, on the basis of statistical analysis

of sets of faces, that whereas gender and expression can be conveyed by low frequency infor-mation, individuation is carried in higher frequency channels. Fig. 5 shows the class imagesfor all sinusoids separately for each of the five scales. For all tasks, including individuation, alarge portion of the information distinguishing the classes appears to be at 4 and 8 cycles/image.This shows that human observers categorizing faces by identity do chose to make use of lowfrequency information in performing their categorizations. Furthermore, this low frequency in-


Humans vs. Baboons

Martin-Malivel, Mangini, Fagot, and Biederman (2006)


Humans vs. Baboons

Next, to determine the similarities between the baboons’ andhumans’ use of information, we compared 1,000 bootstrapsamples pulled from each baboon’s data with bootstrap data from

each of the six leave-one-out human populations. Averagingacross the 6,000 similarity measures gave the average similaritybetween a baboon’s information use and the humans’ informa-tion use. The percentile rank of the average similarity value wasless than 0.5% (for B06) and 0.8% (for B07). The across-speciesdifferences were significantly greater than the within-humansdifferences (B06: prep 5 .966; B07: prep 5 .956).

Principal Component Analysis (PCA)Although the bootstrap analysis provided a test for the plannedcomparison of baboons’ and humans’ information use, we alsosubjected the CIs (unlabeled as to which species they came from)to a PCA. The PCA extracted the dimensions accounting for thegreatest variance among the individuals’ CIs. The first principalcomponent answered the question: What is the greatest differ-ence among the eight CIs? The first step in the PCA was tosubtract themean of all the CIs from all the images. This was doneso that the first principal component would represent the di-mension that accounted for the most variance among the images(rather than the mean of the images themselves). The principalcomponents were computed as the eigenvectors of the innerproduct of thematrix containing all of themean-subtractedCIs asvectors. The eigenvalues of these components represent theamount of variance accounted for by each principal component.Results showed that the first principal component (after the

mean of all subjects had been subtracted) accounted for 20% ofthe variance (chance 5 14%). Because there is no standardmethod for testing the statistical significance of a principalcomponent, we performed a Monte Carlo simulation. The nullhypothesis tested was that the eight human and baboon CIs wereno more redundant than a set of eight ‘‘random observer’’ CIs.These random CIs were generated by taking 1,000 randomlygenerated noise patterns and randomly assigning them to the‘‘human’’ and ‘‘baboon’’ categories without any consideration oftheir visual structure. The eigen-decomposition of the eightrandom CIs was performed as described for the human andbaboon data. The eigenvalues for the random observers wererecorded, and the process was repeated 10,000 times to obtaina null distribution. The eigenvalue of the first principal compo-nent of our human-baboon PCAwas greater than the eigenvaluesof all 10,000Monte Carlo simulations (prep5 .996), showing thatthere was more redundancy in the eight human and baboon CIsthan could be expected by chance. In other words, the 20% ofvariance accounted for by the first principal component repre-sents a significantly reliable difference among the subjects.Figure 4a shows the correlation between each subject’s CI and

the first principal component. This component clearly splits thesubjects into two distinct groups: humans and baboons. Theimages to the left of the graph illustrate the information subjectsused to classify the images as human (left) and baboon (right).The top pictures (row 1) illustrate the information that humanstended to use, and the bottom pictures illustrate the information

Fig. 3. Individual results. The column on the left shows the classificationimage (CI) for each human (H1–H6) or baboon (B06, B07) subject. EachCI was obtained by subtracting the average of the noise patterns that in-duced the subject to respond ‘‘baboon’’ from the average of the noisepatterns that induced the subject to respond ‘‘human.’’ The CIs presentedhere show the information that each subject used to categorize the picturesas ‘‘human.’’ The middle column shows the result of adding the CI to theoriginal ambiguous morph, thus re-creating the typical image elicitinga ‘‘human’’ response. The column on the right shows the result of sub-tracting the CI from the original ambiguous morph, thus re-creating thetypical image eliciting a ‘‘baboon’’ response.

Volume 17—Number 7 603

J. Martin-Malivel et al.

Martin-Malivel, Mangini, Fagot, and Biederman (2006)Next, to determine the similarities between the baboons’ andhumans’ use of information, we compared 1,000 bootstrapsamples pulled from each baboon’s data with bootstrap data from

each of the six leave-one-out human populations. Averagingacross the 6,000 similarity measures gave the average similaritybetween a baboon’s information use and the humans’ informa-tion use. The percentile rank of the average similarity value wasless than 0.5% (for B06) and 0.8% (for B07). The across-speciesdifferences were significantly greater than the within-humansdifferences (B06: prep 5 .966; B07: prep 5 .956).

Principal Component Analysis (PCA)Although the bootstrap analysis provided a test for the plannedcomparison of baboons’ and humans’ information use, we alsosubjected the CIs (unlabeled as to which species they came from)to a PCA. The PCA extracted the dimensions accounting for thegreatest variance among the individuals’ CIs. The first principalcomponent answered the question: What is the greatest differ-ence among the eight CIs? The first step in the PCA was tosubtract themean of all the CIs from all the images. This was doneso that the first principal component would represent the di-mension that accounted for the most variance among the images(rather than the mean of the images themselves). The principalcomponents were computed as the eigenvectors of the innerproduct of thematrix containing all of themean-subtractedCIs asvectors. The eigenvalues of these components represent theamount of variance accounted for by each principal component.Results showed that the first principal component (after the

mean of all subjects had been subtracted) accounted for 20% ofthe variance (chance 5 14%). Because there is no standardmethod for testing the statistical significance of a principalcomponent, we performed a Monte Carlo simulation. The nullhypothesis tested was that the eight human and baboon CIs wereno more redundant than a set of eight ‘‘random observer’’ CIs.These random CIs were generated by taking 1,000 randomlygenerated noise patterns and randomly assigning them to the‘‘human’’ and ‘‘baboon’’ categories without any consideration oftheir visual structure. The eigen-decomposition of the eightrandom CIs was performed as described for the human andbaboon data. The eigenvalues for the random observers wererecorded, and the process was repeated 10,000 times to obtaina null distribution. The eigenvalue of the first principal compo-nent of our human-baboon PCAwas greater than the eigenvaluesof all 10,000Monte Carlo simulations (prep5 .996), showing thatthere was more redundancy in the eight human and baboon CIsthan could be expected by chance. In other words, the 20% ofvariance accounted for by the first principal component repre-sents a significantly reliable difference among the subjects.Figure 4a shows the correlation between each subject’s CI and

the first principal component. This component clearly splits thesubjects into two distinct groups: humans and baboons. Theimages to the left of the graph illustrate the information subjectsused to classify the images as human (left) and baboon (right).The top pictures (row 1) illustrate the information that humanstended to use, and the bottom pictures illustrate the information

Fig. 3. Individual results. The column on the left shows the classificationimage (CI) for each human (H1–H6) or baboon (B06, B07) subject. EachCI was obtained by subtracting the average of the noise patterns that in-duced the subject to respond ‘‘baboon’’ from the average of the noisepatterns that induced the subject to respond ‘‘human.’’ The CIs presentedhere show the information that each subject used to categorize the picturesas ‘‘human.’’ The middle column shows the result of adding the CI to theoriginal ambiguous morph, thus re-creating the typical image elicitinga ‘‘human’’ response. The column on the right shows the result of sub-tracting the CI from the original ambiguous morph, thus re-creating thetypical image eliciting a ‘‘baboon’’ response.

Volume 17—Number 7 603

J. Martin-Malivel et al.


In more detail

that the baboons used (row 3). Inspection of these images revealsthat the baboons categorized the faces mainly on the basis ofcontrast between the eyes and the surrounding head region,responding ‘‘human’’ in the case of a lighter eye area and ‘‘ba-boon’’ in the case of a darker eye area, whereas the humans paidattention to a broader range of facial features, such as the shapeof the nose and mouth and facial contour.

DISCUSSION

The results demonstrate that different species given identicaltraining regimens can utilize substantially different information

to achieve comparable performance. This difference could notbe discovered from experiments in which only the performanceover all trials was computed, without considering the responsesto individual trials. By correlating observers’ responses to ran-dom noise patterns trial by trial, we were able to determine thatthe two species used different information to achieve successfulcategorization. The PCA shows that the largest difference amongthe CIs was the difference between the CIs of humans andbaboons. Both baboons relied heavily on the coarse contrastbetween the eyes and the surrounding face, unlike humans, whorelied on both coarse and detailed information across the entireface.

Fig. 4. Results of the principal component analyses (a) for human and baboon subjects and (b) for human andbaboon subjects and two runs of the theoretical observer (TO). The individual bars represent the correlationsbetween each subject’s classification image and the first eigenvector. The images along the ordinate of each graphillustrate the dimension accounting for the greatest variance among subjects. The three rows depict, respectively,the information used by subjects with high positive projections (11) on the first principal component, by subjectsat the midpoint (0 projection), and by subjects with high negative projections (!1) to categorize faces as human(left) or baboon (right). The subjects with high positive projections tended to use many of the finer details acrossthe entire face, including the eyes, nose, lips, and jawline, whereas those with high negative projections tended touse a coarse contrast between the eyes and the surrounding face to make their discriminations.

604 Volume 17—Number 7

Interspecies Differences in Categorizing Faces


Reverse Correlation History

• Auditory work (Ahumada & Lovell, 1971)

• Low level visual perception: humans differ from ideal observers (Ahumada, 1996; Beard & Ahumada, 1998)

• Sparked host of studies on visual perception:

• Letter discrimination (Watson & Rosenholtz, 1997)

• Perceptual learning (Knoblauch, Thomas, & D’Zmura, 1999)

• Illusory contours (Gold, Murray, Bennett, & Sekuler, 2000)

• Many more! (see Eckstein & Ahumada, 2002)


• Higher order visual perception (face processing):




• Bubbles (Gosselin & Schyns, 2001)





• Mangini & Biederman (2004)





• Mangini & Biederman (2004)

• And now: Social cognition?



Visualizing bias

Dotsch, Wigboldus, Langner, & van Knippenberg (2008)


Chinese, Japanese, or Korean?


Japanese






Chinese




Chinese



Image classification task

MoroccanChinese



Low Moderate High



Low Moderate High


Part 2:Trait ratings

-2

-1

0

1

2

Low Moderate High

ST IAT

(N = 70)

-2

-1

0

1

2

Low Moderate High

ST IAT

F(2, 68) = 43.95, p < .001, partial η! = .56F(2, 68) = 39.00, p < .001, partial η! = .53

Criminal Trustworthy


Criminal Classification Image


Moroccan + Criminal

+ =


Moroccan + Criminal

+ =


Criminality added


Original


Results

15

20

25

30

35

Low Prejudice High Prejudice

% C

ateg

oriz

ed a

s M

oroc

can

Moroccan Faces Criminal Moroccan Faces

GLM with prejudice as continuous variable:Prejudice main effect: F(1, 76) = 7.45, p < .01, partial η! = .09

Face main effect: F(1, 76) = 24.35, p < .01, partial η! = .24Face x Prejudice interaction: F(1, 76) = 5.58, p = .02, partial η! = .07.


Other Reverse Correlation Examples


Memory bias for attractive alternatives

Karremans, Dotsch, & Corneille (in prep)


Conditions

UnattractiveAttractive

(N = 82, all females)


Base image (50% morph)


Main effect

Attractive


Main effect

Unattractive


Interaction (within attractive condition)

Relation > no relationNo relation > relation


Interaction (within unattractive condition)

Relation > no relationNo relation > relation


Ingroup projection

Imhoff, Dotsch, Bianchi, Wigboldus, & Banse (in prep.)


Portugal vs. Germany

Germany's striker Miroslav Klose and Portuguese defender Pepe argue during the Euro 2008 Championships quarter-final football match between Portugal and Germany at the St. Jakob Park Stadium on June 19, 2008 in Basel, Switzerland.


Germany


Portugal


Base image (50% morph)


Results

German European(N = 53)


Results

Portuguese European(N = 50)


General discussion

• Classification image is a index of decision strategy, not necessarily a picture in the head

• Can be spontaneous

• Not necessarily exploratory

• Holds lots of promise for social cognition research


Thank you!

My advisors:

• Daniel Wigboldus

• Ad van Knippenberg

Contact me:

• [email protected]

My collaborators:

• Oliver Langner

• Alexander Todorov

• Johan Karremans

• Olivier Corneille

• Roland Imhoff

• Mauro Bianchi


mailto:[email protected]


Dimensions of face evaluation

Dotsch, Todorov, & Wigboldus (in prep)Saturday, November 6, 2010

Primary trait dimensions

Trustworthy and Untrustworthy

Dominant and Submissive










+ =

Threat

Dominant Untrustworthy


Hi-Speed Threat

100 ms 500 ms


High Speed Threat

100 ms


High Speed Threat

500 ms


High Speed Threat

500 ms > 100 ms


Thank you!

My advisors:

• Daniel Wigboldus

• Ad van Knippenberg

Contact me:

• [email protected]

My collaborators:

• Oliver Langner

• Alexander Todorov

• Johan Karremans

• Olivier Corneille

• Roland Imhoff

• Mauro Bianchi




Documents

Reverse Correlation Methods for Social Cognition...Reverse Correlation Methods for Social Cognition Ron Dotsch Daniel Wigboldus Ad van Knippenberg Gent, March 9, 2010 Saturday, November