Charisma Perception from Text and Speech
Andrew Rosenberg
NLP Group Meeting
11/03/05
11/03/05 NLP Group Meeting 2
Overview
• Background• Speech Study• Transcript Study• Conclusion & Future Work
11/03/05 NLP Group Meeting 3
Overview
• Background– What is charisma?– Does charismatic speech exist?– Why study charismatic speech?
• Speech Study
• Transcript Study
• Conclusion
11/03/05 NLP Group Meeting 4
Background - What is charisma?
(What do I mean by charisma?)• Not “closed door”, face-to-face charisma. • Rather, political (or religious) charisma
– The ability to attract, and retain followers by virtue of personality as opposed to tradition or laws. (Weber ‘47)
• E.g. Ghandi, Hitler, Che Guevara.
• Charismatic speech: Speech that encourages listeners to perceive the speaker as “charismatic”.
11/03/05 NLP Group Meeting 5
Background - Is there such a thing as charismatic speech?
• Pro: – Potential charismatic leaders must communicate
with would-be followers.– Charismatic leaders have historically had a
particular gift at public speaking• Hitler, MLK Jr., Castro.
• Con:– Charisma as a relationship between leader and
followers.– The mythologizing of a charismatic leader extends
beyond public address.
11/03/05 NLP Group Meeting 6
Background - Why study charismatic speech?
• General scientific interest.
• Feedback system for politicians and academic instructors.
• Identification of potential charismatic leaders
• Automatic generation of “charismatic-like” speech
11/03/05 NLP Group Meeting 7
Overview
• Background
• Speech Study– Questions Addressed– Experiment Design– Analyses of Responses
• Transcript Study
• Conclusion
11/03/05 NLP Group Meeting 8
Speech Study - Questions
• Do subjects agree about what is charismatic?
• What do subjects mean by charismatic?
• What makes speech charismatic?
11/03/05 NLP Group Meeting 9
Speech Study - Experiment Design
• Subjects: Friends and colleagues, not compensated monetarily
• Interface: Presentation of 45 short speech segments (2-30secs) via a web form
• Dependent variables: 5-point Likert scale ratings of agreement on 26 statements.
• Duration: avg. 1.5 hrs, min 45m, max ~3hrs
11/03/05 NLP Group Meeting 10
Speech Study - Experiment Design
• Interface– http://www1.cs.columbia.edu/~amaxwell/survey/
11/03/05 NLP Group Meeting 11
Speech Study - Experiment Design
• Materials: 45 tokens of American political speech
• Speakers: 9 Candidates for Democratic Party’s nomination for President– Gen. Clark, Gov. Dean, Rep. Edwards, Rep.
Gephardt, Sen. Kerry, Rep. Kucinich, Sen. Lieberman, Amb. Moseley Braun, Rev. Sharpton
• Topics: Postwar Iraq, Healthcare, Bush’s Tax plan, Reason for Running, Content-Neutral
11/03/05 NLP Group Meeting 12
Speech Study - Analysis
• How much do subjects agree?– Using the weighted kappa statistic with
quadratic weighting, mean kappa was 0.213 across all subject responses.
• Do subjects agree differently based on the stimuli?– No, there were no systematic differences
across all tokens
11/03/05 NLP Group Meeting 13
Do subject agree differently on the 26 statements?
• Most consistent statements
• Charisma: 0.224 (8th)
• Least consistent statements
The speaker is accusatory 0.512
The speaker is passionate 0.458
The speaker is intense 0.431
The speaker is angry 0.404
The speaker is enthusiastic 0.362
The speaker is trustworthy 0.037
The speaker is reasonable 0.070
The speaker is believable 0.074
The speaker is desperate 0.076
The speaker is ordinary 0.115
11/03/05 NLP Group Meeting 14
What do subjects mean by “charismatic”?
• Using kappa we determined which pairs of statements were most closely and consistently correlated with the charismatic statement.
The speaker is enthusiastic 0.606
The speaker is charming 0.602
The speaker is persuasive 0.561
The speaker is boring -0.513
The speaker is passionate 0.512
The speaker is convincing 0.503
11/03/05 NLP Group Meeting 15
Are certain speakers more charismatic than others?
• Yes, there is a significant difference between speakers (p=1.75e-2)
• Most charismatic – Rep. Edwards (3.73)– Rev. Sharpton (3.40)– Gov. Dean (3.32)
• Least charismatic– Sen. Lieberman (2.38)– Rep. Kucinich (2.73)– Rep. Gephardt (2.77)
11/03/05 NLP Group Meeting 16
Does the genre or topic of speech affect judgments of
charisma?• The tokens were taken from debates,
interviews, stump speeches, and a campaign ad– Stump speeches were the most charismatic.
(3.28) – Interviews the least. (2.90)
• Topic does not affect ratings of charisma.
11/03/05 NLP Group Meeting 17
Does recognizing a speaker affect judgments of charisma?• Subjects were asked to identify which, if
any, speakers they recognized at the end of the study.
• Subjects rated recognized speakers (3.28) significantly more charismatic than those they did not (2.99).
11/03/05 NLP Group Meeting 18
What makes speech charismatic?Acoustic/Prosodic and Lexical Properties
Examined• Duration (secs)• Min, max, mean, stdev F0
– Raw and normalized by speaker
• Min, max, mean, stdev intensity
• Speaking rate (syls/sec)• Length (words, syls)• 1st, 2nd, 3rd person pronoun
density• Function to content word
ratio
• Mean syllables/word• Number and ratio of
disfluencies and repeated words
• Mean words per intermediate and intonational phrase
• Number of intonational, intermediate, and internal phrases
• Mean and stdev of normalized F0 and intensity across phrases
• Require manual labeling of phrase boundaries
11/03/05 NLP Group Meeting 19
What makes speech charismatic?Properties showing positive correlation with
charisma • More Content
– Length in secs, words, syllables, and phrases
• Higher and more dynamic raw F0– Min, max, mean, std. dev. of F0 over male speakers
• Greater intensity– Mean intensity
• Higher in a speaker’s pitch range– Mean normalized F0
• Faster speaking rate– Syllables per second
11/03/05 NLP Group Meeting 20
What makes speech charismatic?Properties showing positive correlation with
charisma • Greater variation of F0 and intensity across phrases
– Std. dev. of normalized phrase F0 and intensity
• The use of more first person pronouns– First person pronoun density
• The use of polysyllabic words– Lexical complexity (mean syllables per word)
• Speaking fluidly– Number and ratio of disfluencies negatively correlate
• Repeat yourself– Number and ratio of repeated words
11/03/05 NLP Group Meeting 21
Overview
• Background• Previous Work• Speech Study• Transcript Study
– Questions Addressed– Experiment Design– Analyses of Responses
• Comparisons to Speech results
• Conclusion
11/03/05 NLP Group Meeting 22
Transcript Survey - Questions
• When reading a transcript of speech, do subjects rate charisma consistently?
• What do subjects mean by charisma? – Do they mean the same thing when referring to
text and speech?
• How does what is said influence subject ratings of charisma, compared to how it is said?
11/03/05 NLP Group Meeting 23
Transcript Survey - Experiment Design
• Subjects: 24 paid participants found – http://newyork.craigslist.org – “Talent gigs” section
• Interface: Presentation of 60 short transcripts (words…) via a web form
• Dependent variables: 5-point Likert scale ratings of agreement on 26 statements.
• Duration: avg. 1.5 hrs, min 45m, max ~3hrs
11/03/05 NLP Group Meeting 24
Transcript Survey - Design
• Interface:– http://www1.cs.columbia.edu/~amaxwell/textsurvey/A/
11/03/05 NLP Group Meeting 25
Transcript Study - Design
• Materials: 60 of 90 tokens of American political speech– The 90 transcripts were the 45 used in the speech
study, and 45 longer paragraphs– Each subject was presented with all 45 short
(mean ~28 words) and a semi-random set of 15 long transcripts (mean ~130 words)
• Speakers: Identical to Speech Study• Topics: Identical to Speech Study
11/03/05 NLP Group Meeting 26
Transcript Study - Design
• Examples:– Token 1:
We’re driving seniors out of medicare into HMOs. Every provision that would’ve brought down the cost of prescription drugs, the drug companies were against em all. They all came out.
11/03/05 NLP Group Meeting 27
Transcript Study - Design
• Examples:– Token 2.
…and I’d like to begin by, saying that I hope that, this afternoon’s talk will be an opportunity to challenge some underlying assumptions that we have about the world cause that’s why I’m uh running for President.
11/03/05 NLP Group Meeting 28
Transcript Study - Design
• Examples– Token 3:
…stabilize iraq because we occupy it. Yet he will not talk about the deficits in the fifty states we already occupy.
11/03/05 NLP Group Meeting 29
Transcript Study - Design
• Examples– Token 4:
…by two thousand five and then let their parents on a sliding scale based on income buy into medicaid at a price much below what they’d have to pay in the market.
11/03/05 NLP Group Meeting 30
Transcript Study - Design
• Some tokens are rated very similarly whether presented as speech or a transcript.– Example 1 always charismatic– Example 2 always uncharismatic
• Others are rated very differently – Example 3 more charismatic in speech– Example 4 in text
11/03/05 NLP Group Meeting 31
Transcript Study - Analyses
• How much do subjects agree?– Using the weighted kappa statistic with
quadratic weighting, mean kappa was 0.149
• Do subjects agree differently based on different stimuli?– No significant differences across all tokens
11/03/05 NLP Group Meeting 32
Do subject agree differently on the 26 statements?
• Most consistent statements
• Charisma: 0.134 (18th)
• Least consistent statements
The speaker is accusatory 0.280
The speaker is angry 0.263
The speaker’s message is clear 0.206
The speaker is friendly 0.197
The speaker is knowledgeable 0.193
The speaker is spontaneous
0.039
The speaker is ordinary 0.048
The speaker is boring 0.050
The speaker is desperate 0.064
The speaker is enthusiastic 0.093
11/03/05 NLP Group Meeting 33
What do subjects mean by “charismatic”?
• Using the kappa statistic determined which pairs of statements were most closely correlated with the charismatic statement.
The speaker is charming 0.576
The speaker is enthusiastic 0.511
The speaker is persuasive 0.503
The speaker is powerful 0.485
The speaker is convincing 0.483
The speaker is passionate 0.446
11/03/05 NLP Group Meeting 34
What do subjects mean by “charismatic”?
• Those statements that cooccur with the charismatic are in the speech and transcript study overlap greatly
• From this we conjecture that subjects employ a consistent functional definition of “charismatic” – Namely “charming, enthusiastic,
persuasive, convincing and passionate”
11/03/05 NLP Group Meeting 35
Does the identity of the speaker affect judgments of charisma?
• There is a significant difference between speakers (p=1.67e-10)
• Most Charismatic:– Gen. Clark (3.61)– Sen. Kerry (3.56)– Gov. Dean (3.54)
• Least Charismatic:– Sen. Lieberman (3.03)– Rep. Kucinich (3.12)– Amb. Mosley-Braun (3.23)
11/03/05 NLP Group Meeting 36
Does the genre of a transcript affect judgments of charisma?• Genre demonstrates a significant
influence on charisma (p=9.18e-14)
• Stump (3.34) and debate (3.32) above mean (3.15)
• Interview below mean (2.85)
11/03/05 NLP Group Meeting 37
Does the topic of a transcript affect judgments of charisma?• Topic was significantly influenced
ratings of charisma (p=1.5e-10)– In speech study, topic had no impact.
• Most charismatic topics:– Content-Neutral (3.64), Reason for running
(3.53) mean:3.36
• Least charismatic:– Taxes (3.12), Iraq (3.22), Healthcare (3.28)
11/03/05 NLP Group Meeting 38
What makes a transcript charismatic?
• More Content– Length in words, or syllables
• Use of more function words– Density of function words
• Use of fewer first person pronouns– First person pronoun density is negatively correlated
• Speak fluidly– Number and ratio of disfluencies
• Repeat yourself– Number and ratio of repetitions
• Lexical complexity (syls/wd) doesn’t matter
11/03/05 NLP Group Meeting 39
Overview
• Background
• Previous Work
• Speech Study
• Transcript Study
• Conclusion– Future Work
11/03/05 NLP Group Meeting 40
Conclusion• Despite not agreeing about what is
“charismatic”, subjects employ a common definition of “charisma”.– “Enthusiasm, passion, charm, persuasion and being
convincing” are consistently used to describe someone is “charismatic”.
• In general, what is said is a dominant force in whether speech is perceived as “charismatic” or not, with how it is said modifying this.
• Acoustic properties broadly reflect enthusiasm and passion
11/03/05 NLP Group Meeting 41
Conclusion - Future Work
• Resynthesis Experiments– By modifying prosody of tokens can we make
Lieberman charismatic? Sharpton uncharismatic?
• Investigating Palestinian Arabic– What are the similarities and differences between
American and Palestinian notions of charisma?– What lexical and acoustic/prosodic properties are
displayed by charismatic Palestinian speech?