Seeing What is Said: New Methods for Extracting Information from Speech

1Identifying Deceptive Speech Julia HirschbergComputer ScienceColumbia University

12CollaboratorsStefan Benus, Jason Brenner, Robin Cautin, Frank Enos, Sarah Friedman, Sarah Gilman, Cynthia Girand, Martin Graciarena, Andreas Kathol, Laura Michaelis, Bryan Pellom, Liz Shriberg, Andreas Stolcke, Michelle Levine, Sarah Ita Levitan, Andrew RosenbergAt Columbia University, SRI/ICSI, University of Colorado, Constantine the Philosopher University, Barnard, CUNY2For collaborations on the work described in this talk.3Everyday LiesOrdinary people tell an average of 2 lies per dayYour new hair-cut looks great.Im sorry I missed your talk but .In many cultures white lies are more acceptable than the truthLikelihood of being caught is lowRewards also low but outweigh consequences of being caughtNot so easy to detect4Serious LiesLies whereRisks highRewards highEmotional consequences more apparentAre these lies easier to detect?By humans?By machines?

5OutlineResearch on DeceptionPossible cues to deceptionCurrent approachesOur first corpus-based study of deceptive speechApproachCorpus collection/paradigmFeatures extractedExperiments and resultsHuman perception studiesCurrent research6A Definition of DeceptionDeliberate choice to misleadWithout prior notificationTo gain some advantage or to avoid some penaltyNot:Self-deception, delusion, pathological behaviorTheaterFalsehoods due to ignorance/error

67Who Studies Deception?Students of human behavior especially psychologistsLaw enforcement and military personnelCorporate security officersSocial services workers Mental health professionals78Why is Lying Difficult for Most of Us?Hypotheses: Our cognitive load is increased when lying becauseMust keep story straightMust remember what weve said and what we havent saidOur fear of detection is increased ifWe believe our target is hard to foolWe believe our target is suspiciousStakes are high: serious rewards and/or punishmentsMakes it hard for us to control indicators of deceptionDoes this make deception easy to detect?89Cues to Deception: Current ProposalsBody posture and gestures (Burgoon et al 94)Complete shifts in posture, touching ones face,Microexpressions (Ekman 76, Frank 03)Fleeting traces of fear, elation,Biometric factors (Horvath 73)Increased blood pressure, perspiration, respirationother correlates of stressOdor?Changes in brain activity??Variation in what is said and how (Adams 96, Pennebaker et al 01, Streeter et al 77)910Spoken Cues to Deception(DePaulo et al. 03)Liars less forthcoming?-Talking time -Details +Presses lips Liars less compelling?-Plausibility -Logical Structure -Discrepant, ambivalent -Verbal, vocal involvement -Illustrators -Verbal, vocal immediacy +Verbal, vocal uncertainty +Chin raise +Word, phrase repetitions Liars less positive, pleasant?-Cooperative +Negative, complaining -Facial pleasantness Liars more tense?+Nervous, tense overall +Vocal tension +F0 +Pupil dilation+Fidgeting Fewer ordinary imperfections?-Spontaneous corrections -Admitted lack of memory +Peripheral details 1011Current Approaches to Deception DetectionTraining humansJohn Reid & AssociatesBehavioral Analysis: Interview and InterrogationLaboratory studies: Production and Perception`Automatic methodsPolygraphNemesysco and the Love DetectorNo evidence that any of these work.but publishing this can be dangerous! (Anders Eriksson and Francisco La Cerda)11Universal Lie ResponseAnders Eriksson and Francisco La Cerdas article "Charlatanry in forensic speech science: A problem to be taken seriously." 12What is needed:More objective, experimentally verified studies of cues to deception which can be extracted automatically13OutlineResearch on DeceptionPossible cues to deceptionCurrent approachesOur first corpus-based study of deceptive speechApproachCorpus collection/paradigmFeatures extractedExperiments and resultsHuman perception studiesCurrent research14Corpus-Based Approach to Deception DetectionGoal: Identify a set of acoustic, prosodic, and lexical features that distinguish between deceptive and non-deceptive speech as well or better than human judgesMethod:Record a new corpus of deceptive/non-deceptive speech Extract acoustic, prosodic, and lexical features based on previous literature and our work in emotional speech and speaker idUse statistical Machine Learning techniques to train models to classify deceptive vs. non-deceptive speech1415Major ObstaclesCorpus-based approaches require large amounts of training data difficult to obtain for deceptionDifferences between real world and laboratory liesMotivation and potential consequencesRecording conditionsIdentifying ground truthEthical issuesPrivacySubject rights and Institutional Review Boards

1516Columbia/SRI/Colorado Deception Corpus (CSC)Deceptive and non-deceptive speechWithin subject (32 adult native speakers)25-50m interviewsDesign:Subjects told goal was to find people similar to the 25 top entrepreneurs of AmericaGiven tests in 6 categories (e.g. knowledge of food and wine, survival skills, NYC geography, civics, music), e.g.What should you do if you are bitten by a poisonous snake out in the wilderness?Sing Casta Diva.What are the 3 branches of government?1617Questions manipulated so scores always differed from a (fake) entrepreneur target in 4/6 categoriesSubjects then told real goal was to compare those who actually possess knowledge and ability vs. those who can talk a good gameSubjects given another chance at $100 lottery if they could convince an interviewer they matched target completelyRecorded interviewsInterviewer asks about overall performance on each test with follow-up questions (e.g. How did you do on the survival skills test?)Subjects also indicate whether each statement T or F by pressing pedals hidden from interviewer1718

19The Data15.2 hrs. of interviews; 7 hrs subject speechLexically transcribed & automatically aligned Truth conditions aligned with transcripts: Global / LocalSegmentations (Local Truth/Local Lie): Words (31,200/47,188)Slash units (5709/3782)Prosodic phrases (11,612/7108)Turns (2230/1573)250+ featuresAcoustic/prosodic features extracted from ASR transcriptsLexical and subject-dependent features extracted from orthographic transcripts1920LimitationsSamples (segments) not independent Pedal may introduce additional cognitive loadEqually for truth and lieOnly one subject reported any difficultyStakes not the highestNo fear of punishmentSelf-presentation and financial reward2021Acoustic/Prosodic FeaturesDuration featuresPhone / Vowel / Syllable DurationsNormalized by Phone/Vowel Means, SpeakerSpeaking rate features (vowels/time)Pause features (cf Benus et al 06)Speech to pause ratio, number of long pausesMaximum pause lengthEnergy features (RMS energy)Pitch featuresPitch stylization (Sonmez et al. 98)Model of F0 to estimate speaker rangePitch ranges, slopes, locations of interestSpectral tilt features21Useful features: Energy features, duration, unit position in interviewLTM model of f0

22Lexical FeaturesPresence and # of filled pausesIs this a question? A question following a questionPresence of pronouns (by person, case and number)A specific denial?Presence and # of cue phrasesPresence of self repairsPresence of contractionsPresence of positive/negative emotion wordsVerb tense Presence of yes, no, not, negative contractionsPresence of absolutely, reallyPresence of hedgesComplexity: syls/wordsNumber of repeated wordsPunctuation typeLength of unit (in sec and words)# words/unit length# of laughs# of audible breaths# of other speaker noise# of mispronounced words# of unintelligible words

22Useful features: positive emotion words, filled pauses

23Subject-Dependent Features% units with cue phrases% units with filled pauses% units with laughterLies/truths with filled pauses ratioLies/truths with cue phrases ratioLies/truths with laughter ratioGender23Useful features: use of filled pauses, laughter, and cue phrases24

2425Results88 features, normalized within-speakerDiscrete: Lexical, discourse, pauseContinuous features: Acoustic, prosodic, paralinguistic, lexicalBest Performance: Best 39 features + c4.5 MLAccuracy: 70.00% LIE F-measure: 60.59 TRUTH F-measure: 75.78 Lexical, subject-dependent & speaker normalized features best predictors2526Some ExamplesPositive emotion words deception (LIWC)Pleasantness deception (DAL)Filled pauses truthSome pitch correlations varies with subject27OutlineResearch on DeceptionPossible cues to deceptionCurrent approachesOur first corpus-based study of deceptive speechApproachCorpus collection/paradigmFeatures extractedExperiments and resultsHuman perception studiesCurrent researchLIWC to profile speakers2728Evaluation: Compare to Human Deception DetectionMost people are very poor at detecting deception ~50% accuracy (Ekman & OSullivan 91, Aamodt 06)People use unreliable cues, even with training2829A Meta-Study of Human Deception Detection (Aamodt & Mitchell 2004)Group#Studies#SubjectsAccuracy %Criminals15265.40Secret service13464.12Psychologists450861.56Judges219459.01Cops851155.16Federal officers434154.54Students1228,87654.20Detectives534151.16Parole officers13240.4229K is # of studiesN is number of participants30Evaluating Automatic Methods by Comparing to Human PerformanceDeception detection on the CSC Corpus32 JudgesEach judge rated 2 interviewsReceived training on one subject.Pre- and post-test questionnairesPersonality Inventory

3031

By Judge58.2% Acc.By Interviewee58.2% Acc.31Compare to 67% accuracy by machine.32What Makes Some People Better Judges?Costa & McCrae (1992) NEO-FFI Personality MeasuresExtroversion (Surgency). Includes traits such as talkative, energetic, and assertive. Agreeableness. Includes traits like sympathetic, kind, and affectionate. Conscientiousness. Tendency to be organized, thorough, and planful. Neuroticism (opp. of Emotional Stability). Characterized by traits like tense, moody, and anxious. Openness to Experience (aka Intellect or Intellect/Imagination). Includes having wide interests, and being imaginative and insightful. 3233Neuroticism, Openness & Agreeableness Correlate with Judges Performance

On Judging Global lies.33Neuroticism was negatively correlated also with pre-test expectations of frequency of lying. Hence less inclined to label segments as lies; and, since data biased toward truthfulness, these people are better on measures favoring True.Openness: if one is open to new experience, one may be better able to base judgments on data seen, rather than biasAgreeableness: also associated with dependent personality disorder (extreme attention to the opinions and affective state of others); thus better able to assess their emotional state in deception?34Other Useful FindingsNo effect for trainingJudges post-test confidence did not correlate with pre-test confidenceJudges who claimed experience had significantly higher pre-test confidenceBut not higher accuracyMany subjects reported using disfluencies as cues to deceptionBut in this corpus, disfluencies correlate with truth(Benus et al. 06)3435OutlineResearch on DeceptionPossible cues to deceptionCurrent approachesOur first corpus-based study of deceptive speechApproachCorpus collection/paradigmFeatures extractedExperiments and resultsHuman perception studiesCurrent researchResearch QuestionsWhat objectively identifiable features characterize peoples speech when deceiving in different cultures? What objectively identifiable audio cues are present when people of different cultures perceive deception? What language features distinguish deceptive from non-deceptive speech when people speak a common language? When one speaker is not a native speaker of that language?

-do these features appear when ppl from various cultures lie / detect lies-both deceptive and non-deceptive speech in speakers:Of the same native languageOf 2 different native languages 36HypothesesH1: Acoustic, prosodic and lexical cues can be used to identify deception in native Arabic and Mandarin speakers speaking English with accuracy greater than human judges. H2: Results of simple personality tests can be used to predict individual differences in deceptive behavior of native American, Arabic, and Mandarin speakers when speaking English. H3: Simple personality tests can predict accuracy of American judges of deceptive behavior when judging Arabic and Mandarin speakers speaking English. H4: Particular acoustic, prosodic and lexical cues can be used to identify deception across native and nonnative English speakers while other cues can only be used to identify deception within English speakers of a particular culture. -8 main ones, highlighting major points-MAIN: Speech cues can be used to detect deception across native and non-native English speakers-we think that some speech cues can only be used to detect deception in speakers of a particular culture

-Personality tests: used to predict differences in deceptive behavior AND the judging of deceptive behavior:-by american, mandarin, and arabic judges looking at native and non-native english speakers

-gender and culture can interact/influence the ability to detect deception and can influence a deceivers speech cues

37H5: Some personality traits can predict individual differences in deceptive behaviors across native and nonnative English speakers while other personality traits can only predict individual differences in deceptive behaviors within a particular culture. H6: Simple personality tests can predict accuracy of Arabic and Mandarin judges of deceptive behavior when judging native American and nonnative American speakers speaking English. H7: Acoustic, prosodic and lexical cues of deception can be mediated by the gender and/or culture of the deceiver and target. H8: Judges' ability to detect deception is mediated by the gender and/or culture of the deceiver.

-gender and culture can interact/influence the ability to detect deception and can influence a deceivers speech cues

38Experimental DesignBackground Information (e.g. gender, race, language)Biographical Questionnaire Fake Resume paradigmPersonal questions (e.g. Who ended your last romantic relationship?, Have you ever watched a person or pet die?)NEO FFIBaseline recordings for each speakerLying gamePayment schemeNo visual contactKeyloggingPayment scheme: Interviewers get $1 extra for every answer they judge correctly but lose $1 for each incorrect judgmentInterviewees get $1 extra for every lie they tell that interviewers judge to be true39Biographical Questionnaire

Samples

Sample 2Sample 1Current StatusData collectionOver 40 pairs have been recorded~30 hours of speechExtracting features, correlations with personality inventoriesParticipant poolAmerican English and Mandarin Chinese speakersRecruited from Columbia and Barnard Future workInclude Arabic speakersFeature extractionAcoustic/Prosodic (i.e. duration, speaking rate, pitch, pause)Lexico/Syntactic (i.e. laughter, disfluencies, hedges)Correlate behavioral variation in lies vs truth with standard personality test scores for speakers (NEO FFI)Machine learning experiments to identify features significantly associated with deceptive vs non-deceptive speech.Native pakistanis instead of arabic?43