McGurk Doesn’t Work: Evidence Against the McGurk Effect...

Synthetic-Lab Natural-Lab Natural-MTurk

ForcedChoice

OpenEnded

Auditory Fusion Visual Other

McGurk Doesn’t Work: Evidence Against the McGurk Effect as a Perceptual Illusion

RESULTS

Laura M. Getz & Joseph C. Toscano[laura.getz, joseph.toscano]@villanova.edu

DISCUSSION

Visualspeechcuesplayanimportantroleinspeechrecognition,andtheMcGurkeffectisaclassicdemonstrationofthis.

REFERENCES

CURRENT EXPERIMENTS

INTRODUCTION

MacDonald,J.,&McGurk,H.(1978).Visualinfluencesonspeechperceptionprocesses. Perception&Psychophysics.

Mallick,D.B.,Magnotti,J.F.,&Beauchamp,M.S.(2015).VariabilityandstabilityintheMcGurkeffect:Contributionsofparticipants,stimuli,time,andresponsetype. Psychonomic Bulletin&Review.

Massaro,D.W.(1998). Perceivingtalkingfaces:Fromspeechperceptiontoabehavioralprinciple.MITPress.

McGurk,H.,&MacDonald,J.(1976).Hearinglipsandseeingvoices. Nature.

Toscano,J.C.,&Lansing,C.R.(2017).Age-relatedchangesintemporalandspectralcueweightsinspeech. LanguageandSpeech.

Expt. Subjects Report“Ba”

Report“Ga”

Report“Da”

McGurk&MacDonald(1976)

3-5yr (n=21) 19% 0% 81%

7-8 yr (n=28) 36% 0% 64%

18-40 yr (n=54) 2% 0% 98%

MacDonald&McGurk (1978) 18-24 yr (n=44) 9% 27% 64%

Wesetouttosystematicallylookattheseindividualdifferences,investigatinganumberoffactorsthatcouldinfluencefusionrates.

Ø Participantdifferences:labvs.online• Lab:VillanovaUniversityIntroPsychologystudents

Agerange:18-21years• Online:Amazon’sMechanicalTurk(MTurk)

Agerange:21-72years

Ø Stimulusdifferences:syntheticvs.natural• Synthetic:Klatt-synthesizedaudio;/ɑ/vs./æ/vowelcontexts

Baldi visuallipmovements;/ba/vs./da/CombinedaudioandvideousingiMovie

• Natural:2maleand2femaletalkers(Mallick etal.,2015)CongruentAVstimuliseparatedandrecombinedinaudB-visG andvisG-audB combinationsusingiMovie

Ø Designdifferences:open-endedvs.3-alternativeforcedchoiceAskedtoreport:Whatdidthespeakersay?

HEAR SEE REPORT

“ba” “da”“ga”

StimulusParticipant Design

McGurk&MacDonad’s explanationfortheillusory“fusion”effectdealswiththewaythesoundsarearticulated.

Bilabial Alveolar Velar

Voiced /b/ /d/ /g/

Voiceless /p/ /t/ /k/

Morerecentworkshowsthattheeffectmaynotbeasrobustaspreviouslybelieved,astheproportionoffusionresponsesdependsonindividualandtaskdifferences(Mallick etal.,2015).

Ø Lower proportionoffusionresponsesoverallthaninoriginalexperiments• Open-endedMTurk fusionresponserate(0.38)similartoMallick etal.

(2015)with“tha”includedasafusionresponse

• Participantdifferences:more fusionresponsesonMTurk thaninlab• Onerelevantindividualdifferencemaybeage,witholderparticipants

more likelytoshowfusioneffect• Thissuggeststhatphoneticcueweightscontinuetochangeacrossthe

lifespan,inlinewithpreviouswork(Toscano &Lansing,2017)

• Stimulusdifferences:syntheticstimuliresultedinmore “other”responses,suggestingthatdespitehighcontrol,wemayneedtousenaturalstimulitoseefusioneffect

• Designdifferences:ineachexperiment,more fusionresponsesto3-alternativeforced-choicethanopen-endeddesign(cf.Mallick etal.,2015)• Similarproportionoffusionresponseswithsinglemodalitytrials

integrated withAVtrialsandblocked design

Ø Ratherthanarobustperceptualillusion,wearguethattheMcGurkeffectisaproductofindividualdifferencesandtaskdemands

• Maybeit’stimetofindamorereliableclassroomdemonstrationofvisualinfluenceonspokenwordrecognition?

SyntheticStimuli:LabParticipantsForcedChoice(N=24) Open-Ended(N=11)

BA DA GA BA DA GA combo otheraudioB 0.93 0.02 0.05 0.73 0.05 0.02 0.00 0.20audioD 0.01 0.95 0.04 0.01 0.72 0.02 0.00 0.25audioG 0.04 0.08 0.88 0.03 0.14 0.75 0.00 0.09visualB 0.89 0.05 0.06 0.52 0.04 0.04 0.00 0.40visualD/G 0.02 0.51 0.46 0.01 0.32 0.23 0.00 0.44AV-congruentB 0.89 0.05 0.06 0.41 0.00 0.02 0.00 0.55AV-congruentD 0.01 0.93 0.06 0.00 0.64 0.02 0.00 0.34AV-congruentG 0.00 0.02 0.98 0.00 0.01 0.94 0.00 0.03AV-audioB-visD/G 0.39 0.34 0.26 0.05 0.01 0.08 0.00 0.86AV-audioG-visB 0.10 0.02 0.89 0.04 0.02 0.88 0.00 0.04

NaturalStimuli:LabParticipantsForcedChoice(N=46) Open-Ended(N=46)

BA DA GA BA DA GA combo otheraudioB 0.98 0.02 0.00 0.98 0.00 0.00 0.00 0.01audioD 0.00 0.99 0.01 0.00 0.99 0.00 0.00 0.00audioG 0.00 0.01 0.99 0.00 0.00 1.00 0.00 0.00visualB 0.99 0.00 0.01 0.99 0.00 0.00 0.00 0.00visualD 0.00 0.91 0.09 0.01 0.85 0.11 0.00 0.03visualG 0.01 0.39 0.60 0.01 0.38 0.57 0.00 0.03AV-congruentB 0.99 0.01 0.00 0.99 0.00 0.01 0.00 0.00AV-congruentD 0.01 0.95 0.04 0.00 0.97 0.02 0.00 0.01AV-congruentG 0.00 0.06 0.94 0.00 0.05 0.94 0.00 0.00AV-audioB-visG 0.75 0.14 0.11 0.74 0.10 0.11 0.00 0.04AV-audioG-visB 0.21 0.01 0.79 0.23 0.00 0.74 0.01 0.00

NaturalStimuli:OnlineMTurk ParticipantsForcedChoice(N=37) Open-Ended(N=39)

BA DA GA BA DA GA combo otheraudioB 0.93 0.04 0.02 0.80 0.02 0.00 0.00 0.18audioD 0.02 0.91 0.07 0.00 0.94 0.03 0.00 0.03audioG 0.02 0.02 0.96 0.00 0.02 0.95 0.00 0.03visualB 0.90 0.08 0.02 0.89 0.02 0.02 0.00 0.06visualD 0.03 0.82 0.15 0.02 0.55 0.14 0.01 0.28visualG 0.04 0.50 0.46 0.01 0.35 0.43 0.01 0.20AV-congruentB 0.94 0.05 0.01 0.92 0.01 0.00 0.00 0.07AV-congruentD 0.01 0.93 0.06 0.00 0.95 0.03 0.00 0.02AV-congruentG 0.03 0.03 0.94 0.01 0.02 0.95 0.00 0.02AV-audioB-visG 0.49 0.41 0.09 0.37 0.17 0.06 0.00 0.40AV-audioG-visB 0.03 0.02 0.95 0.09 0.01 0.77 0.11 0.02

M=45years

M=35years

M=36years

M=39years