Perceiving Transformational InvarianceDiscovering the basis multi-sensory integrationRichard HurtigDepartment of Speech pathology & AudiologyThe University of Iowa

Speech Perception is the result of Sensory IntegrationSign wave speech

Speech Perception is hard!Massaro, 1998

It is not all in the acoustic signal:Stop-Gap Trading Relation[s_lit split]Repp 1985

Top Down EffectsWarren Phoneme RestorationBABY[ba_y] + [__clap_][ba:clap:y]baby with clap in the background

What tells the brain that what the eyes are seeing goes with what the ears are hearing and what the skin is feeling?Temporal Synchrony of Input SignalsMapping to Cortical Convergence ZonesCo-Modulation of Primary Sensory Areas of Cortex

Possible Solutions:

ScrotsPerspective makes thingslook normal

FrescoPerspective allowsForms to emerge

Circular Projections Skull in the Mirror

Anamorphism

When do thingslook right?

Recognizing Faces/ObjectsDepends on Perspectiveand Prior Experience

Monica

Look the same?Thompson, P. (1980). Margaret Thatcher: a new illusion, Perception, 9, 483-484

Spectral Inversion

Upside Down Environmental SoundsSawBirdDoorbellKissRoosterDogCowToilet

Upside Down Speech & Music

MonroeImage Recognition of Distorted Images: Digital Image Morphing

ElvisMadonna

How Much of the signal do we need?Low PassHigh Pass

Filtered Music & SpeechLow Pass 2500The doctor prescribed the drug

Play it BackwardsHow important is temporal order?

DogSawBirdCowToiletKissRoosterDoorbell

Reversed Speech & MusicEdgar Allen Poe

Under What Conditions Do We RecognizeTemporal Inversion?Palindromes/ SymmetrySegmental ElementsBOBSTATSMetrical Schemes000- - - 000

Size/Gender

Frequency ComprehensionTime Domain and Frequency DomainRepresentations of the syllable /ma/

Blue=Original Red=Compressed(75%)

Spectral CompressionTFT ALGORITHM (Hurtig & Turner, Patented 2003)InputSignalOutputSignalA/DD/AFFTIFFTArray Remapping123412350 % Compression50% compressionOriginal 4-tone

Compressed Environmental Sounds80%DogBirdCowDoorbellSawKissRoosterToilet

Compressed Speech & Music80%100%

Compressed Singing Voice:Perception of Gender and Size Original 90% 80% 70%SopranoSopranoTenorBass

Multi-Modal Speech PerceptionMcGurk EffectBAGADA

Multi-Modal Speech PerceptionMcGurk Effect- 2GABABDA/BGA

Synthesized & NaturalMcGurk EffectBA

VA

THA

DAMassaro, 1998Fixed AudioVariable VideoFixed VideoVariable Audio

Perceive [my dad taught me to drive]See [my gag gog me goo grive]Hear [my bab bob me boo brive]

Massaro, 1998

Gesture-Speech ExperimentBa-Da ContinuumMassaro, 1998

Speech Perception In Noise Multi-modal effectsAudio OnlySNR +5 dBAudio & VisualSNR +1 dBAudio & TactileSNR +3 dB

Where does the integration take place?(Posner & Raichle , 1994)

What does increased rCBF say about functional modules?(Posner & Raichle , 1994)

Why There Cant Be A Pre-wired Module Uniquely Reserved for Language In The BrainHow would we account for multi-modal effects?Why would the module have evolved a somato-sensory input?Sur & Leamey (2001) Rerouting visual inputs to auditory cortex

Current Projects Utilizing multi-sensory integration to assess Hearing Aid benefit- with Ruth BentlerImplementing a spectral compression algorithm in a wearable sensory aid- with Chris Turner

Contact InformationRichard HurtigDepartment of Speech Pathology & Audiology, College of Liberal Arts & SciencesThe University of IowaIowa City, Iowa 52242319-335-8730Richard-Hurtig@uiowa.edu