How Sound Can Aid The Information Seeker In Information Retrieval... Or... If You Can’t See The...

How Sound Can Aid The How Sound Can Aid The Information Seeker In Information Seeker In

Information Retrieval...Information Retrieval...

Or... Or...

If You Can’t See The Forest For The If You Can’t See The Forest For The Trees, Trees,

Will The Sound Of The Tree Falling On Will The Sound Of The Tree Falling On You Help? You Help?

Dennis Hage Sherry Meredith

KEY QUESTIONSKEY QUESTIONS• How can sound help in

information retrieval?• What is the difference between

speech, non-speech, and music?• Who benefits from audio-based

information retrieval?

Sonically enhanced retrieval Sonically enhanced retrieval allows for more natural allows for more natural communication between communication between computer and user, allowing computer and user, allowing users to employ two senses users to employ two senses to solve a problem. to solve a problem.

- Stephen - Stephen BrewsterBrewster

DefinitionsDefinitions

• Sound is produced when the air is disturbed in some way (a vibrating object).

• Sonification - The use of non-speech audio to convey information.

Sonification Versus Sonification Versus SpeechSpeech

• Speech:– Slow speed & serial– Language dependent– Many predigested words

• Non-speech sounds:– Similar to graphical icons– Can quickly represent a concept

DefinitionsDefinitions (cont.)(cont.)

• Auditory Display - Non-speech audio to convey information in the computer interface.

• Auditory Display techniques– Sonification– Auditory Icons– Earcons

SonificationSonification

• Sound generation that need not have any direct relationship to the data being generated.

• Removing the display one more step from the data, may also make it richer and more coherent.

Why Use Sound?Why Use Sound?

““Who the hell wants to hear Who the hell wants to hear actors talk?”actors talk?” - - Harry WarnerHarry Warner (on the (on the future of “talkies”- movies with sound)future of “talkies”- movies with sound)

Why Use Sound? Why Use Sound? (Cont.)(Cont.)

• Archaeologists of the future• Human communication is primarily

carried out via speech• Audio cues in our day to day lives - car

honks to computer beeps to angry dog snarls.

Why Use Sound? Why Use Sound? (Cont.)(Cont.)

Acceleration of auditory display Acceleration of auditory display interest:interest:

• Information theory• Powerful and affordable computers• Sound synthesis technology• All of the above (synergy)

The increasing number, size, and The increasing number, size, and complexity of data sets challenge complexity of data sets challenge

existing visualization techniques.existing visualization techniques.

• Seismic data overview 24 hours in several minutes.

• Difficulty sorting out relationships as the number of variables to examine increases.

Why Use Sound? Why Use Sound? (Cont.)(Cont.)

Sounds like, You know…Sounds like, You know…

• ...time itself…is well-suited to represent temporal information.

• Sound can be used to:– Detect changes over time – Detect anomalies in data– Replace a distracting visual element

on a map display.•Example: Pitch is used to replace the time

bar.

Why should we use sound Why should we use sound at all?at all?

• Less interference between tasks using different senses.

• Individuals who are deprived of vision.• Signaling the ongoing status of

background activities.• Can also provide a sense of place.• Insistent quality

““Today’s biology suggests Today’s biology suggests that it’s the arts that lay that it’s the arts that lay the foundation for later the foundation for later academic and career academic and career

success.”success.”

• Eric Jensen, Teaching with the Brain in Mind.

Learning is discovering Learning is discovering what is already built into what is already built into

our brain that will allow us our brain that will allow us to respond to the to respond to the

immediate challenge.immediate challenge.

BrainBrainModelModel

Neural Networks similarNeural Networks similarto jungle organisms:to jungle organisms:

• Respond variously toenvironmental challenges.

• Made up of highly connected components.

CortexCortex(reasoning/(reasoning/

logic)logic)• Cortex size of

large dinner napkin.

• Contains about 10 billion neurons.

CortexLimbicSystem

Brainstem

Cerebellum

R. Sylwester, A Celebration of Neurons

NeuronNeuron

• Soma-body• Dendrites-

receptor sites• Axon-releases

neuro-transmitter• Synapse

G. Edelman, Bright Air, Brilliant Fire

• Grow more connections between cells.• Not lose existing connections.

Key to Key to Getting Getting

Smarter:Smarter: R. Sylwester, A Celebration of Neurons

Columns of Neurons

Six Cellular Layers

Area of web of axon

connections

Brain customizes itself.Brain customizes itself.

• Frequent behavior and thought patterns stimulate connections.

• This creates dominant pathwaysfor nerve impulses.

• Unneeded cells and unused connections get pruned away.

Modular BrainModular Brain

• Interconnected combinations of neural networks process more complicated, related phenomena,– from sounds to phonemes to words,– from lines to triangles to pyramids.

Two critical waysTwo critical waysto enrich the brain:to enrich the brain:

• Make learning challenging.• Provide interactive feedback.

ChallengingChallengingsensory stimulationsensory stimulationis a brain “Nutrient.”is a brain “Nutrient.”

Brain is artist and Brain is artist and scientist:scientist:

• Attempts to discernand understand patterns.

• Creates patterns of its own.

Facts and skills in isolationFacts and skills in isolation

• are organized differently by the brain,• need more practice and rehearsal.

FeedbackFeedback

• reduces uncertainty, lowers stress.• most useful when immediate

and self-generating.• should involve choice.

““Doing worksheets in Doing worksheets in school prepares a student school prepares a student

emotionally to do emotionally to do worksheets in life.”worksheets in life.”

• Robert Sylwester, A Cellebration of Neurons

Learning in contextLearning in context

• Brain tends to string things together:– notes into songs,– steps into dances,– events into stories,– actions into games.

• Robert Sylwester, A Cellebration of Neurons

Cerebellar system Cerebellar system

• first system to mature(along with inner ear).

• information gathering andfeedback source for movement.

• filters and integrates incoming data for complex decision making.

When several sensesWhen several sensesare used to registerare used to registerobjects and events:objects and events:

• information is stored in several interrelated memory networks,

• memory is more accessible and more powerful.

““Processes we had Processes we had considered pure ‘thinking’ considered pure ‘thinking’

are now seen as are now seen as phenomena in which the phenomena in which the cognitive and emotional cognitive and emotional

aspects work aspects work synergistically.” synergistically.”

Elias, et al. Promoting Social and Emotional Learning.

It Depends?It Depends?

• In some cases, redundant information is less effective than audio-only information to the phenomenon of visual dominance. (Tzelgov et al, 1987)

• Our sense of vision often seems much more dominant than our sense of hearing. (Ackerman 1990, Tuan 1993)

It Deep ends? Don’t Go It Deep ends? Don’t Go Overboard: Sound Quality Overboard: Sound Quality

ProblemsProblems

• Low resolution of many auditory variables

• Limited spatial precision• Lack of absolute values• Annoying• Interferes with speech communication• Absence of persistence• No printout• User limitations: The aural equivalent of

color blindness

Sound AdviceSound Advice

• Visualization is often insufficient for comprehending certain features in the data (Voyager 2 data analysis).

• Need to comprehend an abundance of data ... need to interpret and understand.

• Audio’s natural integrative properties are increasingly being proven suitable for presenting high dimensional data without creating information overload for users.

Early History History Uses - Early History History Uses - Titanic Sound BitesTitanic Sound Bites

• Sonar. – Leonardo da Vinci (1490) – Titanic (1912).

Early History History Uses Early History History Uses

• Drums and horns

• Morris code

• Diagnostic listening - mechanics, doctors

• Programmers

Some other successful Some other successful ones...ones...• Hans Geiger in early 1900s...

• Pulse-oximeter - 1980s. oxygen in a patient’s blood• Voyager 2 - rings of Saturn.• Quantum Whistle • Cartography• Yeung (1980) investigated sound as a means of

representing the multivariate data common in chemistry

• These cases illustrate the ability of the auditory system to extract underlying structure and temporal aspects of complex signals that are often important in scientific exploration and discovery.

YOU AUTO KNOW BETTERYOU AUTO KNOW BETTER

• Imagine driving to work, listening to auditory cues, navigate to the file menu which has a print option. You print reports so that they are waiting for you when you arrive. Start your email application and listen to any new email messages, delete and file others.

YOU AUTO KNOW BETTERYOU AUTO KNOW BETTER

• Audio glance presents a similar overview aurally rather than visually of the overall properties of an object. Specifically, email messages.

Sound to Drive by... MapsSound to Drive by... Maps• John Krygier (1994) identified a range of sound

variables that are similar to graphic variables:– Location: Location of a sound in a two- or three- dimensional

sound space.– Loudness: The magnitude of a sound.– Pitch: The highness or lowness (frequency) of a sound.– Register: The relative location of a pitch in a given range of

pitches.– Timbre: The general prevailing quality or characteristic of a

sound– Duration: The length of time a sound is (or is not) heard.– Rate of change: The relation between the duration of sound and

silence over time.– Order: The sequence of sounds over time.– Attack/decay: The time it takes a sound to reach its

maximum/minimum.

Pitch and Curve Pitch and Curve One Sound UseOne Sound Use

• Changes in magnitude of a single variable (pitch).

• Pitch - frequency of a simple tone. It is the feature of a sound by which listeners can arrange sounds on a scale from “lowest” to “highest”.

• CHILDREN BY AGE 10...• Pitch variation over time may be nearly as

effective as variation in curve height on a visual graph.

• May be better than tactile for blind users.

Auditory IconsAuditory Icons

• Caricatures of naturally occurring sounds which convey information by analogy with everyday events. (Gaver, 1986)

• 2 advantages:– Users tend to be more engaged.– Increases flexibility, users don’t have to always

attend to the screen for information.

Auditory IconsAuditory Icons

• Heighten awareness of a distributed community

• Mapping between the sounds and their meaning

• Sources not sounds• Advantage is also its weakness... no real-

world analogy such as renaming a file.• Frivolous? -- but how about visually impaired

or in critical applications where sound can reduce risk of error...

MUSICMUSIC

• Pitch - primary basis for traditional melody• Rhythm- relative changes in the timing of

the attacks of successive events• Tempo- the speed of events• Dynamics - the relative loudness of events

(static or varying)• Timbre - the difference of spectral content

and energy over time (differentiates a saxophone from a flute)

• Location - where the sound is coming from

Earcons Earcons (Micons and Picons)(Micons and Picons)

• Earcons are “non-verbal audio messages that are used in the computer/user interface to provide information to the user about some computer object, operation or interaction.” (Meera M Blattner, 1989)

• Based on synthetic musical tones.• Earcons are constructed from simple building

blocks called motives. (1-3 notes long)

Earcons Earcons (Micons and Picons)(Micons and Picons)

• By using several dimensions of sound, such as pitch, timbre, and rhythm, information can be communicated to the user.

• Hudson and Smith (1996) , have constructed longer earcons which convey useful information about incoming email messages, referred to as audio glances.

• Tool Palettes (Brewster)

• http://ftp.dcs.glasgow.ac.uk/~stephen/demos.shtml

Virtual limitationsVirtual limitations

Virtual limitationsVirtual limitations• One of the main deprivations caused by

blindness is the problem of access to information.

• This information bottleneck must be overcome in order for visually disabled users to access information.

• As GUI has become the standard developing alternative modes of interaction for the blind becomes even more important. Easier for sighted people, disadvantage for impaired.

In Search of Kramer... In Search of Kramer...

• The continued trend towards complex tasks and data challenge our abilities to make sense of things via abstractions such as numbers and categories. Perceptual presentations such as sonification present a means for tapping innate meaning-making skills. Matching these perceptual and cognitive capacities with appropriate tasks and designing effective display systems to effect this match is going to be a large and increasingly subtle challenge.

--Greg Kramer(email April 18, 1999)

THE FUTURE IS HEAR... THE FUTURE IS HEAR...

Or…Or…

Not GUI…Not GUI…

ButBut

SUI (Sound User Interface)SUI (Sound User Interface)

When Pigs Can Fly, Will When Pigs Can Fly, Will you hear them squeal?you hear them squeal?