Multimodality Alistair D N Edwards Department of Computer
Science
http://www-users.cs.york.ac.uk/~alistair/teaching/AMSmasterclass/Multimodality.pptx
Slide 3
Multi-X MULTI-media, MULTI-modal, MULTI- channel... there are
many terms that can be prefixed by the qualifier 'multi', so that
there is a danger of getting diverted into debates about the
meanings of the terms and distracted from the real questions. At
the same time, these discussions often centre on computers; a
linguist or a playwright or a singer would not even use the terms,
far less wonder whether what they do should be prefixed by
'multi'.
Slide 4
Multi-X -Why is it that the debate arises in the context of
computer interaction? -Why is it that it does not arise in other
contexts of communication? -Are there are other contexts in which
the 'multi-X' question arises? -Are we progressing towards a
situation in which the question will become obsolete or
redundant?
Slide 5
How many senses do we have?
Slide 6
Five
Slide 7
Extra senses
http://www.bbc.co.uk/radio4/science/extrasenses.shtml 1Pain
2Balance 3Time 4Temperature 5Digestion
Slide 8
Class of 2015 9 Body position 10 Heat 11 Danger 12 Circadian
Rhythms 13 Hunger 14 Bodily needs 15 Time 16 Pain 17 Fatigue How
many senses do we have?
Slide 9
Class of 2008-10 1 Sight 2 Smell 3 Taste 4 Hearing 5 Touch 6
Balance 7 Direction 8 Humour 9 Body position 10 Heat 11 Danger 12
Circadian Rhythms 13 Hunger 14 Bodily needs 15 Time/Rhythm 16 Pain
17 Fatigue 18 Pressure 19 Empathy 20 Intuition 21 Fear 22 Hunger 23
Tiredness 24 Thirst
Slide 10
What is a sense?
Slide 11
What is a human sense? (Lets not get distracted by questions as
to whether animals or even plants have senses in the way we
mean)
Slide 12
What is a human sense? An input channel? Only for external
information? or internal too?
Slide 13
What is a human sense? An input channel? Only for external
information? or internal too?
Slide 14
Objectives What modes do we use in interaction? Which others
could we use? Why? How?
Slide 15
Visual dominance Psychology Neurology User interfaces The
McGurk effect An example of a cross-modal effect
Slide 16
Visual veracity
Slide 17
Slide 18
The rotating mask illusion
https://www.youtube.com/watch?v=sKa0eaKsdA0 We cannot help but see
the illusion The brain knows that it is impossible The eye is not
just a camera
Slide 19
Cross-modality Were looking at separate modalities but the
total is not necessarily the sum of the parts The role of auditory
cues in modulating the perceived crispness and staleness of potato
chips Journal of Sensory Studies, 19: 347363. doi:
10.1111/j.1745-459x.2004.080403.x
http://onlinelibrary.wiley.com/doi/10.1111/j.1745-459x.2004.080403.x/full
Slide 20
the perception of both the crispness and staleness was
systematically altered by varying the loudness and/or frequency
composition of the auditory feedback elicited during the biting
action. The potato chips were perceived as being both crisper and
fresher when either the overall sound level was increased, or when
just the high frequency sounds (in the range of 2 kHz20 kHz) were
selectively amplified
Slide 21
Some practical(?) possibilities Sounds Haptics Smell
Spearcons Examples 1 2 3 http://sonify.psych.gatech.edu/res
earch/auditorymenus/ elephant elevator
Slide 29
Spearcons Examples 1 2 3 http://sonify.psych.gatech.edu/researc
h/auditorymenus/ elephant elevator parking meter
Slide 30
Sonification
Slide 31
Sounds Why do we not make more use of sounds in
interfaces?
Slide 32
http://www.dcs.gla.ac.uk/~stephen/papers/CHI2000.pdf
TermDefinition Proprioceptive Relating to sensory information about
the state of the body (including cutaneous, kinesthetic, and
vestibular sensations). Haptic Relating to the sense of touch.
Vestibular Pertaining to the perception of head position,
acceleration, and deceleration Kinesthetic Meaning the feeling of
motion. Relating to sensations originating in muscles, tendons and
joints Cutaneous Pertaining to the skin itself or the skin as a
sense organ. Includes sensation of pressure, temperature, and pain.
Tactile Pertaining to the cutaneous sense but more specifically the
sensation of pressure rather than temperature or pain. Force
Feedback Relating to the mechanical production of information
sensed by the human kinesthetic system. Haptics - definitions
Slide 33
Geomagic Touch http://www.sensable.com/ Formerly the Sensable
Phantom Omni
Slide 34
Trisenx Scent Dome http://www.trisenx.com/
Slide 35
iSmell
Slide 36
Slide 37
smells Why do we not make more use of smells in
interfaces?
Slide 38
Conclusions Why do we not make more use of other
senses/channels/modes in interaction? Which ones could we
practically use more? What would be the benefits?