Embed Size (px)
Citation preview
1
Interactive Systems Design
MSc in Communications, Computing and Human-Centred Systems
Dr. Chris Baber
3
Aims
Introduce some basic concepts from Cognitive Psychology
Demonstrate relationships between these concepts and interactive systems design
4
Objectives
After completing this module, students will be able to:
Critique user interfaces through the application of appropriate concepts from cognitive psychology
Offer guidelines on how to improve a design
5
Assignment1. Select ANY mobile telephone user interface
2. Critically evaluate the user interface in terms of:1. layout 2. use of colour3. Gestalt 4. consistency 5. support for navigation
3. Explain how and why the user interface might present problems to users
4. Consider the (non-computer) metaphor(s) that can be related to the design
5. Explain what assumptions you believe that the designers have made in terms of the way that users interpret the user interface
6. Produce a redesign of the user interface that you believe is superior to the current version and explain why your redesign is better
6
Marking Scheme
1. Presentation of system image and screen-shots (images and description) - up to 10 marks
2. Definition of a ‘good’ user interface - up to 10 marks3. Discussion of key topics from cognitive psychology -
up to 30 marks4. Critique of selected user interface, in terms of 2 and
3 - up to 25 marks5. Redesign (images and description) - up to 10 marks6. Why is your redesign superior to the current design?
- up to 10 marksUp to 5 additional marks for presentation of report
7
Useful References
Ji, Y.G. et al., 2006, A usability checklist for the usability evaluation of mobile phone user interface, International Journal of Human-Computer Interaction, 20 (3), pp. 207-231
Bocker, M. and Suwita, A., 1999, The evaluation of the Siemens C10 mobile phone – usability testing beyond ‘quick and dirty’, Proceedings of the 43rd Annual Meeting of the Human Factors and Ergonomics Society, Santa Monica, CA: HFES, pp.379-383
Ziefle, M., 2002, Usability of menu structures and navigation keys in mobile phones: a comparison of the ease of use in three different brands, Proceedings of the 6th International Scientific Conference on Work with Display Units, Berlin, pp.359-361
Jokela, T. et al., 2006, Methods for Quantitative Usability requirements: a case study on the development of the user interface of a mobile phone, Personal and Ubiquitous Computing, 10 (6), 345-355
8
TimetableMonday 29th Tuesday 30th Wednesday 31st
10 - 11 1. Introduction to Module
6. Navigation in user interfaces
10. Paper prototyping
11 - 12 2. Fundamentals of graphical user interface
design
7. Interaction Devices: entering
data
11. Distributed Cognition
12 - 1 3. Link and Layout Analysis
8. Interaction Devices:
Selecting objects
12. Collaborative Work
2 - 3 4. Use of Colour and Highlighting
9. Multimodal HCI 13. Future HCI
4 - 5 5. Commands and Menus
10. Mobile HCI 14. Future HCI
9
1. Introduction
10
Design Principles1. Know your user2. Build on systems that are familiar to your user3. Display available functions4. Ensure Coherence and Consistency5. Visualise states and transitions6. Design shortcuts7. Design Help8. Tolerate user error9. Provide Context10. Manage Focus11. Display Grammar
11
Varieties of TechnologyTechnology mediates human action
on the world in ways that…
Augment
Correct
Extend
Modify
& Disrupt
…human performance and ability
12
Augment / Correct / Extend
Technologies ‘augment’ human abilities SUBSTITUTION: calculators
REPRESENTATION: shopping lists
Technologies ‘extend’ human abilities CORRECTION: spectacles, hearing aids
increasing RANGE: telescope, telephone
13
Modify
Technology changes: Task sequence Required actions Information content
Example: Calculate 9 hours of work at £3.50 per hour.
What if you add another hour?
14
Disrupt
Changes ways of working Requires new ways of thinking or acting
Interferes with activities MicroSoft PaperClip PDA in meetings Mobile phones in lectures
15
How different ‘models of the user’ lead to different assumptions on how to Minimise Human Error
Explain error Reduce error
Idiot Design fault Make as simple as possible
Designer Error free No need
Computer Follow instruction Clarify instruction
Information Processor Breakdown Modes; forcing functions
Active Interpreter Not Compatible Design for mental model
16
‘Instant Experts’
You get a new mobile telephone for Xmas, do you:
1. Read the manual from cover to cover and then switch on the phone?
2. Switch on the phone and try to use it?
3. Switch on the phone and read the manual when you have problems?
17
Using tools
Physical appearance
Knowledge of use
Sequence of activity
18
Affordance
See handle Reach out hand Grasp handle Turn handle Pull door
19
The Cooker Problem #1
Which control acts on which ring?
20
The Cooker Problem #2
Which control acts on which ring?
21
Paul Fitts: Spatial Compatibility
22
Direction of motion Stereotypes
a d
b
c
? ?
Clockwise = increaseClockwise = rightClockwise = away from controlClockwise = increase on scale
1 2 3 4 5 67
1 2 3 4 5 67
23
Clockwise to Increase?
1234567
24
Conclusions…
We have learned ‘routines’ for how to use many sorts of technology
We apply these routines ‘automatically’ When the routines succeed, they are
reinforced When the routines fail, we think about what
we’re doing
25
Expectations and Affordances
People “know” how technology works…
‘Out of box’ experience
Reading instructions
Prior knowledge and expectations
26
Automaticity
Norman and Shallice (1980) Fully automatic processing controlled by SCHEMATA
Partially automatic processing controlled by either Contention Scheduling
Supervisory Attentional System (SAS)
27
Supervisory Attentional System Model
Perceptual System
SupervisoryAttentional
System
Effector System
Contentionscheduling
Triggerdatabase
Control schema
28
Contention Scheduling
Gear changing when driving involves many routine activities but is performed ‘automatically’ – without conscious awareness
When routines clash, relative importance is used to determine which to perform – Contention Scheduling
e.g., right foot on brake or clutch
29
SAS activation
Driving on roundabouts in France Inhibit ‘look right’; Activate ‘look left’ SAS to over-ride habitual actions
SAS active when: Danger, Choice of response, Novelty etc.
30
Know your user…
When designing any form of user interface, you should ask:
What am I trying to help the reader (or user) of this display do?
How does the information I am presenting to them help with their thinking or their activity?
31
Seven Stage Action Model[Norman, 1990]
Form intentionDevelop plan
Perform action
Object in world
Evaluate against goalInterpret object
Perceive state of object
GOAL OF PERSON
32
Routes from Intention to Action
Route 1:Automaticity Apply known routine Compatibility Affordance
Route 2: Formulate plan Problem solving Situated actions
33
Problem Solving
A problem is something that doesn’t solve easily
A problem doesn’t solve easily because: you don’t have the necessary knowledge or, you have misrepresented part of the problem
If at first you don’t succeed, try something else
Tackle one part of the problem and other parts may fall into place
34
Problem Solving
Representation affects strategy
More than one possible solution
Solution limited by boundary conditions
Active involvement and testing
35
Problem
SEND
+ MORE
MONEY
36
Functional Fixedness
Strategy developed in one version of the problem
Strategy might be inefficient
XIV ÷ X vs. XXXX ÷ X
Convert numerals or just ‘see’ 4
37
Interacting with Products as Problem Solving User has a ‘goal’ to achieve using the product
The product offers functions to the user
The user must select the ‘best’ function to progress towards the goal
The interaction can be thought of as movement through a space of possible actions towards a goal
Selection of the ‘best’ action will be influenced by available functions and their interpretation
38
Situated Actions
‘Plans’ imply a specified sequence of actions… plans are representations of action
…but some actions are ‘opportunistic’, i.e., supported by the current context-of-use
Suchman (1992) terms these ‘situated actions’ plans are resources for action
39
2. Fundamental of graphical user interface design
40
Information Design can Influence Thinking
John Snow, 1854m Mapping death from cholera, in London, showed clusters around water pump (and not near brewery)
41
Attributes affecting utility
ReadabilityWordsSyntax
LegibilityFont design
ContrastIlluminationLuminance
ConspicuityGraphic designSpatial coding
Temporal codingShape codingColour coding
Display
Psychological processExtraction of meaning
Comprehension
Mediating Userattributes
42
Gestalt Principles
43
Density and Clutter
Density is related to available screen space E.g 80 x 24 line display = 1920 character spaces
The proportion of filled spaces = Density Density averages 25% but rarely exceeds
50%
44
Levels of Density
70% density 50% density 30% density
Shneiderman, 1992
45
Reducing Density
Grouping / tabulating; Reduce number of words; Reduce number of characters
46
Conclusions
Understanding basic Gestalt principles helps manage focus
Some objects on a display are more conspicuous than others Use highlighting sparingly
Some objects form ‘perceptual groups’ Use this to help design screen layout and to
minimise risk of confusion
47
3. Link and Layout Analysis
Link Analysis Sequence in which objects are used, or attended
to, during a task Data recorded from eye-movement or from
observation of product use Layout Analysis
Define relationship between objects in terms of functional grouping and sequence of use
48
3. Link and Layout Analysis
49
Eye-Movement ‘Heat Maps’
50
Reading
Saccades and Fixations
Anticipation and Inferences
Interpretation
51
Saccades and Fixations
52
Reading from Computer Screens
Reading Speed Computer 20% - 30% slower than paper
BUT only when Refresh rate < 60Hz Resolution < 640x480 pixels
Differences essentially related to character discrimination data-driven perception (see slide 27)
53
Link Analysis Worked Example:scan of initial layout
Cassette door
Radio displayBANDASPMME-SCAN
ST
DX/LOA
SCAN
SEEK
CD
1 2 3
4 5 6B
54
Link Analysis Worked Example:revised layout
Cassette door
Radio display
BANDASPMME-SCAN
ST
DX/LO
A
SCAN
SEEK
CD 1 2 3
4 5 6
B
55
Layout Analysis Worked Example:initial layout
Cassette door
Radio displayBANDASPMME-SCAN
ST
DX/LOA
SCAN
SEEK
CD
1 2 3
4 5 6B
56
Layout Analysis Worked Example:functional grouping
Cassette door
Radio displayBANDASPMME-SCAN
ST
DX/LOA
SCAN
SEEK
CD
1 2 3
4 5 6B RADIO
RADIO
CASSETTE
57
Layout Analysis Worked Example:redesign within functional groupings
Radio display
BANDASPMME-SCAN
ST
DX/LOA
SCAN
SEEK
CDCassette door
1 2 3
4 5 6
B
58
Context
When presented with ambiguous stimuli our prior knowledge of the world helps us to make sense of it
59
Context
60
InterpretationKnowledge of what you are “looking at” can aid in interpretation
JA CKAN DJI
LLW ENTU PTH
EHI LLTOFE
TCH APAILO
FWA TER
Organisation of information is also useful
A B C
1 20 15 10
2 32 16 12
3 27 17 11
Gestalt also influences interpretation
61
Concepts
How do you know a chair is a chair?
A chair has four legs…does it? A chair has a seat…does it?
62
Mental models
Craik Internal representation of external reality Every good monitor needs a model of the world it
is monitoring Johnson-Laird
Strategies of knowledge assimilation
63
Mental Models
Van Dijk and Kintsch (1983) Text processed to extract propositions, which are
held in working memory; When sufficient propositions in WM, then linking
performed; Relevance of propositions to linking proportional
to recall; Linking reveals ‘gist’
64
Inferences
Comprehension typically requires our active involvement in order to supply information which is not explicit in the text
1. Mary heard the ice-cream van coming
2. She remembered her pocket money
3. She rushed into the house.
65
Inference and Recall
Thorndyke (1976): recall of sentences from ‘Mary’ story 85% correct sentence 58% correct inference –
sentence not presented 6% incorrect inference
66
Scripts, Schema and Frames Schema = chunks of knowledge
Slots for information: fixed, default, optional
Scripts = action sequences Generalised event schema (Nelson, 1986)
Frames = knowledge about the properties of things
DESIGN CAN HELP CALL APPROPRIATE CONCEPTS TO MIND, e.g., through the use of metaphor
67
VisiCalc Xerox Star
68
WIMP
Windows Icons Menus Pointing devices
69
Metaphor - Pros
WOZNY (1989) Metaphors allow users to compare knowledge of the world
with what is happening in the system, and use analogy in initial reasoning with the system
MARCUS (1998) “A familiar concept in an unfamiliar environment can add
appeal and when users are new to a topic or application domain, seeing familiar references can reduce tension, stress, boredom, confusion, anxiety, and alienation, while increasing their self-assurance, calm, interest, engagement and dependence upon the user interface.”
70
Designing with Metaphors
MARCUS, 1994 MADSEN,1994 CARROL ET AL. 1988
ORGANISE: provide simple, clear, consistent
metaphors
Listen to how users understand their computer
Identify candidate metaphors
ECONOMISE: maximise effectiveness of minimal
set of metaphors
Build on existing metaphors
Detail metaphor / software matches with respect to
user scenarios
COMMUNICATE: match metaphors to user
capabilities
Use predecessor artefacts as metaphors
Identify likely mismatches and their implications
MANAGE MISMATCHES Note metaphors already implicit in problem
description
Identify design strategies to help minimise
mismatches
71
Smilowitz, “Do metaphors make web browsers easier to use?”
http://www.baddesigns.com/mswebcnf.htm
72
Integral metaphor (library) ‘Standard browser’ interface (no metaphor)
73
Integral metaphor (travel) Composite metaphor (library + travel)
74
Metaphor - Cons
“Those who think that one should use metaphors in design are destined to produce crappy designs.” (DON NORMAN)
PIRHONEN (2003) A metaphor cannot cover the whole domain of the referent
And so limits functionality of the design (HARRISON ET AL. 1998) Once a person has learned the application, the metaphor becomes redundant
HUDSON (2002) Metaphor can lead to inappropriate assumptions on error recovery
COOPER (1995) Searching for the ‘magic metaphor’ can be one of the biggest mistakes an interface
designer can make A Metaphor could offer a tiny boost to learnability at first but eventually limits
functionality There are not enough metaphors to go around Metaphors do not scale well
75
4. Use of Colour and Highlighting
76
Contrast Effects
This text is quite easy to read because the contrast between figure and ground is high
This text is more difficult to read because the contrast between figure and ground is lower
This text is very hard to read because the contrast between figure and ground is minimal
77
Galitz, W.O., 1996, The Essential Guide to User Interface Design Potential problems:
screen backgrounds being more attention grabbing than screen data
overuse of colour as a code (the colour itself meaning something to the screen viewer), which forces the user to interpret a colour’s meaning before the message it is communicating can be reacted to.
78
Galitz, W.O., 1996, The Essential Guide to User Interface Design
Use colour to assist in formatting a screen:
relate or tie elements into groupings break apart separate groupings of information associate information that is widely separated on the
screen highlight or call attention to important information by
setting it off from other information
79
Tufte, E., 1989, Visual Design of the User Interface1. The border of an active window should be light in
value (to avoid clutter with other windows), yet deeply saturated (to provide a conspicuous signal)
2. Yellow is the only colour jointly satisfying both these conditions, and therefore proves valuable for bordering windows
3. A good way to avoid colorjunk is to use colours found in nature, particularly towards the lighter side, such as blues, greys, yellows
4. Nature’s colour are familiar and have a widely accepted harmony
80
Apple Computer Inc., 1992, Macintosh Human Interface Guidelines Use of Colour in Windows:
Distinguishes the active window from other windows and enhances the appearance of user controls on the window frame.
Scroll bars and title bars are gray User controls are coloured to make them more apparent Inactive window borders are gray to make them recede into background and active windows black
border stands out. User can change colours from colour control panel User chosen highlight colours will be used in windows and dialog boxes.
Limit the number of Colours To maintain consistency with operating system, use as few colours as possible. Fewer colours results in less visual clutter on the screen
Colours on Gray Colours look best against a background of neutral gray Colours will stand out more if the background and surrounding areas are gray black, or white
Beware of Blue Light Blue is the colour most difficult to distinguish. Avoided for text, thin lines, and small shapes. If you want things to be unobtrusive, thought, light blue is the perfect colour.
81
Factors that Affect Contrast and VisibilityVariable Effect Example
Contrast Visibility Black print on grey
Illumination Contrast Sensitivity (CS)
Reading map in poor light
Polarity Black on white better than white on black
Designing slides
Spatial frequency Optimum CS at 3 C/D Ideal size of text font given viewing distance
Visual accommodation CS Map reading during night driving
Motion CS Reading a road sign while moving
Wickens et al., 1998, An Introduction to Human Factors Engineering
82
Data-driven perception
Activation of neural structures of sensory system by pattern of stimulation from environment
83
Theory-driven perception
Perception driven by memories and expectations about incoming information.
84
Exercise
x
i. X on outside front cornerii. X on insideback corner
85
Visual Illusions
Old Woman or Young girl?
Rabbit or duck?
http://www.genesishci.com/illusions2.htm
86
KEYPOINT
Perception limits are set by sensory / neural mechanisms; but beyond these limits, perception can be cognitively controlled;
Sensory experiences interpreted in a CONTEXT and derive from a variety of sources
87
Conclusions
Humans constantly seek patterns in the world around them…
…and use these patterns to impose meaning on what they see or hear.
Good design encourages this search for pattern…
…bad design makes people see patterns when none are intended.
88
5. Commands and Menus
89
Commands and Icons
GREP –V ^$filea>fileb
A B
90
Command Languages
Advantages
Fast
Efficient
Precise
Concise
Flexible
User initiated
Disadvantages
Training
Regular use
High memory load
Poor error handling
91
Commands Languages
Command languages rely on recall Draper (1985)
UNIX users poor recall on written test, but excellent recall when using computer
Black and Moran (1982) Learn 8 commands. Performance best for ‘infrequent,
discriminating’ words, e.g., insert, delete. Random words also good.
92
Knowledge in the HeadKnowledge in the World“Menus relieve users of the need to remember
command names, but not of the need to know what functions can be performed by some commands…”
[Mayes et al., 1988]
Knowledge held by users and recalled‘Knowledge’ prompted by objects in world
93
Command Languages Guidelines
Kidd (1982): Choose memorable, nonconfusable command word Use consistent command formats Keep command strings short Provide help Use natural syntax Place optional / least used commands at end of list Ensure useful defaults If errors frequent, then revert to computer initiated
94
Menus
Advantages
Minimal typing
Low memory load
Defined structure
Disadvantages
Speed
Screen space
Poor for data entry
Computer initiated
95
Menus
Single menu Binary options
Do you want instructions? Y / N
Linear sequence Series of interdependent menus
Sequence of forms for printing
Tree structure Semantic network
8 items per menu x 2 levels for time and error
96
Tree structure
Menu Trees reduce error rate when compared with random Search faster over trees than alphabetic or
random Semantic networks
97
Menu Guidelines
Task semantics Meaningful groupings Short cuts Items brief Consistent terminology Headings relevant Limit to 8 items
98
Icons
Concrete vs. Abstract
Reference and interpretation
Verbal labels can speed response
99
Icon design
Marcus (1992) Lexical: pixel shape, colour, blinking Synatatics: lines, pattern, size, shape Semantics: concrete, abstract, part Pragmatics: legibility, utility, identification,
recognition Dynamics: highlighting, selection
100
Icon Guidelines
Represent object in familiar manner Limit number of different icons Make icon conspicuous Ensure selection made clear Ensure icons belong to ‘families’
101
Working Memory Experiments
102
Central executive
Articulatory control process
Auditory word presentation
Visual word presentation
Phonological store
Visual Cache
Inner scribe
Baddeley’s (1986) Model of Working Memory
103
Slave Systems
Articulatory loop Memory Activation Rehearsal capacity
Word length effect and Rehearsal speed Visual cache
Visual patterns Complexity of pattern, number of elements etc
Inner scribe Sequences of movement Complexity of movement
104
Recognition vs Recall
Recall Retrieve from memory
Remembering print command
Recognise To bring back into awareness through prompt
Recognising print icon Recognising print item in menu
105
Recognition vs Recall
“The boxer chewed the meat” Recall: who chewed the meat? Recognition: did the boxer chew the meat?
Recognition easier than recall More information retrieved under recognition
than recall
106
Long Term Memory as a Semantic Network
ANIMAL
Has SkinCan moveEatsBreathes
BIRDCan fly
Has WingsHas feathers
FISHHas finsCan swimHas gillsCANARY
Is YellowCan sing
107
Levels and Reaction time
A canary is a canary
A canary is a bird
A canary is an animal
A canary is a fish
A canary can sing
A canary can fly
A canary has skin
A canary has gills
Collins & Quillian, 1969
0.9
1
1.1
1.2
1.3
1.4
1.5
0 1 2 False
Levels of Sentences
Mean
Reacti
on
Tim
e (
s)
Property
Category
108
Canaries
Different times to verify the statements: A canary is a bird A canary can fly A canary can sing
Time proportional to movement through network
109
Spreading Activation
When searching semantic network, activate paths for search
The active a node, the more easily information can be obtained from it
Semantic Priming Effects
110
Forgetting
Encoding failure Failure of consolidation
Storage failure Disruption by new or existing information Associative interference
Two responses associated with same stimulus
Retrieval failure
111
6. Navigation in User Interfaces
112
Navigating Menus
113
Hypertext
Bush (1945) Nelson (1963) Hyperlinks between objects in
document Navigation
Following links through the document
114
Navigation Aids
115
Searching the World Wide WebScanning modes Information need Information seeking Information use
Undirected Viewing General areas of interest; specific need to be revealed
Sweeping
Scan broadly a diversity of sources, taking advantage of what's easily accessible"
Browsing
Serendipitous discover
Conditioned Viewing Able to recognize topics of interest
"Discriminating"Browse in pre-selected sources on pre-specified topics of interest
"Learning"Increase knowledge about topics of interest
Informal Search Able to formulate simple queries
"Satisfying"Search is focused on area or topic, but a good-enough search is satisfactory
"Selecting"Increase knowledge on area within narrow boundaries
Formal Search Able to specify targets in detail
"Optimizing"Systematic gathering of information about an entity, following some method or procedure
"Retrieving"Formal use of information for decision-, policy-making
http://www.firstmonday.org/issues/issue5_2/choo/#fig1
116
Choice of Aid x Search Style
Hammond and Allinson (1989)
117
Information Foraging
Priolli, P. ,2007, Information Foraging Theory ‘information scent’ left on path of search by users More users on path means more ‘scent’ Users attempt to predict what information will
obtained from following a given path
118
6. Interaction Devices: Entering Data
119
Generic Forms of Interaction
Select Object Drag Object Change Orientation of Object Enter Data
120
ConversationKevin: Want to see that film?
Brian: the uh (500 ms)… with the bloke from that other um that prison thing, [Shaw…]
Kevin: [The Shawshank Redemption] Tim something – [married that bird with the eyes]
Brian: [Robbins](laughs) Yeah…. 'The Green… whatsit?
Kevin: OK. There’s one at uh ... (looks at watch - 1.2 s) ... 20 to?
121
Human Communication
Units of analysis Proposition Lack of ‘proper’ grammar
Adjacency Pairs Question-answer
Turn-taking Over-lapping
Extralinguistic and Paralinguistic cues Back channels Gestures
122
Turn Taking
When it is your ‘turn’? Subtle cues from partner
When they are finishing When they want to speak
Unsubtle cues from machine Speak after beep Respond from list
123
Asymmetrical Conversations
Telephone Short phrases Task oriented Problem of silence
Expert / naïve 90% of doctors ‘turns’ are questions Negotiation of knowledge level
124
Is speaking to machines natural?
Turn taking Input / Feedback Turn taking cues
Asymmetrical partners Domain / World Knowledge Goals Language abilities Interpretation of ‘tokens’
125
Is speaking to machines natural?
Politeness You don’t say ‘please’ to a machine
Social Norms What happens when machine violates norms? Should synthesised speech sound human? What are the ‘norms’?
126
Back channelling from machines
MicroSoft Office Assistant
Process Control training simulation
Nodding ATM / TVM
127
Breakdown and Repair
Redundancy Shared knowledge / anticipation Focus signals
Given/new Deixis
Comprehension signals Communication signals
128
Eye contact and gaze
to convey interest signal comprehension and communication provide sense of engagement establish social presence
129
Video tunnel
monitor
camera
mirror
half-silvered mirror
130
7. Interaction Devices: Selecting Objects
131
Deixis
Pointing to things using
words and gestures:
“Please can I have that.”
“Please can I have that tart.”
“Please can I have that fruit tart.”
“Please can I have that strawberry tart.”
132
Deixis in HCI
Point Move cursor onto object Indicate object (highlighting)
Select Click button
Manipulate Click button
133
Select Object Step One: Indicate Object
Enter Objects Name or I.D. Type Write Speak
Position Selector on Object Direct: touchscreen, stylus, pen Indirect: cursor (using mouse, trackball, joystick)
Step Two: Confirm Selection Press key to confirm Press button to confirm Tap object to confirm
134
Device Models
Buxton’s 3-state device model
State0
State1
State2
135
Application
State0
State1
State2
Out of range
Pen on
Pen off Button up
Button down
select drag
136
Different pointing devices
Device State0 State1 State2
Touchscreen X
Pen X X X
Joystick X X
Mouse X X
137
8. Multimodal Human Computer Interaction
138
Divided Attention
Dual tasks require people to divide attention
Limited attentional resource that is shared between tasks
Depends on tasks, e.g., talk and drive, sight read music and shadow speech
139
Allport et al. (1972)
Participants presented with essay, either visual and auditory
Recognition test far worse with auditory 0
10
20
30
40
50
Pictures Visual
Words
Auditory
Words
Control
Experiment
Err
ors
in R
eco
gnit
ion (
%)
140
Task Similarity Interference when use same stimulus modality – visual or
auditory
Interference when use same stage of processing – input: central: output
Interference when use same memory codes – verbal or visual
Interference when use same response codes – spoken or manual
141
Practice and Expertise
Highly practised dual-task performance, e.g., 6 weeks read + take dictation
Expert pianists can sight-read + shadow; expert typists can touch-type + shadow
Performance strategies Reduced demand Reduced resource load
142
Contrasting Definitions
Technological Computers can present information using different
display modes and can receive information from different devices
Human Humans can receive information via different
senses and perform actions using different effectors
143
Modalities
Modality Refers to human sensory modalities
Sight Hearing Touch Smell , Taste
and response modalities Speech Manual
Multimodal system Supports HCI using more than one sensory and
response modality
144
Types of Multimodal System
Task
Goal
Dependent Independent
Independent Exclusive Concurrent
Dependent Alternate Synergistic
145
Exclusive
Separate tasks, each with their own goals Performance on tasks interact Example:
Surveillance system with camera control Enter data + control camera Data entry delayed by camera control
146
Alternate
Separate tasks with shared goals Performance on tasks interact Example:
Surveillance and target identification Control camera and zoom-in on targets Identify and report targets
147
Concurrent
Separate tasks, each with their own goals Tasks do not affect each other Example:
Target identification with vehicle control Identify targets + manoeuvre vehicle
148
Synergistic
Separate tasks with shared goal Tasks do not affect each other Example:
Mark distance on map ‘How far from here to here?’ Pen + speech
149
Put That There
Spoken commands
Manual gestures
Bolt (1980)
Put that red
triangle…
…there
150
IBM OpenSpace
151
QuickSet
152
Pointing and Speaking
People do not interact multimodally with multimodal interfaces
Speech and pointing is not dominant Around 20% of
conversation, and 14% of multimodal HCI
153
Sequencing and Timing
Not all actions are simultaneous
99% of pen + speech has pen actions first
154
Languages of Multimodal HCI
Speech: ‘place a boat dock on the east, no, the west end of Reward Lake’
Pen: [draws rectangle] ‘add dock’
155
Future Multimodal HCI
Development of future multimodal systems “…depends on knowledge of the natural integration patterns that typify people’s use of combined input modes.”
“The design of multimodal systems that blend modes synergistically depends on…”
properties of different modes the information they conveyhow multimodal input is synchronized
156
9. Future Human Computer Interaction
157
Weiser’s Scales of Interaction
Inch, foot, yard Tab, pad, board
158
Inch = Pad
ParcTAB PDA Individual many networkable
159
Foot = pad
Notebooks Tablet E-paper Individual several
160
Yard = board
Smartboards Plasma displays Control mimics Public sharable
161
Forms of interaction
Implicit interaction devices Sensors Awareness Recognition technologies Natural
Explicit interaction devices Keyboard, pointing
162
WIMP considered ‘fatal’?
Brad Rhodes The user has screen real-estate to burn
too much information on screen
The user has Fine motor control Eye-hand coordination
Digital information is a primary task
http://www.hitl.washington.edu/people/grof/VRAIS98/Rhodes.html
163
Wearable WIMP
Clear mapping between user action and goal Windows
Multiple views for multiple functions Icons
Presentation of graphical signs and symbols for prompts and commands
Menus Presentation of restricted option set
Pointing devices Object selection and manipulation
164
Windows on Wearables
Unclutter world Minimal information views Design display to augment world
Emergent features Design display to support pattern recognition
Integrate display Design display to interact with world
165
WIMP considered ‘fatal’?
Brad Rhodes The user has screen real-estate to burn
too much information on screen
The user has Fine motor control Eye-hand coordination
Digital information is a primary task
http://www.hitl.washington.edu/people/grof/VRAIS98/Rhodes.html
166
Wearable WIMP
Clear mapping between user action and goal Windows
Multiple views for multiple functions Icons
Presentation of graphical signs and symbols for prompts and commands
Menus Presentation of restricted option set
Pointing devices Object selection and manipulation
167
Windows on Wearables
Unclutter world Minimal information views Design display to augment world
Emergent features Design display to support pattern recognition
Integrate display Design display to interact with world
168
Icons on Wearables
Reduce text
Reduce information load
169
Text vs. Graphics
Baber et al., 1999, Mobile Networks and Applications 4
Mean Reaction Times
Graphical TextualHMD 1.45s 1.85sVDU 1.0s 1.5s
170
Menus on Wearables
Clarify options
Restrict activity
Access and enable through world activity
171
Emergent Properties
Baber et al., 1999, Interact’99
172
Limited Options
Bristow, WECAPC
173
Integrate with world
Baber et al., 2000, Interact’00
174
Pointing on Wearables
Selection on display
Selection in the world
175
Selection in the World
Baber et al., 2000, Interact’00
176
Selection from Display
Carnegie-Mellon Rotary dial selector
Accelerometer for simple gestures
Speech recognition
Bass et al., 1997, CHI’97
177
Augmenting the World
178
Digital Desk
Real paper on desk is scanned by a camera,
computer images are projected onto the paper
179
Graspable Objects
BUILD-IT Move blocks on
table
Change layout of room on displays
Fjeld 1999
180
GeoSPACE
Hiroshi Ishii – MIT
Physical objects to manipulate map display
181
Rekimoto’s InfoBoard
Pick and place objects using pen
Move objects between laptops or virtual storage
182
Data Storage
Data containers from Philips
183
Data tiles
Magnetic tiles linked to ‘data’
Projection labels tiles
184
Rekimoto’s Data Tiles
185
Haptic VR
Provide touch to virtual objects using force-feedback devices
186
Haptic Augmentation
Use real objects to control virtual representations
187
Handheld AR
PDA used to provide ‘views’ on world
Views change with PDA movement
188
Information Appliances
189
Domestic Appliances
Microprocessors in cars, dishwashers, washing machines, televisions etc.
Domestic appliances as ‘embedded systems’
190
Norman (1999)
Information Appliance:
“An appliance specializing in information: knowledge, facts, graphics, images, video, or sound. An information appliance is designed to perform a specific activity, such as music, photography, or writing. A distinguishing feature of information appliances is the ability to share information among themselves.”
D.A. Norman, 1999, The Invisible Computer, MIT Press, p. 53
191
Pervasive / Ubiquitous
Invisible Computer
Embedded Systems
Networks
192
Information Appliances
Digital camera: Capture image Immediate (shared) viewing of image Download image onto computer
Edit image Email image
193
Three Axioms
Design Axiom 1: Simplicity The complexity of the appliance is that of the task not
the tool. The technology is invisible.
Design Axiom 2: Versatility Appliances are designed to allow and encourage novel,
creative interaction
Design Axiom 3: Pleasurability (fun) Products should be pleasurable, fun, enjoyable. A joy to
use, a joy to own.
D.A. Norman, 1999, The Invisible Computer, MIT Press, p. 67
194
Current Trends
Device marriages digital camera + telephone PDA + telephone + MP3 + digital camera
Problems Visibility
Which device is currently ‘on’ Accessibility
How to switch devices / modes? Functionality
What functions are actually used?
195
11. Distributed Cognition
196
Using an Abacus for Addition
6 + 2 = 8
Heaven beads = 5 each
Earth beads = 1 each
8 + 2 = 10
197
Shopping Lists
Construct list Using the process of writing the list to support
decision making Remember to consult list
Check the list during shopping Use the list to help navigate the store
Reading and interpret list Make sure everything is bought
198
Shopping Lists
Memory aid Do we only buy what’s on the list? Do we buy items not on the list?
Additional tasks Does writing the list create a new task?
Modified task Does using the list change the way we shop?
199
‘Things that make us smart’
Cognitive Artefacts used to assist everyday activity Shopping List Knotted handkerchief Calculator Diary
200
Using Lists
Construct list Pre-computation [Hutchins]
Remember to consult list Checklists in inspection
Reading and interpret list Following checklists accurately
201
Cognitive Artefacts
Distribute actions across time Pre-computation
Distribute actions across actors Distributed cognition
Change actions required
202
Artefacts & Representation
Surface representation Display and maintenance of symbols on a
visible surface
Internal representation Storage and organisation of symbols
External representation
203
Mapping
Naturalness Related to directness of mapping Related to expertise / familiarity
Appropriateness Information should be appropriate to the task
(neither more nor less)
204
Distributed Cognition
Share knowledge and information across people, things, actions
Shopping lists
Ed Hutchins on calculating ships speed
E. Hutchins, 1990, The technology of team navigation, In Galegher et al. Intellectual Teamwork, LEA
205
Calculating Ships Speed
1. D=RT, (R=D/T) using pencil and paper
2. D=RT, using calculator
3. 3-minute rule: 3-minutes = 1/20 of an hour, and 100yds in 1/20 of a nautical mile
1500 yds in 3 minutes = 15nmph
206
Calculating Ships Speed
4. Nautical Slide Rule
207
Calculating Ships Speed
Knowledge-in-the-head [1] 2000 yds nautical mile 60 minutes in hour D=RT Transposition of equations
Knowledge-in-the-head [2] 3-minute rule: schema for shortcut
208
Calculating Ships Speed
Knowledge-in-the-World Nautical slide rule
Replace calculation with manipulation Colleagues
Draw upon experience of others Teamwork
Plotter Bearing taker Bearing timer-recorder
209
12. Collaborative Working
210
Traditional HCI
One User One Computer One Location One Task?
211
Limitations of this view…
Task focus on computer AND on other artefacts, e.g., paper, telephone, filing cabinet etc.
Other artefacts imply more than one location Office work often collaborative SO: computer only part of the work domain and is
not designed to fit with other parts
212
Personal Computers in the Office
1970s: development of the ‘desktop metaphor’ Objects on screen analogous to objects in world,
e.g., folders, files etc. Manipulation of objects on screen analogous to
manipulation of objects in world, e.g., open folders, put files into folders
BUT: no link between virtual and real objects
213
Changing HCI
Many Users Communication and Collaboration Multiple Access (WWW)
Many Computing Devices Embedded Systems PDAs, cell-phones and beyond
Many Locations Physical location (mobility) Virtual locations (WWW)
Many Tasks
214
Computer Mediated Communication (CMC)
Same place Different place
Same time
(synchronous)
Face-to-face Telephone
Different time
(asynchronous)
Post-in notes Letter
215
CMC
Synchronous Speech: telephone, videophone, videoconference Text: SMS, chatrooms, instant messaging
Asynchronous Speech: voicemail, answerphone Text: bulletin boards, newsgroups, computer conferencing
Mixed with other tasks Electronic Meeting Room
216
Shared Representations
Shared Calendars Access appointments diary to view or edit
(depending on privileges)
Shared Awareness
Shared Activities
217
Shared Awareness
Portholes: Xerox 1992
Low resolution images of people in their offices
Overhearing / overseeing
218
Shared Activities
Work on same text Add comments Restructure
Work on same drawing Modify Suggest alternatives
219
Collaborative Virtual Environments (CVEs)
Distributed VR systems
Individuals collaborate in virtual landscapes to share information and work together
220
CVEs
Meeting places Multiple representations Shared context
Shared knowledge, environment, tasks, objects Awareness of others Communication
221
Reading Virtual Control Room
Avatars
VirtualDisplayBoards
Augmented Displays
VirtualMonitors
ToVirtualPlant
222
Interaction in Virtual Space
With other actors
With virtual objects
With real objects?MagicBook project
223
Collaborating in real space
Technology to bring the virtual world to the real world
Shared representations on familiar surfaces
Surfaces to support interaction
224
RoomWareG
MD
-IPSI, W
iege, Wilkhahn
Source: Carroll, p. 569
ConnecTable
CommChair
InteracTable
DynaWall
225
Shared Awareness
Portholes: Xerox 1992
Low resolution images of people in their offices
Overhearing / overseeing
226
Exercise #.1
1. Apple 2. North 3. Charlie 4. Nickel 5. Red 6. Banana 7. South 8. Fred 9. Cent 10. Green 11. Pear 12. East 13. George 14. Dime 15. Yellow 16. Grape 17. Wayne 18. Dave 19. Penny 20. Blue
Make a note of the word numbers and enter into table
How many people wrote: West or Orange? What pattern do you notice in the words?
227
Exercise #.2
1. Latch 2. Bream 3. Hot 4. Mayor 5. Jab6. Busk 7. Else 8. Wage 9. Clog 10. Jowl 11. Chap 12. Big 13. Smug 14. Duck 15. Trout 16. Blot 17. Reek 18. Tape 19. List 20. Skirt
Make a note of the word numbers and enter into table
What pattern do you notice in the words? Did you do as well as with the first list?
228
Exercise #.3
1. Time 2. House 3. Mit 4. Stab 5. Solve6. Draft 7. Say 8. Off 9. Boil 10. Court 11. Greet 12. Slot 13. Hand 14. Dirt 15. Clot 16. Stale 17. Out 18. Dumped 19. Stone 20. Dice
• Make a note of the word numbers and enter into table
• Did you do as well as with the other lists?
