8/4/2019 HCI-5 Evaluation Techniques
1/40
IAS2223
HUMAN-COMPUTER INTERACTION
Topic 5
Evaluation Techniques
8/4/2019 HCI-5 Evaluation Techniques
2/40
EVALUATION TECHNIQUES
Evaluation
y tests usability, functionality and acceptability of an
interactive system
y occurs in laboratory, field and/or in collaboration
with users
y evaluates both design and implementation
y should be considered at all stages in the design life
cycle
8/4/2019 HCI-5 Evaluation Techniques
3/40
GOALS OF EVALUATION
assess extent of system functionality
assess effect of interface on user
identify specific problems
8/4/2019 HCI-5 Evaluation Techniques
4/40
EVALUATION THROUGH
EXPERT ANALYSIS
Cognitive Walkthrough
Heuristic Evaluation
Model-based evaluation
8/4/2019 HCI-5 Evaluation Techniques
5/40
COGNITIVE WALKTHROUGH
Proposed by Polson et al.
y Require a details review of a sequence of actions
y evaluates design on how well it supports user in
learning tasky usually performed by expert in cognitive
psychology
y expert walks though design to identify
potential problems using psychologicalprinciples
y Main focus: learning through exploration
y forms used to guide analysis
8/4/2019 HCI-5 Evaluation Techniques
6/40
COGNITIVE WALKTHROUGH
Requirements:
i. A specification / prototype of the system
ii. A description of task user is to perform
on the system
iii. A written list of actions needed to
complete the task
iv. An indication about users experience& knowledge
8/4/2019 HCI-5 Evaluation Techniques
7/40
COGNITIVE WALKTHROUGH (CTD)
For each task walkthrough considers
y what impact will interaction have
on user?
y what cognitive processes are
required?
y what learning problems may
occur?
8/4/2019 HCI-5 Evaluation Techniques
8/40
IN EACH ACTION PERFORM:
Analysis focuses on goals and knowledge: does
the design lead the user to generate the correct
goals?
Will users see that the action is available? Does user know that the action is the one that
they needed?
After action is taken, will user understand the
feedback they get?
8/4/2019 HCI-5 Evaluation Techniques
9/40
HEURISTIC EVALUATION
Heuristics - guideline / general principlesthat can guide a decision that has alreadybeen made.
usability criteria (heuristics) are identified
design examined by experts to see if these areviolated
Example heuristicsy system behaviour is predictable
y system behaviour is consistent
y feedback is provided
Heuristic evaluation `debugs' design.
8/4/2019 HCI-5 Evaluation Techniques
10/40
NIELSENS 10 HEURISTICS:
Visibility of system status
Match between system & real world
User control & freedom
Consistency & standardsError prevention
Recognition rather than recall
Flexibility & efficiency of use
Aesthetic & minimal design
Help users recognize, diagnose & recoverfrom errors
Help & documentation
8/4/2019 HCI-5 Evaluation Techniques
11/40
MODEL-BASED EVALUATION
Results from the literature used to support orrefute parts of design.
Care needed to ensure results aretransferable to new design.
Cognitive models used to filter design optionse.g. GOMS (goals, operators, methods, selections)
prediction of user performance.
Design rationale can also provide usefulevaluation information
8/4/2019 HCI-5 Evaluation Techniques
12/40
EVALUATING THROUGH USER
PARTICIPATION
8/4/2019 HCI-5 Evaluation Techniques
13/40
LABORATORY STUDIES
Advantages:y specialist equipment available
y uninterrupted environment
Disadvantages:y lack of context
y difficult to observe several users cooperating
Appropriatey if system location is dangerous or impractical for
constrained single user systems to allow controlledmanipulation of use
8/4/2019 HCI-5 Evaluation Techniques
14/40
FIELD STUDIES
Advantages:y natural environment
y context retained
y Interaction occur in actual use
Disadvantages:y distractions
y noise
Appropriatey where context is crucial for longitudinal studies
8/4/2019 HCI-5 Evaluation Techniques
15/40
Evaluating Implementations
Requires an artefact:
simulation, prototype, full implementation
8/4/2019 HCI-5 Evaluation Techniques
16/40
EMPIRICAL METHODS:
EXPERIMENTAL EVALUATION
controlled evaluation of specific aspects of
interactive behaviour
evaluator chooses hypothesis to be tested
a number of experimental conditions are
considered which differ only in the value of
some controlled variable.
changes in behavioural measure are
attributed to different conditions
8/4/2019 HCI-5 Evaluation Techniques
17/40
EXPERIMENTAL FACTORS
Participants
y who representative, sample size
Variables
y things to modify and measure Hypothesis
y what youd like to show
Experimental design
y
how you are going to do it
8/4/2019 HCI-5 Evaluation Techniques
18/40
VARIABLES
independent variable (IV)
characteristic changed to produce different
conditions
e.g. interface style, number of menu items
dependent variable (DV)
characteristics measured in the experiment
e.g. time taken, number of errors.
8/4/2019 HCI-5 Evaluation Techniques
19/40
HYPOTHESIS
prediction of outcomey framed in terms of IV and DV
e.g. error rate will increase as font size decreases
null hypothesis:y states no difference between conditions
y aim is to disprove this
e.g. null hyp. = no change with font size
8/4/2019 HCI-5 Evaluation Techniques
20/40
EXPERIMENTAL DESIGN
within subjectsy each subject performs experiment under each
condition.
between subjectsy each subject performs under only one condition
8/4/2019 HCI-5 Evaluation Techniques
21/40
ANALYSIS OF DATA
Before you start to do any statistics:y look at data
y save original data
Choice of statistical technique depends ony type of data
y information required
Type of datay discrete - finite number of values
y continuous - any value
8/4/2019 HCI-5 Evaluation Techniques
22/40
ANALYSISSTATISTICAL TECHNIQUES
parametricy ANOVA (analysis of variance)
y robust
y powerful
non-parametricy do not assume normal distribution
y less powerful
y
more reliable
contingency tabley classify data by discrete attributes
y count number of data items in each group
8/4/2019 HCI-5 Evaluation Techniques
23/40
ANALYSIS OF DATA(CONT.)
What information is required?
y is there a difference?
y how big is the difference?
y
how accurate is the estimate?
Parametric and non-parametric tests mainly
address first of these
8/4/2019 HCI-5 Evaluation Techniques
24/40
8/4/2019 HCI-5 Evaluation Techniques
25/40
SUBJECT GROUPS
larger number of subjects
more expensive
longer time to `settle down even more variation!
difficult to timetable
so often only three or four groups
8/4/2019 HCI-5 Evaluation Techniques
26/40
DATAGATHERING
several video cameras
+ direct logging of application
problems:y synchronisation
y sheer volume!
one solution:
y record from each perspective
8/4/2019 HCI-5 Evaluation Techniques
27/40
8/4/2019 HCI-5 Evaluation Techniques
28/40
THINKALOUD
user observed performing task
user asked to describe what he is doing andwhy, what he thinks is happening etc.
Advantagesy simplicity - requires little expertise
y can provide useful insight
y can show how system is actually use
Disadvantagesy subjective
y selective
y act of describing may alter task performance
8/4/2019 HCI-5 Evaluation Techniques
29/40
COOPERATIVE EVALUATION
variation on think aloud
user collaborates in evaluation
both user and evaluator can ask each other
questions throughout
Additional advantages
y less constrained and easier to use
y user is encouraged to criticize systemy clarification possible
8/4/2019 HCI-5 Evaluation Techniques
30/40
PROTOCOL ANALYSIS
Methods for recording user actions:
paper and pencil cheap, limited to writing speed
audio good for think aloud, difficult to match with otherprotocols
video accurate and realistic, needs special equipment,obtrusive
computer logging automatic and unobtrusive, largeamounts of data difficult to analyze
user notebooks coarse and subjective, useful insights, goodfor longitudinal studies
Mixed use in practice.
audio/video transcription difficult and requires skill.
Some automatic support tools available
8/4/2019 HCI-5 Evaluation Techniques
31/40
AUTOMATED ANALYSIS EVA(EXPERIMENTALVIDEOANNOTATOR)
Workplace project
Post task walkthrough
y user reacts on action after the event
y used to fill in intention
Advantages
y analyst has time to focus on relevant incidents
y avoid excessive interruption of task
Disadvantagesy lack of freshness
y may be post-hoc interpretation of events
8/4/2019 HCI-5 Evaluation Techniques
32/40
POST-TASK WALKTHROUGHS
transcript played back to participant for
comment
y immediatelyp fresh in mind
y delayedp evaluator has time to identify questions
useful to identify reasons for actions and
alternatives considered
necessary in cases where think aloud is not
possible
8/4/2019 HCI-5 Evaluation Techniques
33/40
QUERY TECHNIQUES
Interviews
Questionnaires
8/4/2019 HCI-5 Evaluation Techniques
34/40
INTERVIEWS
analyst questions user on one-to -one basisusually based on prepared questions
informal, subjective and relatively cheap
Advantagesy can be varied to suit context
y issues can be explored more fully
y can elicit user views and identify unanticipated
problemsDisadvantages
y very subjective
y time consuming
8/4/2019 HCI-5 Evaluation Techniques
35/40
QUESTIONNAIRES
Set of fixed questions given to users
Advantages
y quick and reaches large user group
y can be analyzed more rigorously
Disadvantages
y less flexible
y less probing
8/4/2019 HCI-5 Evaluation Techniques
36/40
QUESTIONNAIRES (CTD)
Need careful design
y what information is required?
y how are answers to be analyzed?
Styles of question
y general
y open-ended
y scalary multi-choice
y ranked
8/4/2019 HCI-5 Evaluation Techniques
37/40
PHYSIOLOGICAL METHODS
Eye tracking
Physiological measurement
8/4/2019 HCI-5 Evaluation Techniques
38/40
EYE TRACKING
head or desk mounted equipment tracks the
position of the eye
eye movement reflects the amount of
cognitive processing a display requiresmeasurements include
y fixations: eye maintains stable position. Number
and duration indicate level of difficulty with
display
y saccades: rapid eye movement from one point of
interest to another
y scan paths: moving straight to a target with a
short fixation at the target is optimal
8/4/2019 HCI-5 Evaluation Techniques
39/40
8/4/2019 HCI-5 Evaluation Techniques
40/40
CHOOSINGAN EVALUATION METHOD
Factors distinguish evaluation techniques:
when in process: design vs. implementation
style of evaluation: laboratory vs. field
how objective: subjective vs. objective
type of measures: qualitative vs. quantitative
level of information: high level vs. low level
level of interference: obtrusive vs. unobtrusiveresources available: time, subjects,
equipment, expertise