Embed Size (px)
Citation preview
Suzanne Kieffer
ECOVAL: A Framework for Increasing the Ecological Validity in Usability Testing
Jean VanderdoncktUgo Braga Sangiorgi
Université catholique de Louvain Louvain School of Management Louvain-la-Neuve, Belgium
Ecological validity
Experimental design
Natural setting
x x
xxxxxx
xx x x
x xxx
Environment
Tasks
Stimuli
Ecological validity
High
Low
Experimental design
Natural setting≈
Experimental design
Natural setting≠
Internal validityWas the study well done?
External validityCan we generalize the findings to particular persons, settings and times?
Ecological validityTo which extent can we generalize the findings?
User interface design
Ecological validity
Overall validity
Conclusions of usability evaluations
What’s the problem with ecological validity?
No formal definition that enables To objectively quantify the ecological validity of
experimental designs
To capture the critical attributes of the real-world environment
LOGICAL IDITYVALECO ECOVALA framework for increasing the ecological validity in usability testing
Outline The ECOVAL framework
ECOVAL Guidelines GAMBIT
Case study Operability and usefulness of ECOVAL ROI of increased ecological validity Recommendations
The ECOVAL framework
The guidelines
No objects
Mock objects
Real objects
Only verbalized
Mimicked/verbalized
For real
GAMBIT – the supporting tool Production of prototypes
Easy Quick Cost-effective
Real user interface performances Interaction Navigation Responsiveness
GAMBIT – how to
1
Produce screens
GAMBIT – how to
2
Create behaviour
GAMBIT – how to
3
Conduct experiment
GAMBIT – how to
4
Download log file
• Q1: To which extent is ECOVAL applicable for controlling the ecological validity?
• Q2: What are the possible outcomes/benefits of increased ecological validity ?
Case study
Hot-Dip Galvanizing on Continuous Lines
Monitoring report
Task Control Result
Task Control Result
Artefact:paper checklist
Section > Subsection
Function – Name - Date – Shift
Section > Subsection
300 items a day
Timeline
Task analysis
User profile
Usability goal
setting
Screen design
Usability evaluation
Usability evaluationPaper versus GAMBIT prototype
Ecological validity
Ratio Paper-to-Gambit for ecological validity of 1:1.17
Balsamiq mockups
Experimentations
Paper GAMBIT
5 key users x 2
10 participants (30’) 18 participants (1 hour)
5 key users x 3+ 3 user representatives
V16 participants
V212 participants
New version of the prototype
Apparatus
Results and discussion
Early detection of usability problems Increased user efficiencies Increased user satisfaction
Cost-benefit analysis
130 hours
18 hours
Cost-benefit analysis (continued)
$74,480
$194,480
Paper GAMBIT
Ecologicalvalidity 1 1.17
Datacollection
Manual & face-to-faceDifficult & tedious Unstructured processObtrusive
Automated & remoteComfortableStructured processUnobtrusive
Outcomes Compliant with the literature
Early detection of usability problemsIncreased user productivityDecreased late design changesIncreased organizational efficiency
Recommendations
Take away
{ECOVAL, Guidelines, GAMBIT}
Case study Operability Usefulness ROI
Recommendations
