WCIT 2014 Peter Elkin - Human computer interaction, evaluation, usability testing, study design and methods

Human Computer Interaction Evaluation, usability testing,

study design and methods

Presented by: Peter L. Elkin, MD, MACP, FACMI

With Studies from EvaLAB

Marie-Catherine Beuscart-Zéphir, PhD

1

Purpose

The purpose of this workshop is to discuss why human factors engineering is important for safe and effective clinical systems and to learn the Usability methodology and how to develop and run a Usability study

3

Learning ObjectivesIdentify when to employ Usability TestingDiscern how to apply Usability testing to the

development of clinical systemsDescribe how to run a Usability studyDescribe how to interpret the results of a

usability studyDecide when not to apply Usability testingRecognize the linkage between patient safety

and usable systems

4

UsabilityUsability is the “effectiveness, efficiency and

satisfaction with which a specified set of users can achieve a specified set of tasks in a particular environment.” (International Standard Organization: ISO 9241)

Usability is the EXTENT to which a product, application, is « usable »A product is more or less usableUsability can be measured

5

Human-centred designdevelopment cycle (ISO 13407)

Plan the human-centredprocess

Understand and specifythe context of use

Specify the user and organizational requirements

Producedesign solutions

Evaluate designs againstrequirements

Meetsrequirements?

6

ISO 13407 & ISO/IEC 12207ISO/IEC-12207 primary lifecycle

Acquisit. & supply

DevelopmentOperation & mainten.Require.

anal.Architect. Design

Qualific. testing

ISO13407

Context of use N/A N/A N/A

User & org. requir. N/A N/A N/A

Design solutions N/A N/A

Evaluate

7

UsabilityEffectiveness: accuracy and completeness with which users

achieve specific goals

Efficiency: resources expended in relation with effectiveness

Satisfaction: comfort and acceptability of use

Context of use: users, goals, tasks, equipment, physical and

social environment

Task: a goal (what’s to be done) and the conditions

(constraints) to reach the goal

Activity: the set of actions (mental and behavioural) carried out

by the user to perform the task

8

Usability (cont’d)The evaluation of the usability of a product can

have different objectives:Improve the usability of an application (formative

evaluation): Usability EngineeringDuring the design and development phase of

this application (traditional usability engineering lifecycles)

Via the re-engineering of an existing applicationAcquire or not the product, choose the most usable

product (normative evaluation)

9

Usability engineering« Usability engineering provides structured

methods for achieving usability in user interface design during product development » (Deborah Mayhew, 1999)

Usability Engineering is intertwined with Software Engineering and integrated in the Project lifecycle

10

Usability engineering

Understand the project (goals, expected benefits)

Modelof the currentwork situation

Model of there-engineeredwork situationUsability goals

Iterative usability assessment

Usability problemsrecommendations

Analysis of the current work situation

• Analysis, description and model of the context of use

• Analysis, description and model of users, their tasks and activity

• Usability assessment of existing systems

Project lifecycleSystem engineering

Goals, Expected benefits (quality, safety, investment)

Users needsRequirements

SpecificationDevelopment of a concept

Implementation of theconcept

Release

Requirements verification

Specification verification

Product verificationIterative testing

Validation

11

The usability engineering toolboxTheories, conceptual frameworks, models,

methods, techniques, devices …

From various fields: cognitive psychology, cognitive ergonomics, cognitive engineering, usability, ethnology, sociology, software engineering …

12

COGNITIVE psychology, ergonomics, engineering

Theories and modelsDecision making, process control, human error

Methodologies, frameworksCognitive Task Analysis (CTA); activity analysis

MethodsInterviews, questionnaires, protocol analysis,

documents analysis, audio/video-recorded observations, …

13

UsabilityMethods:

Usability inspection: heuristic evaluation, cognitive walkthrough

Usability testsFocus groups

Techniques, devices:Usability labs

14

Rationale for the use of HFESo What?

Who Cares?

What is in it for you?

15

So What?IBM – “2/3 of all healthcare dollars spent on IT

are spent on Systems which are never used or are used for less than 3 months.”

Patient Safety can be compromised by poorly designed systemsHas led to deaths Is a source of unnecessary liability

16

Who Cares?Administration

Limiting Costs (Mayo saved 1.5M, day 1)Physicians

Limiting ErrorsIncreased Efficiency

PatientsGreater Safety

17

What is in it for you?Greater ReliabilityGreater AcceptanceLower Cost of ImplementationLower Cost for TrainingIncreased Patient SafetyImproved Clinical Reputation of your

Organization

18

Reasons for Hard-to-Use Products

19

Reasons for Hard-to-Use ProductsDevelopment Emphasis was on the Machine or

System, Rather than the User

20


System, Rather than the UserTarget Audience is a Moving Target

21


System, Rather than the UserTarget Audience is a Moving TargetLack of Design Verification (Science vs.

“Common Sense”)

22



“Common Sense”)Development Teams are not well Integrated

23



“Common Sense”)Development Teams are not well IntegratedSkills necessary for the interface Design are

Different than those necessary for the Technical Implementation of Systems!

24

User-Centered Design

25

User-Centered DesignRequires an early focus on users and tasks

26

User-Centered DesignRequires an early focus on users and tasksRequires empiric measurement of product usage

27

User-Centered DesignRequires an early focus on users and tasksRequires empiric measurement of product usageRequires iterative design cycles

DesignTestingModificationRe-Design

28

Human Factors Analysis

29

Human Factors AnalysisContextual Inquiry

Understanding End-User Needs

30


Understanding End-User NeedsCompetitive Usability Evaluations

Expose Gaps in Existing Systems

31



Expose Gaps in Existing SystemsLow-Fidelity Prototyping

Rapid turnaround of Identified Program Changes

32





Modified Focus GroupsUnderstand Relevant Workflow and Processes

33





Modified Focus GroupsUnderstand Relevant Workflow and Processes

34

Usability Studies

35

Usability StudiesDevelopers and Evaluators

36

Usability StudiesDevelopers and EvaluatorsObserve without Intervention

37

Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users Pain

38

Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with Imperfection

39

Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with ImperfectionResist the Impulse to Blame the User

40

Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with ImperfectionResist the Impulse to Blame the UserResist the Impulse to Blame Yourself

41

Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with ImperfectionResist the Impulse to Blame the UserResist the Impulse to Blame YourselfAcceptance

42

Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with ImperfectionResist the Impulse to Blame the UserResist the Impulse to Blame YourselfAcceptanceProgress

43

Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with ImperfectionResist the Impulse to Blame the UserResist the Impulse to Blame YourselfAcceptanceProgress

44

Realities of Human Factors Engineering ~Jakob NielsenYour Best Guess Is Not Good Enough

It is impossible to design an optimal user interface just by giving it your best try.

Users will make unexpected misinterpretations of the interface and perform tasks differently than you expect.

45

The User is Always RightAccept the need to make modifications to fit the

users’ needs and expectations.If users have problems with the software, it is not

their fault.

46

The User is Not Always RightCaution: But it does not follow that the most

usable interfaces can be designed just by asking users what they would like. Users may not understand how changes/enhancements could be beneficial.

47

Designers Are Not UsersDesigners have a great deal of computer

experienceDesigners have an inherent enthusiasm about

computer applicationsDesigners know the conceptual foundation for

the design of the interface“Knowing about a system is a one-way street.

One cannot go back to knowing nothing.”

48

Vice Presidents Are Not Typical UsersCompany executives typically have very different

characteristics than the user population.

49

Usability Engineering is a ProcessThe Usability Engineering process is well refined

and the activities needed to arrive at a good result are fairly constant

However, each project is different and will have different objectives.

50

Why GUI Design Fails

51

Why GUI Design FailsProgrammers/Designers

52


Design to their work patterns, not user work patterns

53


Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptions

54


Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasks

55


Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasksAssume all users will be “Trained”

56


Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasksAssume all users will be “Trained”Plan that users will refer to documentation

57


Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasksAssume all users will be “Trained”Plan that users will refer to documentationProvide all features at the top levels

58


Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasksAssume all users will be “Trained”Plan that users will refer to documentationProvide all features at the top levelsAre unaware of consistency issues or standards

59


Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasksAssume all users will be “Trained”Plan that users will refer to documentationProvide all features at the top levelsAre unaware of consistency issues or standards

60

Bird Flu

61

Usefulness: What Does This Mean?

Utility - does the system encompass needed functionality

Easy to learn

Efficient to use

Easy to remember

Few Errors

Subjectively pleasing

Usefulness -can system be used to achieve desired goals?

Usability -how well users can use functionality?

62

What is a Usability Study?A usability study is a replicable study of

consumer/product or consumer/process interactions conducted in a controlled, simulated environment.

63

Essential ComponentsTo accomplish a valid study, we follow a specific

protocol and have multiple participants interact with the same situations (scenarios).

It is important to observe several participants interacting with the software in order to identify trends and prioritize issues. The goal is to improve the process or product, not to train the participants.

64

What can be studied?Products or Processes True classical experiments with large sample

sizes and complex test designs Informal, less complex studies designed for

rapid processing of results

Note: Each type of study has different objectives, as well as different time and resource requirements.

65

What can be learned?

66

What can be learned?Features/Functions that work

67

What can be learned?Features/Functions that workFeatures/Functions that need improvement

68

What can be learned?Features/Functions that workFeatures/Functions that need improvementUser/Consumer work patterns and mental

models

69


modelsUser/Consumer expectations and desires

70


modelsUser/Consumer expectations and desiresPrioritize areas for improvement - unique

advantage

71


modelsUser/Consumer expectations and desiresPrioritize areas for improvement - unique

advantage

72

Objectives Addressed Through the Study DesignWhat are the general navigational weaknesses? Ease of use / Intuitive nature of application

interface (Do controls and functions make sense?)

User satisfaction?Efficient use of screen

real estate (space)? Are fields and widgets placed intuitively on the

screen?

73

Objectives Addressed Through the Study DesignWill the application make users’ jobs easier? /

Will it make users more productive? Is the system responsive/ fast enough?Is the complexity of the application appropriate

for the defined / intended user group(s)? What On-line Help functions are needed?What type(s) / degree of training is needed?

74

Limitations of Usability Testing

75

Limitations of Usability TestingTesting is always an artificial situation

76

Limitations of Usability TestingTesting is always an artificial situationTest Results do not Prove that a Product Works

77

Limitations of Usability TestingTesting is always an artificial situationTest Results do not Prove that a Product WorksParticipants are rarely truly representative of

their target Population

78

Limitations of Usability TestingTesting is always an artificial situationTest Results do not Prove that a Product WorksParticipants are rarely truly representative of

their target PopulationTesting is not Always the Best Technique

(To Test or Not to Test?)vs. Expert EvaluationFocus Groups

79

When Usability Study

80

When Usability StudyWhen Studying Complex Processes without

option to Study Components Separately

81


option to Study Components SeparatelyLab Environment is Vastly different than User

Environment

82



EnvironmentLab Space will not accommodate the Test

83



EnvironmentLab Space will not accommodate the TestWhen feedback will not be Utilized

84



EnvironmentLab Space will not accommodate the TestWhen feedback will not be UtilizedSubjects cannot be found (Typical Users)

85



EnvironmentLab Space will not accommodate the TestWhen feedback will not be UtilizedSubjects cannot be found (Typical Users)

86

87

Usability Lab

88

89

90

91

92

93

94

95

96

97

98

Illustration

Overview of the re-engineering project of a CPOE application carried out at the EVALAB

Focus on usability tests of the system

99

The projectPartners:

University Hospital of LilleSyndicat Inter-hospitalier de Bretagne (SIB)

Project and context:Re-engineering of an existing Computerized

Physician Order Entry (CPOE) application: Génois

100

The project (cont’d)

Medication Ordering, [dispensing], and administration process

Hospital setting, survey of the therapeutic process during the patient’s stay

101

Medication orderingDaily medical rounds

At bedside

Follow-up of Medication

administration

Medication decision making

Medication ordering

Preparation of 24h pillboxes

Administrationvalidation

patient’s medication orders list











The project: Re-engineering of an existing CPOE application: Génois (SIB)

103











Goals and Expected Benefits

104

Company (SIB): Improve the usability of the

CPOE application (Génois) Support the re-engineering of

the application (re-design HCI)

Hospital: Assess the feasibility of the

installation of this CPOE Improve the quality and

safety of the medication process











Identify the current work situation

105

Identify the Work SituationUsers: physicians, nursesSystems: paper-based / computer-based

106











Analysis of the context of use

Analysis of Users’ tasks & activity

Usability assessment of existing systems

107

Analysis of the Work Situation (1 of 3)

Usability tasks Methods

Analysis of the context of use Interviews: head nurses, managers, senior physicianAnalysis (inventory) of systems’ functionsOn site observations (ethno methodology)

Analysis of Users’ tasks & activity•Individual

•Collective (cooperation)

Cognitive Task AnalysisActivity analysisDocuments analysis (content analysis)Self confrontation interviews

Analysis of communications (oral, written, through the system)

Usability assessment of existing systems Computer-based (CPOE):Usability inspection Usability tests

Paper-based:Analysis of the format, content, availability, redundancy of supports / contentsUsers complaints and incidents observed or reported

108

Analysis of the Work Situation (2 of 3)

Usability tasks Methods






Usability assessment of existing systems Computer-based (CPOE):Usability inspection Usability tests


109

Analysis of the Work Situation (3 of 3)Usability tasks Methods






Usability assessment of existing systems

Computer-based (CPOE):Usability inspection Usability tests


110

Outcomes of the analysis of the work situation: Illustration on two tasks

Physician: information gathering / patient’s current medications for therapeutic decision making

Nurse: information gathering / patient’s current medication for the preparation of medication administration (24 hours pill-box)

111

Physician’s task: comprehend the patient’s list of meds

Usability assessment

Activity analysis• Doctors need the list of current

meds on one page• Need the duration of each

order immediately available (no calculation)

• Need the recent modifications of the treatment highlighted

Context of use• Never performs this task

alone (nurse or resident present)

• Frequency of encounters with the patient varies from twice per day to once in 2 weeks

Paper-based system• The medication order sheets (list

of orders written by the physician) are not usable

• The physicians prefer to use the nurses Medication Administration Record to update their representation of current treatments

CPOE system (inspection, on site observations)

• Obligation of scrolling through the screen page to read the entire list of meds

• Display of modifications of the treatments suffers from lack of intuitivity

• Duration is not directly available (computation required)

Nurse’s task: prepare the pillboxes (24h administration)

Usability assessment

Activity analysis• Nurses need the list of current

meds on one page, sorted out by route

• Need to be informed immediately and very clearly of the medication changes ordered by the doctors along their rounds or visits

Context of use• No problem when the

nurses accompany the doctors along their rounds

• If not, great difficulties to update their knowledge about the patients current treatment and to remain aware of the modifications of orders

CPOE system (on-site observations)

• Difficulties with the 24h administration schedule

• The system requires the printing of the preparation plans: in case of frequent modifications of the treatment, there are a lot of printed plans, and a risk of confusion

Paper-based system (on-site observations)• The medication order list is not

usable• The only synthesized

document (MAR) is not available at the location where the pillboxes are prepared

Usability tests (1) PhysiciansPhysicians

Test the usability of information display for the information gathering task supporting the decision making

Check for recent changes in the current therapeutic treatment

Test orders entry functions

114

Usability tests (2) NursesNurses

Test the usability of the functions supporting the preparation of the 24h pillboxes (display of information, editing of preparation plans)

Test the usability of administration validation

115

Usability testsThe usability labPreparation of the testsExcerpts (video)ConclusionsRecommendations

116

Observers screens

Observers

Technician’scontrol

Facilitator control screens

Control roomDirect observation room

Main experimental&

simulation room

Experimental & simulation room

Experimental & simulation room

One-way mirrors

Usability testsThe usability labPreparation of the testsExcerpts (video)ConclusionsRecommendations

118

Create a realistic simulated environmentConfigure the system to accommodate the

characteristics of users activityCreate realistic patient’s records (from real

cases)Write precise scenarios

119

RecommendationsItems in the Interface that worked well and safelyItems in the Interface that worked poorly or

inconsistentlyIdentify those features that are capable of causing

Harm.Recommendations for specific engineering

improvements to the interface design

120

Conclusions

121

ConclusionsUsability Studies

122


Useful Mechanisms for Objective Evaluation

123


Useful Mechanisms for Objective EvaluationDesigned to Answer Specific Questions

124


Useful Mechanisms for Objective EvaluationDesigned to Answer Specific QuestionsDesigned to Discover Questions to Answer

125


Useful Mechanisms for Objective EvaluationDesigned to Answer Specific QuestionsDesigned to Discover Questions to AnswerKeeping patients safe from software related medical

error

126

“The Best Way to Predict the Future,

127


Is to Create It.”

128

Quote: Peter Drucker – Harvard University


Is to Create It.”

129

Healthcare

WCIT 2014 Peter Elkin - Human computer interaction, evaluation, usability testing, study design and methods