33
1 User Centered Design and Evaluation

1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

  • View
    217

  • Download
    0

Embed Size (px)

Citation preview

Page 1: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

1

User Centered Design and Evaluation

Page 2: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

2

Overview

• Why involve users at all?• What is a user-centered approach?• Evaluation strategies

• Examples from “Snap-Together Visualization” paper

Page 3: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

3

Why involve users?

Page 4: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

4

Why involve users?

• Understand the users and their problems• Visualization users are experts• We do not understand their tasks and information needs• Intuition is not good enough

• Expectation management & Ownership• Ensure users have realistic expectations • Make the users active stakeholders

Page 5: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

5

How to involve users?

• Workshops, consultation• Co-authored papers

• Visualization papers• Domain-specific papers

• User becomes member of the design team• Vis expert becomes member of user team

Page 6: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

6

More a philosophy than an approach. Based on:– Early focus on users and tasks: directly

studying information needs and tasks – Empirical measurement: users’ reactions and

performance with prototypes– Iterative design

What is a user-centered approach?

Page 7: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

7

Focus on Tasks• Users’ tasks / goals are the driving force

– Different tasks require very different visualizations

– Lists of common visualization tasks can help • (e.g., Shneiderman’s “Task by Data Type

Taxonomy”)

– Overview, details, zoom, filter, relate, history, extract

– But user-specific tasks are still the best

Page 8: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

8

Focus on Users

• Users’ characteristics and context of use need to be supported

• Users have varied needs and experience – E.g. radiologists vs. GPs vs. patients

Page 9: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

9

Understanding users’ work • Ethnography

- Observers immerse themselves in

workplace culture for days/weeks/months

• Structured observation / Contextual inquiry

- Much shorter than ethnography (a few hours)

- Often at user’s workplace

• Meetings / collaboration

Page 10: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

10

Design cycle

• Design should be iterative– Prototype, test, prototype, test, …– Test with users!

• Design may be participatory

Page 11: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

11

Key point

• Visualizations must support specific users doing specific tasks

• “Showing the data” is not enough!

Page 12: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

12

Evaluation

Page 13: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

13

Types of Evaluation Studies

• Compare design elements– E.g., coordination vs.

no coordination (North & Shneiderman)

• Compare systems– E.g., Spotfire vs. TableLens

• Usability evaluation of a system– E.g., Snap system (N & S)

• Case studies– E.g., bioinformatics, E-commerce, security

Page 14: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

14

How to evaluate with users?

• Quantitative Experiments- Controlled laboratory studies

Clear conclusions, but limited realism

• Other Methods– Observations– Contextual inquiry– Field studies

More realistic, but conclusions less precise

Page 15: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

15

How to evaluate without users?

• Heuristic evaluation

• Cognitive walkthrough?– Hard -- tasks ill-defined & may be

accomplished many ways• GOMS / User Modeling?

– Hard – tasks are ill-defined & not repetitive

Page 16: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

16

Snap-Together Vis

Customcoordinatedviews

Page 17: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

17

Questions

• Is this system usable?– Usability testing

• Is coordination important? Does it improve performance?– Experiment to compare coordination vs.

no coordination

Page 18: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

18

Usability testing vs. Experiment

Usability testing

• Aim: improve products• Few participants• Results inform design• Not perfectly replicable• Partially controlled

conditions• Results reported to

developers

Quantitative Experiment• Aim: discover knowledge• Many participants• Results validated

statistically • Replicable• Strongly controlled

conditions• Scientific paper reports

results to community

Page 19: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

19

Usability of Snap-Together Vis

• Can people use the Snap system to construct a coordinated visualization?

• Not really a research question• But necessary if we want to use the

system to answer research questions

• How would you test this?

Page 20: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

20

Summary: Usability testing

• Goals focus on how well users perform tasks with the prototype

• May compare products or prototypes• Major parts of usability testing

– Time to complete task & number & type of errors (quantitative performance data)

– Qualitative methods (questionnaires, observations, interviews)

• Informed by video

Page 21: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

21

Usability Testing conditions

• Major emphasis on- selecting representative users- developing representative tasks

• 5-12 users typically selected• Test conditions are the same for

every participant

Page 22: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

22

Controlled experiments• Strives for

– Testable hypothesis– Internal validity:

• Control of variables and conditions• Experiment is replicable• No experimenter bias

– External validity • Results are generalizable

– Confidence in results • Statistics

Page 23: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

23

Testable hypothesis

• State a testable hypothesis– this is a precise problem statement

• Example:Searching for a graphic item among 100 randomly placed similar items will take longer with a 3D perspective display than with a 2D display.

Page 24: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

24

Controlled conditions

• Purpose: Internal validity– Knowing the cause of a difference

found in an experiment– No difference between conditions

except the ideas being studied

• Trade-off between internal validity (control) and external validity (generalizable results)

Page 25: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

25

Confounding Factors

• Group 1Visualization A in a room with windows

• Group 2Visualization B in a room withoutwindows

What can you conclude if Group 2 performs the task faster?

Page 26: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

26

What is controlled

• Who gets what condition– Subjects randomly assigned to groups

• When & where each condition is given• How the condition is given

– Instructions, actions, etc.– Avoid actions that bias results (e.g.,

“Here is the system I developed. I think you’ll find it much better than the one you just tried.”)

• Order effects

Page 27: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

27

Order Effects

Example: Search for XX with Visualizations A and B with varied numbers of distractors

1.Randomization• E.g., number of distractors: 3, 15, 6,

12, 9, 6, 3, 15, 9, 12…

2.Counter-balancing• E.g., Half use Vis A 1st, half use Vis

B first

Page 28: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

28

Experimental designs

• Between-subjects design: Each participant tries one condition

– No order effects

– Participants cannot compare conditions

– Need more participants

• Within-subjects design: All participants try all conditions

– Must deal with order effects (e.g., learning or fatigue)

– Participants can compare conditions

– Fewer participants

Page 29: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

29

Statistical analysis• Apply statistical methods to data

analysis– confidence limits:

•the confidence that your conclusion is correct

•“p = 0.05” means:–a 95% probability that there is a true

difference–a 5% probability the difference

occurred by chance

Page 30: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

30

Types of statistical tests

• T-tests (compare 2 conditions)• ANOVA (compare >2 conditions)• Correlation and regression• Many others

Page 31: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

31

Snap-Together Vis Experiment

• Is coordination important? Does it improve performance?

• How would you test this?

Page 32: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

32

Critique of Snap-Together Vis Experiment

• Statistical reporting is incomplete– Should look like (F(x, y) = ___, p = ___) (I.e.

provide F and degrees of freedom)– Provide exact p values (not p < 0.05)

• Limited generalizability– Would we get the same result with non-text

data? Expert users? Other types of coordination? Complex displays?

• Unexciting hypothesis – we were fairly sure what the answer would be

Page 33: 1 User Centered Design and Evaluation. 2 Overview Why involve users at all? What is a user-centered approach? Evaluation strategies Examples from “Snap-Together

33

Take home messages

• Talk to real users!

• If you plan to do visualization research (especially evaluation) you should learn more about HCI and statistics