Qualitative Evaluation

Material Sources:

Dr. Saul Greenberg, Univ. Calgary John Kelleher, IT Sligo

2

Definition How to quickly evaluate prototypes by observing

people’s use of them How specific methods can help you discover

what a person is thinking about as they are using your system produces a description, usually in non-numeric terms may be subjective

3

Methods Methods

Introspection Extracting the conceptual model Observational Evaluation

simple observation think-aloud constructive interaction ethnography

Query via interviews and questionnaires Continuous evaluation via user feedback and field

studies

4

Designer tries the system (or prototype) out does the system “feel right”? most common evaluation method

Problems not reliable as completely subjective not valid as introspector is a non-typical user

Intuitions and introspection are often wrong

Introspection Method

5

Conceptual Model Extraction Show user the prototype / screen snapshots Have the user try to explain

what all elements are what they would do to perform a particular task

Excellent for extracting a novice’s understanding of system

Poor for examining system exploration and learning

6

Observational Evaluation

Quickly highlights difficulties

Verbal protocols valuable source of information

Can be used for rapid iterative development

Rich qualitative data

Observation can affect user activity and performance levels.

Analysis of data can be time-consuming and resource-consuming

Advantages Disadvantages

7

Direct Observation Evaluator observes and records users interacting with

design/system in lab

user asked to complete a set of pre-determined tasks a specially instrumented usability lab may be available

in field user goes through normal duties

Excellent at identifying gross interface problems Validity/reliability depends on how controlled /contrived the

situation is Three general approaches:

simple observation (See “User Testing”)

8

Thinking Aloud Protocol Asked the user to talk while doing the task

Have them tell what they are doing or trying to do as they perform the task

Have them read a set of instructions aloud before they begin the task Helps to get them going Can use the task description for this

Remind them to talk if they stop But be low-key about it

Spontaneous comments are best – don’t direct user comments with specific questions: “What do you think those prompts mean?” “Why did you do that?”

9

Pros and Cons ofThinking Aloud Pros

finds many usability problems finds why they occur (process data) small number of test users (3 to 5) usable early in development process requires little administrator expertise generates colourful quotes

Cons having to think aloud can influence user’s problem-solving

behaviour cannot provide performance data (bottom-line data) Cannot rely on user explanations (Maier, Nisbett)

10

Constructive Interaction Method

Two people work together on a task normal conversation between the two users is monitored

removes awkwardness of think-aloud

Variant: Co-discovery learning use semi-knowledgeable “coach” and naive subject together make naive subject use the interface

Results in naive subject asking questions semi-knowledgeable coach responding provides insights into thinking process of both beginner and

intermediate users

11

Recording Observations Evaluator may forget, miss, or misinterpret events Paper and pencil

primitive but cheap evaluators record events, interpretations, and extraneous observations hard to get detail (slow writing); coding schemes help…

Audio recording good for recording talk produced by thinking aloud/constructive

interaction Hard to synch user actions and annotations

Video recording can see and hear what a user is doing one camera for screen, another for subject (picture in picture) can be intrusive during initial period of use

12

Coding Scheme Example Tracking a person’s activity in the office

13

Ethnography

Ethnographic study methods recognise that the investigator will have to interact directly with the subject, but while taking sufficient care to gain reasonably complete and objective information.

An ethnographic study will attempt to observe subjects in a range of contexts, over a substantial period of time making a full record using any possible means

(photography, video and sound recording as well as note-taking) of both activities and artefacts that the subject interacts with.

14

Survey Evaluation

Addresses users’ opinions and understanding of interface.

Can be made to be diagnostic Can be applied to users and

designers Questions can be tailored to

the individual Rating scales lead to

quantitative results Can be used on a large group

of users

User experience is important Low response rates (especially

by post) Possible interviewer bias Possible response bias Analysis can be complicated

and lengthy Interviews very time-

consuming

Advantages Disadvantages

15

Interviews Excellent for pursuing specific issues

vary questions to suit the context probe more deeply on interesting issues as they arise good for exploratory studies via open-ended questioning often leads to specific constructive suggestions

Problems: accounts are subjective time consuming evaluator can easily bias the interview prone to rationalization of events/thoughts by user

user’s reconstruction may be wrong Users recall away from context poor

16

Plan a set of central questions could be based on results of user observations gets things started focuses the interview ensures a base of consistency

Try not to ask leading questions Start with individual discussions to discover different

perspectives, and continue with group discussions the larger the group, the more the universality of

comments can be ascertained also encourages discussion between users

How to Interview?

17

Questions to elicit task detail

Questions Purpose

1. What do you do? Obtains the user’s goal

2. Why do you do it? Obtains method

3. How do you do it? Obtains the subtask. Used recursively for each subtask

4. What are the preconditions for doing this?

To find out what outside influences there are

5. What are the results of doing this?

To examine the product and see what the purpose is

6. What errors occur? Error capture

7. How do you correct them? Error correction

18

Challenges Users exhibit poor knowledge of exceptions (even

regularly occurring ones) Users exhibit poor recollection of work domain in

general Ethnography a possible solution for widening

social and organisation context Users exhibit little correlation between what they

said they would prefer and what they ultimately did prefer

Interviews not touch-stone for system acceptability

19

Post-observation interview to clarify events that occurred during system use perform an observational test create a video record of it have users view the video and comment on

what they did excellent for grounding a post-test interview

avoids erroneous reconstruction users often offer concrete suggestions

Retrospective Testing

20

Questionnaires & Surveys

Preparation “expensive”, but administration cheap can reach a wide subject group (e.g. mail)

Can be repeated to establish trends does not require presence of evaluator easy to quantify and analyse results only as good as the questions asked less flexible than interview Good for subjective responses

21

Establish the purpose of the questionnaire what information is sought? how would you analyze the results? what would you do with your analysis?

Do not ask questions whose answers you will not use! (e.g. how old are you?)

Determine the audience you want to reach Determine how you will deliver and collect the questionnaire

interview administered questionnaire on-line for computer users web site with forms surface mail (include SAE for better response) determine the demographics

Questionnaire Design

22

Questionnaire Design

Keep the number of questions low Only questions with answers that you can’t get other ways Only questions that will have a direct impact on functional

requirements Avoid asking for everything Avoid questions that users perceive as self-incriminating or

inferior Ask clear questions

precision in framing questions a problem Ask questions that users can answer validly and reliably

Does the user store information in this way? Does the user remember such information? Will the user be inclined to answer your question truthfully?

23

Styles of Questions

Open-ended questions asks for unprompted opinions good for general subjective information

but difficult to analyze rigorously

I found the following aspects particularly easy to use (please list 0–3 aspects):______________________________________________________________________________________________________

24

Style of Questions Closed questions

restricts the respondent’s responses by supplying alternative answers makes questionnaires a chore for respondent to fill in can be easily analyzed but watch out for hard to interpret responses!

alternative answers should be very specific Do you use computers at work:

often sometimes rarelyvs

In your typical day, do you use computers: over 4 hrs a day between 2 and 4 hrs daily between 1and 2 hrs daily less than 1 hr a day

25

Style of Questions Likert Scalar

ask user to judge a specific statement on a numeric scale (5, 6 or 7 point scale best)

scale usually corresponds with agreement or disagreement with a statement

The new drawing package is easy to use:

1 2 3 4 5Strongly agree agree neither agree nor disagree disagree strongly disagree

26

Style of QuestionsSemantic Differentials

sliding scale between opposing pairs of adjectives (5 or 7 point scale best).

Rate the Design and Go drawing package on the following criteria:

Very Quite Neutral

Quite Very

Simple 1 2 3 4 5 Complex

Professional 1 2 3 4 5 Unprofessional

Reliable 1 2 3 4 5 Unreliable

Attractive 1 2 3 4 5 Unattractive

27

Style of Questions Ranked

respondent places an ordering on items in a list useful to indicate a user’s preferences forced choice

Rank the usefulness of these methods of issuing a command(1 most useful, 2 next most useful..., 0 if not used)

__2__ command line__1__ menu selection__3__ control key accelerator

28

Style of Questions

Combining open-ended and closed questions gets specific response, but allows room for

user’s opinion

It is easy to recover from mistakes:

Disagree…………….Agree 1 2 3 4 5

comment: undo facility is really helpful

29

Pros and Cons of Questionnaires

Collect subjective user view of system Both quantitative and qualitative data Useful supplementary information Simple Cheap

Subjective data often unreliable

30

Continuous Evaluation Usually done in later stages of development

(i.e. beta releases, delivered system) Good for monitoring problems of system in actual use Problems can be fixed in next releasea) User feedback via gripe lines

users can provide feedback to designers while using the system Email and web forms special built-in gripe facility telephone hot line (help desks) suggestion box (in-house operations)

best combined with trouble-shooting facility users always get a response (solution?) to their gripes

31

Continuous Evaluationb) Case/field studies

careful study of “system usage” at the site E.g. Quicken by Intuit

good for seeing “real life” use external observer monitors behaviour or gets feedback via

methods described above Advantages and Dis-advantages

natural environment context retained (ecological validity) longitudinal studies possible cost distractions noise

32

Summary Observing a range of users use your system for specific tasks

reveals successes and problems Qualitative observational tests are quick and easy to do Several methods reveal what is in a person’s head as they are

doing the test Particular methods include:

Conceptual model extraction Direct observation

Simple observation Think-aloud Constructive interaction

Query via interviews, retrospective testing and questionnaires Continuous evaluation via user feedback and field studies

33

Summary of Evaluation Methods