Evaluation Methods

1. Analytical Evaluations • without users

2. Field Studies • with users • at their site

3. Usability Testing • with users• controlled by tester

1.Evaluation Methods

Without UsersText: Analytical Evaluation

A. Cognitive Walkthroughs

B. Heuristic Evaluations

C. Predictive Models– Action Analysis– Theoretical Models

• Use HCI Experts to simulate users

Cognitive Walkthroughs• Simulate a user’s problem solving process

• Task-oriented

• Evaluate designs and prototypes for Ease of Learning

• Formalized way of imagining people's thoughts and actions when they use an interface for the first time

Goes like this:

• You have a prototype or a detailed design

• You know who the users will be.

• You select one of the tasks.

• Try to tell a believable story about each action a user has to take to do the task

Can uncover several kinds of problems

• Can question assumptions about what the users will be thinking

• Can identify controls that are obvious to the design engineer but may be hidden from the user's point of view

• Suggest difficulties with labels and prompts

• Can note inadequate feedback

Who should do a walkthrough, and when?

• You can

• Other designers, users, …

• Not supervisors

• Done as develop and extend system

What should you look for during the walkthrough?

You try to tell a story about why the user would select each action in the list of correct actions

• Will users be trying to produce whatever effect the action has? • Will users see the control (button, menu, switch, etc.) for the action? • Once users find the control, will they recognize that it produces the effect they want?

• After the action is taken, will users understand the feedback they get, so they can go

on to the next action with confidence?

Failure Stories • Users often aren't thinking what designers expect them to think

• Users' ability to locate the control -- not to identify it as the right control, but simply to notice that it exists!

• Will they realize that this is the control they're after? • Even the simplest actions require some kind of feedback, just to show that the system

"noticed" the action

B. Heuristic Evaluations

• Use common-sense and usability guidelines and standards

• Analyst needs HCI Knowledge

• Good for critiquing others

• In past, not so good for improving interfaces

Jacob Nielsen and Rolf Molich • Breakthrough in the use of heuristics?

• Early 90’s

• Developed and tested a procedure for using them to evaluate a design.

• Not necessarily task-oriented

Nielsen and Molich's Nine Heuristics

• Simple and natural dialog - Simple means no irrelevant or rarely used information. Natural means an order that matches the task.

• Speak the user's language - Use words and concepts from the user's world. Don't use system-specific engineering terms.

• Minimize user memory load - Don't make the user remember things from one action to the next. Leave information on the screen until it's not needed.

• Be consistent - Users should be able to learn an action sequence in one part of the system and apply it again to get similar results in other places.

• Provide feedback - Let users know what effect their actions have on the system.

• Provide clearly marked exits - If users get into part of the system that doesn't interest them, they should always be able to get out quickly without damaging anything.

• Provide shortcuts - Shortcuts can help experienced users avoid lengthy dialogs and informational messages that they don't need.

• Good error messages - Good error messages let the user know what the problem is and how to correct it.

• Prevent errors - Whenever you write an error message you should also ask, can this error be avoided?

Nielsen and Molich's PROCEDURE

• Based on the observation that no single evaluator will find every problem with an interface

• Different evaluators will often find different problems.

• PROCEDURE:– Have several evaluators use the nine heuristics to identify problems with the interface

– Each evaluator should do the analysis alone

– Combine the problems identified by the individual evaluators into a single list

• by a single usability expert • Or by a group

3. Action AnalysisText: “Predictive Models”

• GOMS

• Keystroke Model

• Fitts’ Law

GOMS

• Goals, Operators, Methods, Selection Rules

– Goals: what a user wants to achieve

– Operators: Physical actions user performs to achieve goals

– Methods: Exact actions performed

– Selection Rules: Used to choose Methods

• Early 1980’s by Stuart Card, Tom Moran, and Alan Newell

• Attempt to model users’ cognitive processes

EXAMPLE

• Goal: Delete a word in a sentence• Method using Menus

– Step 1 Recall must highlight word– Step 2 Recall this is a ‘cut’– Step 3 Recall ‘cut’ is in edit menu– Select & execute ‘cut’

• Method using Delete key– Recall where to position cursor– Recall which key is ‘delete’ key– Press ‘delete’ key to delete each letter

• Operators– Click mouse– Drag cursor over text– Select menu– Move cursor tto command– Etc.

• Selection Rules– Delete text using mouse and menu– Delete text using delete key

Keystroke- level analysis • Also developed by Card et al (1983)

• Provides actual numerical prediction of user performance

• (Show Slide 17, chapter 15)

Fitts’ Law

• Predicts the time it takes to reach a target using a pointing device

• Function of the distance from the target object & the object’s size. • So further away & the smaller the object, the longer the time to locate it and point to

it.

T = k log2 (D/S +1.0)

• k = approx 200 msec/bit

2. Field Studies

• In natural setting

• Used early in design or to evaluate

• Evaluators develop relationship with user

• Ethnographic Study

• For evaluation, same issues as for early design

Usability Testingwith users

• Product being tested, not the user• Two components:

– User Test– User satisfaction questionnaire

• Task-oriented• Measures:

– Time– Number

Slides, Chap. 14: Slide 3, 4, 6 -10, 13

Creating a Usability Test

1. Identifying the Test Goals

• Test goals should be specific, not general as in "testing if it works".

EXAMPLES • Can the user use FTP to fetch a particular document?

• Can the user make the robot climb stairs?

• Discover how users navigate a site

• Can the child send an email to a friend?

Creating a Usability Test

2. Choose a Test Method

• Formative Evaluation which is done early in a project's design and used to develop the design

• Summative Evaluation which is done when a project is completed.

• Comparative Evaluation which compares two ways of presenting the same information

• Protocol Analysis which asks users to speak aloud their thoughts either while performing a task (concurrent verbalization) or after (retrospective verbalization).

Creating a Usability Test

3. Identify the characteristics of the test subjects.

• Demographics

• Experience

• Related to Application

Creating a Usability Test

4. Develop realistic tasks

– Should relate to goals– Specific

Creating a Usability Test

5. Order the tasks

– By Importance?– By other criteria?

Creating a Usability Test

6. Determine Performance Measures

• Quantitative– counting the number of test subjects who finish a particular task– how long each task takes– how many errors each makes– how many questions asked while performing the task– etc.

• Qualitative – comments from the test subjects while performing tasks– observations of the test team– etc.

• Combination of quantitative and qualitative

Creating a Usability Test

7. Create the test materials

– Be sure to include instructions

Administering the Test

– Thank your participants– Do not embarrass participants– Informed Consent Form (see pg. 637)– Assure them of confidentiality if appropriate– Might include incentives (food, $, …)– See pages 653-5 for some script ideas– Analyze your data carefully