CSCD 487/587Human Computer Interface

Winter 2013

Lecture 19Evaluation

Overview

Evaluation is integral to the design process Evaluators collect information about users' or

potential users' experiences when interacting with your interface

They do this in order to improve its design Evaluation focuses on both usability of system

– How easy it is to learn and to use and On users' experience when interacting with

system– How satisfying, enjoyable, or motivating

Overview Many different evaluation methods Which to use depends on the goals of the evaluation Evaluations can occur in a range of places such as

laboratories, people's homes, outdoors, and work settings

Evaluations usually involve observing participants and measuring their performance

– In usability testing, experiments, or field studies

There are other methods, however, that do not involve participants, such as modeling user behavior

Tend to be approximations of what users might do when interacting with an interface … not measuring users

Why You Need to Evaluate

Designers should not presume that everyone is like them or that following set guidelines would guarantee them good usability

Evaluation is needed to check that users can use product and like it

Why You Need to Evaluate

Bruce Tognazzini – Usability Consultant “Iterative design, with its repeating cycle of design and testing, is the only validated methodology in existence that will consistently produce successful results If you don’t have user-testing as a part of your design process you are going to throw buckets of money down the drain”

Why You Need to Evaluate

Toganazzini’s 5 resons to evaluate1. Problems are fixed before product is shipped, not after 2. The team can concentrate on real problems, not imaginary ones 3. Engineers code is sharply reduced 4. Time to market is sharply reduced 5. Upon 1st release, your sales department has a rock solid design it can sell without having to pepper their pitches with how well next release will work

Why You Need to Evaluate

Evaluation as part of the Design Process

design

implementationevaluation

What to Evaluate

It’s a Continuous Process Users and their tasks Observe, measure, and analyze

their performance with the system

Basically anything for which we need feedback

Important for designer to check and make sure that they understand user requirements

Where Do We Evaluate

Laboratories – Formal experiments On-line – Gather feedback from on-line

user statistics Homes – Users can provide feedback

in a natural setting

When to Evaluate

Depends on what we are evaluating Requirements

Gather them, evaluate, Did we capture them? Iterate Evaluate again May do this several times

Design Evaluate as develop interface and product Known as Formative Evaluation

Product Completion Evaluate to assess user satisfaction Known as Sumative Evaluation

When to Evaluate

New product Use mockups, sketches, and other low fidelity

prototyping techniques are used to represent design ideas

Upgrading existing products Compare user performance and attitudes and

contrast new products with the previous versions

Evaluation is a key ingredient for a successful design

Evaluation Types

1. Controlled Settings Labs or Living Labs User's activities are controlled Do usability testing and experiments

2. Natural Settings On-line communities and public

places No control over user activities Field studies

Evaluation Types

3. Settings Not Involving Users Experts and consultants research Predict and model aspects of

interfaces You Choose a method based on ….

Deciding how much control is needed to find out how an interface is used

Controlled Settings

Evaluators can control what users do Also reduce outside influences and distractions Usability Testing

Purpose - Evaluate User Interfaces Experiments, observations,

interviews and questionnaires In a controlled setting

Goal Determine whether an interface is

usable by intended population

Controlled Settings

Usability Testing continued Has been done for years as part of Human

Computer Interaction Findings of test summarized in a usability

specification Changes in design can be specified

according to usability spec

Example Paper prototype session viewed in class one

example

Natural Settings Evaluate People in Natural Settings

Purpose Identify opportunities for new technology Establish requirements for a new design New context for existing product

Field Studies Can be in person or virtual On-line communities Chat rooms

Goals Examine social processes, collaborations

and cooperation between users

Natural Settings

Field Studies While techniques, such as focus groups, usability tests, and

surveys, can lead to valuable insights, most powerful tool in the toolbox is the 'field study'

Field studies get teams immersed in environment of their users and allow them to observe critical details for which there is no other way of discovering

Examples: Watch users in shopping malls, watched system administrators, watched paper flow through manufacturing facility

What do you think is biggest downside ? Biggest downside to field studies is the cost to the

organization, Scheduling the visits, taking team members out of the office for several days, and finishing the analysis can have a huge impact on a project's resources

http://www.uie.com/articles/field_studies/

Measuring man or dog?

Field StudiesThe Power of Field Studies Even a short field study, such as two or three

half-day visits, can yield tremendous value. From these we can learn:

Terminology and processes– What do users do and how do they

talk about it? – While users can describe a process or

share terms in an interview format, watching them work points out subtleties that they are unaware of

Field StudiesThe Power of Field Studies Even a short field study, such as two or three

half-day visits, can yield tremendous value. From these we can learn:

Context – What are the external forces that will

impact the design? – Do the user's requirements change

when they are rushed or up against a deadline?

– People have trouble describing the context of their work, however it's easy for outsiders to observe

Field StudiesThe Power of Field Studies Even a short field study, such as two or three

half-day visits, can yield tremendous value. From these we can learn:

Similarities and differences– Visiting multiple sites can allow a

team to collect a rich amount of information about commonalities that appear across environments, along with variations that will impact design decisions

– Such as providing switches, options, and optional features

Research Not with Users

Researcher has to model how interface is likely to be used

Predict user behavior Identify usability problems

Example Heuristic evaluation of an interface

Applies knowledge of user behavior

Context system will be used Steps through a scenario and

answers set of questions

Heuristic Evaluation Originally used in evaluation of screen-

based applications Heuristic evaluation is discount usability

engineering method for quick, cheap, and easy evaluation of a user interface design

Heuristic evaluation is most popular usability inspection method

Heuristic evaluation is done as systematic inspection of a user interface design for usability

Heuristic EvaluationJakob Nielsen

Most usability engineering methods will contribute substantially to the usability of an interface …

…if they are actually used.

Heuristic Evaluation

Goal of heuristic evaluation is to find usability problems in design so that they can be integrated into iterative design process

Heuristic evaluation involves having a small set of evaluators examine interface and judge its compliance with recognized usability principles (the "heuristics")

Dates back to the 1990's

Heuristic Evaluation

How does it work?Evaluators use a checklist of basic usability heuristics Evaluators go through an interface twice

• 1st Pass get a feel for the flow and general scope• 2nd Pass refer to checklist of usability heuristics and

focus on individual elements

The findings of evaluators are combined and assessed

http://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/

Heuristic EvaluationUsability Heuristics (original, unrevised list)

Simple and natural dialogueSpeak the users’ languageMinimize the users’ memory loadConsistencyFeedbackClearly marked exitsShortcutsPrecise and constructive error messagesPrevent errorsHelp and documentation

COMMENTS?

Heuristic Evaluation

Debriefing sessionConducted in brain-storming modeEvaluators rate the severity of all problems identifiedUse a 0 – 4, absolute scale

• 0 I don’t agree that this is a prob at all• 1 Cosmetic prob only• 2 Minor prob – low priority• 3 Major prob – high priority• 4 Usability catastrophe – imperative to

fix

Heuristic Evaluation

How does H.E. Differ from User Testing? Evaluators have checklists Evaluators are not target users Evaluators decide on their own how they want to proceed Observer can answer evaluators’ questions about the domain or give hints for using the interface Evaluators say what they didn’t like and why; observer doesn’t interpret evaluators’ actions

Heuristic Evaluation

What are the shortcomings of H.E.?Identifies usability problems without indicating how they are to be fixed

• “Ideas for appropriate redesigns have to appear magically in the heads of designers on the basis of their sheer creative powers.”

Cannot expect it to address all usability issues when evaluators are not domain experts / actual users

Have a short video of Heuristic Evaluation …http://www.youtube.com/watch?v=8pDJwjacZnw

On-line Analytics

Web Analytics Measurement, collection and

reporting of user behavior Understand and optimize web usage Tools help analyze log files of

interaction data and web traffic http://www.kaushik.net/avinash/qualitative-

web-analytics-expert-heuristic-evaluations/

Web Analytics

What do they tell you? Data such as

– Number of visitors to a website,– Where they are from,– Which pages they view, and– Which links they click

Measured, collected, analyzed and reported Web analytics are relevant to usability practitioners in that

they can provide insight into the large-scale behavior of website users to understand and improve (optimize) the website

What wouldn't they tell you?

http://www.usabilitybok.org/web-analytics

Web Analytics

What wouldn't they tell you? Web analytics cannot provide answers

– About user motivations or underlying needs and goals

– Web analytics may indicate that users are abandoning a checkout process at a particular point, but they cannot be used to explain why this is happening

– Usability testing of issues found through web analytics brings deeper understanding needed to fix these usability problems

– Additionally, in-person observations of users can lead to insight that informs what metrics are worthwhile to collect, and how to interpret them.

Summary

User evaluation is a critical part of user centered design

Uses many of the techniques we already discussed

Plus, can use Heuristics as a quick and cheap way to explore user interface problems

Web statistics through website analytics can provide valuable data but is no substitute for more in-depth user testing

34

End

Two more Weeks of Classes !!!