Week 5 – 2015HCI – COMP3315

Judy Kay

CHAI: Computer Human Adapted Interaction research group

Human Centred Technology Cluster for Teaching and Research

School of Information Technologies

Looking back a little….

Some nice personas from previous students

Think-alouds

The usability tool

But before we start TA….

Mental models

• Relevant things that a person believes• Designer

– How the system works– About the user’s mental model

• User(s)– Beliefs about the system

• What it does• What the interface elements mean• Relevant background knowledge

Concrete example:

“Cannot save file…. No space”

http://normfujisaki.com/wp-content/uploads/2013/09/mental-model.jpg

Note:– Beliefs …. Not facts– Mental models drive predictions, planning actions– Individual MMs differ …. Designer MM versus

“user” MM– MMs change….

Mental modelshttp://www.nngroup.com/articles/mental-models/

• Nielsen highlights: – Strongly impacts all aspects of interaction– Mismatched mental models are common,

especially with designs that try something new• What mismatches can there be?

Class activity

Make a table of 10 core elements of the mental model for CUSPOne row per concept: eg university subject, university degree, prerequisite

One column each for: designers, high school student, info3315 student

And in the cell for each indicate if they “know about” that concept

Note: Try to think of beliefs/concepts that are understood differently by these groups

Rows -- concepts Columns --

1. designers, 2. high school student, 3. info3315 student

Why do mental models matter for think-aloud?

You will be able to answer this soon….

How to address mental model problems

• Fix the system – make it match users' mental models– eg If people look for something in the wrong place, then

move it to the place where they look for it. • Fix the user - improve users' mental models so that

they more accurately reflect your system. – eg explaining things better – making labels clearer to make the UI more transparent

(even though the underlying system remains unchanged)– Help that insists that the user pays attention

Class activity

How do you discover a person’s mental model?

Some useful distinctions

• Slips: – correct user model, inadvertent incorrect action– eg car “malfunction”, tap caps lock accidentally

• Mistakes: incorrect mental model

• Use this tightened vocabulary for your think-aloud reports

The main act…

Of the lecture and the semester

Be ready to comment on what you conclude from the next slide

"Thinking aloud may be the single most valuable usability engineering method." I wrote this in my 1993 book, Usability Engineering, and I stand by this assessment today. The fact that the same method has remained

#1 for 19 years is a good indication of the longevity of usability methods.

Thinking Aloud: The #1 Usability Toolby Jakob Nielsen on January 16, 2012

So how do you do a think-aloud?

As you will by now be looking forward to doing in Assignment 2…

Demonstrating think-aloud

Demonstrate Thinking Aloud by Showing Users a Video

by Jakob Nielsen on September 1, 2014 Topics: User Testing

Services

• Userzoom• “can run unmoderated task-based studies with

geographically dispersed participants over any web-based interface (website, prototype, mock-up). Participants take the study simultaneously, in their natural context, using their own PC or device.”

• Users think• User testing

– Let’s watch their video, noting the tasks and critiquing them (just one in class)

Think aloud protocols

• Ask user to “think aloud” as they use the interface• Role of video, audio taping• Importance of a team member to take notes

• Think aloud• Helps observer interpret what is going on• Gives access to user’s “mental model”• Gives qualitative data mainly• Gives very useful but only approximate timing data

Facilitating think aloudmakes you – experimenter -- really valuable

• What are you thinking now?• What do you think that message means?

(only after the user has noticed the message and is clearly spending time on it)

• don't help user except withHow do you think you can do it?• if user appears surprised, Is that what you expected to happen?

Overview

• This is an example of an empirical method

• Absolutely relies on defining good tasks for the user to do• Abstract versus concrete

• Task design and avoiding leading the user

• Preliminaries for a study

• Conducting the study

• Making use of the data

Why do mental models matter for think-aloud?

You will be able to answer this soon….

Design cycleWhile your budget can fund another iteration do:

• User Centred Design• Define criteria for success• Define abstract and matching concrete tasks

users should be able to do

• Prototype construction (Assignment 1 is conceptual prototype….)

• Usability study– Decide just what data to collect– Test design of experiment for timing (trial it)– Recruit users– Run study

done

First things first

• Do we have the tasks right?• Abstract tasks• Concrete instances of them

– Concrete?– Relevant? – Not lead the user?– Minimalist?– Good coverage?

Class activity: - define some core tasks for testing this part of CUSP

Recruiting users

• How representative are they?– similarity to intended user population– Age– Gender– experience in area– interest/motivation– computer literacy

• What effect does user population have for conclusions?

How many users?

• Why You Only Need to Test with 5 Users• by Jakob Nielsen on March 19, 2000 • http://www.nngroup.com/articles/why-you-

only-need-to-test-with-5-users/

Insights: 0 users? 1 user? 2 users?

http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

What does this really mean?

• If first test shows catastrophic problems, should you still do 5?

• Never bother with more than 5?• 80% is good enough?

NO! NO! NO! NO!

It is all about budgets, user groups….

• If your interface is to be used by very different groups of people, you need to do the think-aloud with each group eg.– Children– Elderly– Different cultures, languages……

• Iterate!!!!! – Use same budget of effort to test each iteraction

• “In the early 1990s … fairly large-sample formative usability studies … determined that the … first five participants observed in a formative usability study should usually reveal about 85% of the problems available for discovery in that iteration, where the properties of the study (type of participants and tasks employed) place limits on what is discoverable. But over time, in the minds of many usability practitioners, the rule became simplified to “All you need to do is watch five people to find 85% of a product’s usability problems.”

Formative evaluation

• As in your Assignment 2• How many participants?• User populations?

– Within community interested in UX– Those who are not– Ages, broad range

Summative evaluation

• How many participants?– Why?

• User populations?– Within community interested in UX– Those who are not– Ages, broad range

Stages of running an evaluation

1. Preparation

2. Introduction

3. The test

4. Questionnaire/interview

5. Debriefing

6. Analysis, reflection, summarising, reporting, conclusions for action

Steps 1- 5 done for each user test, as run

Step 6 is applied mainly after several users

Preparation

• Materials for consent• Set up machine, room, environment• Check all of them• Check user instructions• Do a mental run-through

Be sure not to waste user's time because of your lack of preparation!

Call for volunteer to do the think-aloud of CUSP

Whole class acts as observers, making notes

Introduction

• Welcome user, explain purpose of test– make clear system tested not user– confidentiality– anonymity of reporting– opt out at any time– what is recorded

• Invite any other questions to here– explain procedure– if appropriate, do demo– invite questions

The test

• User works through experiment....– recording– ensure user feels supported– show pleasure at problems identified– critical to help user if stuck

• Questionnaire/interview– open and closed

What data should you collect?

• Observe – direct/indirect– take notes– video/audio/software monitor– software logs for timing

• Questionnaire: – Open

– Best things about <this interface> – If you could change one thing about <this interface>,

what would it be?

– Closed (later in the semester only)

Debriefing

• Thank user• Remind them of usefulness of results• Pause to make sure all data collected• All notes written• May ask user to confirm details

collected

Analysis and reporting each aspect, time, success……

Pitfalls

• Defining the right concrete tasks– Test all key aspects– Multiple tasks for same aspects

• Instructions to the users– Do NOT lead the user– Take particular care not to use words that

are identical to terms on the interface

Benefits of think aloud

• “show what users are doing and why they are doing it while they are doing it in order to avoid later rationalisations”

(Nielsen, Usability Engineering, Academic press 1993, p195)

• Cheap• Slows users down

– studies show users may work faster with fewer errors due to care on critical elements

Problems of think aloud

• Unnatural context and situation (do you talkto yourself?)• People filter, want to please, do not want to look foolish or inept• Hawthorn effect• Experimenter can bias results

• Directly eg via task choice

• Inadvertently eg gasp, brief frown

• Not directly quantitative• Add cognitive load to users• User's “theories” must be interpreted with care• Slows users down• Users are aware they are being observed so behave accordingly

Naturalised think-aloud

• Multi-user interaction– Two (or more) users work on task– Conversation is natural– Observer collects dialogue

Summary

• Top method for formative evaluations

• Also good for summative but needs more users

• Relatively inexpensive

• Can identify major flaws

• And may indicate causes of user problems

• May give access to user's mental model

• Alters activity => meaningfulness

• This is a major part of your next assignment