Upload
david-travis
View
103
Download
0
Embed Size (px)
DESCRIPTION
Slides from my Usability Testing Bootcamp workshop at UX Lisbon, May 2011.
Citation preview
How to Plan and Moderate a Usability Test
Dr David Travis, Userfocus ltd
#uxlx #ut
Who am I?
?
I currently run a consultancy firm called Userfocus. We work with clients like Yahoo!, the World Health Organisation and the UK Tax Office. Yes, even they are using these techniques to make sure you pay what you owe.
You can still buy my book on amazon!
What you’ll learn
Design and plan usability tests.
Recruit the right test participants.
Apply usability testing checklists to make sure your test runs without a hitch.
Develop test tasks that are relevant to your customers and your business.
Organise and prepare usability testing sessions.
Moderate 'thinking aloud' usability tests in an unbiased and balanced manner.
Code, collect and interpret behavioural data.
Interview your partner
Where do you work and what do you do?
What one thing do you already know about usability testing?
What do you want to be able to do differently as a result of the training?
How to Plan and Moderate a Usability Test
What so special about a usability test?
Johansson, P., Halls, L., Sikström, S & Olsson, A. (2005) Failure to Detect Mismatches Between Intention and Outcome in a Simple Decision Task. Science, 310: 116-119.
When people are asked to justify their choices, they do so readily — even when they never made the choice. We need to be aware of this confabulation when running any kind of user research.
Lichfield Cherokee 3 Man Tent Red
Lichfield Crow 2 Man Tent Denim/Navy
Lichfield Dakota 3 Man Tent Denim/SkyBlue
Confidence Holiday 6 Man 2 Room Tent
Lichfield Dakota 3 Man Tent Denim/SkyBlue
Confidence Holiday 6 Man 2 Room Tent
Lichfield Crow 2 Man Tent Denim/Navy
Lichfield Cherokee 3 Man Tent Red
Lichfield Dakota 3 Man Tent Denim/SkyBlue
Confidence Holiday 6 Man 2 Room Tent
Lichfield Crow 2 Man Tent Denim/Navy
Lichfield Cherokee 3 Man Tent Red
Felfernig, A. et al. (2007) Persuasive Recommendation: Serial Position Effects in Knowledge-Based Recommender Systems. Lecture Notes in Computer Science, 4744: 283-294.
The first tent presented was chosen 2.5 times more often than any other. Yet people ‘explained’ their choices using logic and rationality.
“The accuracy of subjective reports is so poor as to suggest that any introspective access that may exist is not sufficient to produce generally correct or reliable reports.”
- Nisbett, R.E., and Wilson, T.D. “Telling more than we can know: Verbal reports on mental processes.” Psychological Review, 84 (1977), pp. 231-259.
What is a usability test?
Behaviour is observed — what users do (rather than what users say they do).
Participants perform real tasks — tasks that are meaningful to them and to the business.
Involves real users — not just someone at the next desk.
Observations and data are recorded and the data are analysed.
Quick Activity: Usability Blankety Blank
If we’re going to test for usability, we better define it first.
Listen to the definitions of usability and call out the missing words.
“Usability means that the people who use the product can do so quickly and easily to accomplish their own tasks. This definition rests on four points: (1) Usability means focusing on USERS; (2) people use products to be productive; (3) users are busy people trying to accomplish tasks; and (4) users decide when a product is easy to use.”
- Janice (Ginny) Redish and Joseph Dumas (1999). A Practical Guide to Usability Testing.
“Usability means that the people who use the product can do so quickly and easily to accomplish their own tasks. This definition rests on four points: (1) Usability means focusing on USERS; (2) people use products to be productive; (3) users are busy people trying to accomplish tasks; and (4) users decide when a product is easy to use.”
- Janice (Ginny) Redish and Joseph Dumas (1999). A Practical Guide to Usability Testing.
“Usability is a term used to denote the ease with which people can employ a particular tool or other human-made object in order to achieve a particular GOAL.”
- Wikipedia.
“Usability is a term used to denote the ease with which people can employ a particular tool or other human-made object in order to achieve a particular GOAL.”
- Wikipedia.
“The capability in human functional terms to be used easily and EFFECTIVELY by the specified range of users, given specified training and user support, to fulfil the specified range of tasks, within the specified range of environmental scenarios.”
- Brian Shackel (1991). ‘Usability—context, framework, definition, design and evaluation’. In B. Shackel and S. J. Richardson (eds) Human factors for informatics usability.
“The capability in human functional terms to be used easily and EFFECTIVELY by the specified range of users, given specified training and user support, to fulfil the specified range of tasks, within the specified range of environmental scenarios.”
- Brian Shackel (1991). ‘Usability—context, framework, definition, design and evaluation’. In B. Shackel and S. J. Richardson (eds) Human factors for informatics usability.
“At first, using words that all started with “E” was a word game, but I was also looking for a way to make the dimensions of usability easy to remember and the 5Es were born. I decided on: Effective; EFFICIENT; Engaging; Error Tolerant; Easy to Learn.”
- Whitney Quesenbery (2004) ‘Balancing the 5Es: Usability’. Cutter IT Journal, Feb, 2004.
“At first, using words that all started with “E” was a word game, but I was also looking for a way to make the dimensions of usability easy to remember and the 5Es were born. I decided on: Effective; EFFICIENT; Engaging; Error Tolerant; Easy to Learn.”
- Whitney Quesenbery (2004) ‘Balancing the 5Es: Usability’. Cutter IT Journal, Feb, 2004.
“It is important to realize that usability is not a single, one-dimensional property of a user interface. Usability has multiple components and is traditionally associated with these five usability attributes: learnability, efficiency, memorability, errors, SATISFACTION.”
- Jakob Nielsen (1993), Usability Engineering.
“It is important to realize that usability is not a single, one-dimensional property of a user interface. Usability has multiple components and is traditionally associated with these five usability attributes: learnability, efficiency, memorability, errors, SATISFACTION.”
- Jakob Nielsen (1993), Usability Engineering.
“Usability is a function of the ease of use (including learnability when relevant) and the acceptability of the product and will determine the actual usage by a particular user for a particular task in a particular CONTEXT.”
- Nigel Bevan, Jurek Kirakowski and Jonathan Maissel (1991) ‘What is Usability?’ Proceedings of the 4th International Conference on HCI, Stuttgart, September 1991.
“Usability is a function of the ease of use (including learnability when relevant) and the acceptability of the product and will determine the actual usage by a particular user for a particular task in a particular CONTEXT.”
- Nigel Bevan, Jurek Kirakowski and Jonathan Maissel (1991) ‘What is Usability?’ Proceedings of the 4th International Conference on HCI, Stuttgart, September 1991.
usersgoals
effectiveness efficiencysatisfactioncontext
So based on these quotations, it seems that usability has something to do with:
• USERS• GOALS• EFFECTIVENESS• EFFICIENCY• SATISFACTION
and
• CONTEXT
tHere’s a definition we can use that brings all of these together…
ISO 9241 definition of usability (from ISO 9241-11 and ISO 9241-210)
“Extent to which a product, system or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” - Effectiveness
The accuracy and completeness with which users achieve specified goals.
- Efficiency
The accuracy and completeness of goals achieved in relation to resources.
- Satisfaction
Freedom from discomfort, and positive attitudes towards the use of the product, system or service.
ISO 9241 definition of usability (from ISO 9241-11 and ISO 9241-210)
“Extent to which a product, system or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” - Effectiveness
The accuracy and completeness with which users achieve specified goals.
- Efficiency
The accuracy and completeness of goals achieved in relation to resources.
- Satisfaction
Freedom from discomfort, and positive attitudes towards the use of the product, system or service.
THIS M
EANS THAT USABIL
ITY IS
NOT JUST
ABOUT MAKIN
G THINGS EASY TO U
SE
1234
With usability, we ask if people can actually use a product.
You don’t need many people to identify a problem that needs to be fixed.
For example, which knob would you use to light the gas under the back right hob?
1234
answer: 4
Usability testing then gives you these kinds of insights to improve designs.
“Testing one user early in the project is better than testing 50 near the end.”
- Steve Krug
So what defines a usability test?
Usability tests don’t require:- A usability lab
- Observers
- A moderator
- Screen recording
- Video recording
All these are ‘nice to have’.
Focusing on these alone puts you at risk of running a ‘cargo cult’ usability test.
How to Plan and Moderate a Usability Test
A live test
Deconstruct a usability test
Try to deconstruct the usability test.
How would you describe what happened when?
Are there different phases?
Q. Would you like to take part in a usability evaluation?
As a thank-you, you’ll get a free copy of E-Commerce Usability!
YES: Remain standing
NO: Sit down
Q. In the last 6 months, have you bought any electronic goods online, such as a computer, a camera or a mobile phone?
YES: Remain standing
NO: Sit down
Q. Do you use a colour printer at home?
YES: Remain standing
NO: Sit down
Q. Is your printer a laser printer or an ink-jet printer?
LASER: Sit down
INK JET: Remain standing
DON’T KNOW: Remain standing
Q. Have you done any of these activities in the last 2 weeks? Give yourself 1 point for each ‘YES’ answer.- Edited the html code on a web page
- Sold something on eBay
- Created or commented on a blog
- Installed software on your computer
- Signed into your online bank account
- Played an online game
- Downloaded and listened to podcasts
- Shared photographs or videos on web sites like flickr, Twitpic or YouTube
- Added a page to a social bookmarking site like del.icio.us, reddit or Digg
- Subscribed to a syndicated ("RSS") web feed
7 or more: Sit down
6 or fewer: Remain standing
Oh no! Your home printer is broken.
You can spend up to £100 on a new one.
The printer must be wi-fi enabled and it must work with a Mac.
Find the best printer within your budget.
Participant Recruitment ? ? ?
???
Deconstruct the usability test
Identify the test goals
Screen and recruit participants
Identify the test tasks
Greet the participant
Get the video consent form / NDA signed
Demonstrate and practice thinking aloud
Probe with open questions
How to Plan and Moderate a Usability Test
Planning a usability test
The Five W’s (and one H)
Why are you running the test?
Where will it take place?
When will it take place?
Who will be the test participants?
What system (and what functionality) will you be testing?
How will you collect and analyse the data?
Usability Test Plan Toolkit
The test plan toolkit contains guidance and checklists for running a usability test.
You get an editable version in Microsoft Word when you sign up for refresher training (free!)
!"#$%&'()*+,-*.&/00'01*/*2(/3'&')4*)#()5*6##%(*3#+,-#*)"#*)#()*
! "#$%%!&%'&!()*%+&,-%'!! "#$%%!(.!&/%!0%1&2$%!'%&!&/1&!3,44!)%!&%'&%5!! "#$%%!61$&,+,61.&!6$(0,4%'!! 7$%1&%!61$&,+,61.&!'+$%%.%$!! 8.#1#%!$%+$2,&9%.&!0,$9!! :((;!(2&!1.<!%=2,69%.&!<(2>44!.%%5!?%@#@!$((9A!+(962&%$A!-,5%(!+19%$1B!
!
7*6##%(*3#+,-#*)"#*)#()*
!C$,&%!&1';!'+%.1$,('!!C$,&%!&%'&!641.!1.5!'%.5!&(!'&1;%/(45%$'!0($!$%-,%3!! D.-,&%!()'%$-%$'!
!
8*6##%*3#+,-#*)"#*)#()*
! 7/%+;!$%+$2,&9%.&!,'!(.!'+/%524%!! E,.14,'%!1.5!,''2%!&%'&!641.!!! F%.5!(2&!166(,.&9%.&!&,9%A!916!1.5!-,5%(!+(.'%.&!0($9!&(!61$&,+,61.&!! 7/%+;!&/%!6$(&(&<6%!'266($&'!&/%!'+%.1$,('!G!9(5,0<!'+%.1$,('!,0!.%+%''1$<!! D0!1.<&/,.#!<(2>$%!2',.#!.%%5'!)1&&%$,%'A!#%&!'61$%'!
!
8*9/4*3#+,-#*)"#*)#()*
! H$,.&!'+%.1$,('!(.!,.5%I!+1$5'!0($!%1',%$!/1.54,.#!! H$,.&!&%'&!'+$,6&!! H$,.&!61$&,+,61.&!=2%'&,(..1,$%'!?J!6%$!61$&,+,61.&B!! H$,.&!-,5%(!+(.'%.&!0($9'!?61$&,+,61.&'!91<!0($#%&!&(!)$,.#!&/%!(.%!<(2!'%.&B!! H$,.&!KD.'&$2+&,(.'!0($!()'%$-%$'L!! H$,.&!&/%!'+%.1$,('!(.!1!',.#4%!'/%%&!! M2.!6,4(&!&%'&!3,&/!1!0$,%.54<!4(+14!G!9(5,0<!&%'&!'+$,6&!1.5!'+%.1$,('!1'!.%+%''1$<!!N)&1,.!+1'/!0($!61$&,+,61.&!,.+%.&,-%'!! H$,.&!$%+%,6&'!0($!61$&,+,61.&!,.+%.&,-%'!! 7$%1&%!1!&,9%&1)4%!'/(3,.#!61$&,+,61.&!'+/%524%!G!,.+425%!%1+/!61$&,+,61.&>'!6/(.%!.29)%$!! O%4%6/(.%!&(9($$(3>'!61$&,+,61.&'!1.5!+(.0,$9!1&&%.51.+%!! 891,4!()'%$-%$'!1.5!+(.0,$9!1&&%.51.+%P!1&&1+/!&/%!&%'&!641.!! Q,'1)4%!1.<!'(0&31$%!&/1&!9,#/&!,.&%$$26&!&/%!&%'&!?4,;%!+14%.51$!$%9,.5%$'A!%91,4!14%$&'A!,.'&1.&!9%''1#,.#B!! :((;91$;!&/%!',&%!&(!91;%!,&!%1',%$!0($!<(2!&(!0,.5!52$,.#!&/%!&%'&!! 8.'2$%!<(2!/1-%!1!+(.&1+&!.29)%$!0($!1.<!&%+/.,+14!'266($&!<(2!91<!.%%5!
:"#*;,-0'01*,+*)"#*)#()*
! H2&!KD.'&$2+&,(.'!0($!()'%$-%$'LA!&/%!4,'&!(0!'+%.1$,('!1.5!1!+(6<!(0!&/%!&%'&!641.!,.!&/%!()'%$-1&,(.!$((9!! 7/%+;!&/%!6$(&(&<6%!3($;'!! 7/%+;!&/%!$%+($5,.#!%=2,69%.&!R!'+$%%.!$%+($5,.#!'(0&31$%!3($;'!?125,(!1.5!-,5%(B!! 7/%+;!&/%!-,5%(!1.5!125,(!0%%5'!,.!&/%!()'%$-1&,(.!$((9!
!
<#+,-#*#/$"*./-)'$'./0)*/--'=#(*
! M%S'%&!&/%!3%)!',&%!&(!,&'!5%0124&!'&1&%T!+4%1$!&/%!)$(3'%$!/,'&($<A!5%4%&%!+((;,%'A!4(15!1!)41.;!)$(3'%$!61#%!! O2$.!(00!&/%!'+$%%.!?)2&!;%%6!&/%!+(962&%$!$2..,.#B!! F&1$&!&/%!$%+($5,.#!%=2,69%.&!! M1.5(9,'%!&/%!'+%.1$,(!+1$5'!
!
>"#0*)"#*./-)'$'./0)*/--'=#(*
!U1.1#%!%I6%+&1&,(.'!G!%I641,.!&/%!'%&&,.#!! "';!&/%!61$&,+,61.&!&(!+(964%&%!1!-,5%(!+(.'%.&!0($9!
!
>"#0*)"#*./-)'$'./0)*'(*+'0'("#9*
! "';!&/%!61$&,+,61.&!&(!+(964%&%!1!6('&S&%'&!=2%'&,(..1,$%!! 7/%+;!,0!()'%$-%$'!/1-%!1.<!0(44(3S26!=2%'&,(.'!! H1<!&/%!,.+%.&,-%!1.5!#%&!1!',#.%5!$%+%,6&!! F&(6!&/%!$%+($5,.#!G!.19%!&/%!$%+($5,.#!166$(6$,1&%4<!! H$%61$%!0($!&/%!.%I&!61$&,+,61.&!
!
?)*)"#*#09*,+*#/$"*9/4@(*)#()'01*
! Q,'+2''!&/%!0,.5,.#'!3,&/!()'%$-%$'!G!1#$%%!(.!&/%!&(6!,''2%'!! :1+;26!1.5!'%+2$%!144!$%+($5,.#'!
!
8*9/4*/+)#-*)"#*)#()*
! F%.5!1!&/1.;S<(2!.(&%!&(!%1+/!61$&,+,61.&!!
>')"'0*8*6##%*,+*)"#*)#()*
! H$%'%.&!0,.5,.#'!1.5!1#$%%!.%+%''1$<!+/1.#%'!!
Contents of the Test Plan
Purpose of usability test
Product under test
Participants & responsibilities
Evaluation procedure
Data collection
Appendices- Screener, consent form, checklists, etc.
Recruiting participants
Step 1: Use personas- If you don’t have any, create assumption
personas based on low / high technological
expertise and low / high domain knowledge
Step 2: Go where your users congregate- Friends & family
- Customer lists
- Professional societies
- LinkedIn, Twitter
- Recruitment agencies
Step 3: Filter with an online screener
Step 4: Follow up by phone
A taxonomy of tests
Portable test lab
The moderator takes the test to the participant and sits at the participant’s desk or cubicle. If observers are present, they squeeze in and sit where they can.
A taxonomy of tests
Single room setup
The moderator and participant sit at a desk in a small office. Observers sit at the back of the room and either crane their necks or watch the test on a separate computer running screen sharing software. Observers are close enough to hear what the moderator and participant say to each other.
A taxonomy of tests
Classic testing lab setup
The moderator and participant sit in one room. Observers can see the moderator and participant through a one-way mirror and hear their conversation through the lab’s built-in audio system. Observers get a close-up view of the participant’s screen via screen sharing software.
A taxonomy of tests
Classic benchmark test
The participant sits in one room and works alone. The moderator and observers can see the participant through a one-way mirror and get a close-up view of the participant’s screen via screen sharing software.
A taxonomy of tests
Multi-room setup
The moderator and participant sit in one room. Observers sit in another room elsewhere in the building. Observers watch the participant’s screen via screen sharing software projected on the wall and hear the participant through the computer’s speakers.
A taxonomy of tests
Remote moderated test
The participant carries out the test from his or her home or office desk. The moderator uses screen sharing software to see the participant’s screen and presents the tasks to the participant over the phone. Observers use conferencing software to view the test over the Internet.
A taxonomy of tests
Remote unmoderated test
The participant carries out the test from his or her home or office desk. The computer presents the tasks to the participant and the participant signals when he or she has completed the task. Nobody observes or moderates the test.
How to Plan and Moderate a Usability Test
Writing Test Tasks
Red routes
Roads in London with red lines on them are known as red routes: these are the key road arteries in London. Transport for London does everything it can to keep these routes clear.
Your product or web site has “red routes” too. They are the critical “user journeys” with your product.
Red Route Examples (from Gerry McGovern (2010), The Stranger’s Long Neck).
What are the Red Routes for a university web site?
- Can I study [subject]?
- What does it cost?
- What qualification will I earn?
- How will this help me in my job or career?
- Where do I go to take this course?
- Can I study part time, while working?
Red Route Examples (from Gerry McGovern (2010), The Stranger’s Long Neck).
What are the Red Routes for a government web site that aims to help small business?- What grants are available to my
company?
- How do I apply for funding?
- What funding is available to a start-up?
- How do I apply for funding?
- How can I reduce my tax liability?
Red Route Examples (from Gerry McGovern (2010), The Stranger’s Long Neck).
What are the Red Routes for a health web site?
- Based on my symptoms, what disease/condition do I have?
- What are the basic fact about [condition / disease]?
- How do I book an appointment online?
- How can I get advice from a health professional?
- Based on my symptoms, what should I do next?
Brainstorm red routes
You have 300 seconds.
Think about a product, system or service that you are working on at the moment.
Brainstorm a handful of red routes — the ‘headline’ tasks.
The context of use
People approach tasks differently based on the context of use.- Imagine you need to book a flight and hotel
in Paris for a 2-day trip with your partner.
- Imagine you need to book a flight and hotel in Paris for a 2-day business trip.
The red routes — finding a hotel, booking a flight — are the same but the context of use alters the way you would carry out these tasks.
You build context into your test tasks with scenarios.
“Search for jobs by geographic
area”
Red route
“You’re fed up with the long journey to
work. Find all the suitable job
openings within a reasonable
commute of your home.”
Scenario
Expand your red routes into scenarios
What are the implications of this scenario for the design?
We can offer a better experience if we show “commuting time” as an option in the search results. And note that this may not be the same as distance, it depends on transport connections.
You wouldn’t get this from the original red route which doesn’t have the contextual information.
“Buy a PSP”
Red route
“Darren wants to buy a Sony PlayStation
Portable and a suitable game for his 12
year-old son for his birthday this Monday.
Because of the short notice, he won’t buy
the product unless he can collect it from a
store before his son’s birthday.”
Scenario
Expand your red routes into scenarios
What are the implications of this scenario?
• We need to offer cross-channel support (buy online, pick up in store)
• We need to be able to show shop stock levels• We need to show store locations and directions
relative to the customer’s location• We need to offer a reservation system so the PSP can
be put aside• We need to cross sell suitable games by age group, or
at least classify games according to age group and gender
Six-step scenario checklist
Is it really a red route?Is it specific and
measurable?
Does it describe a complete activity
(integrated, not simple tasks)?
Does it describe what the user wants to do (not how the user will
do it)?
Does it include enough information to complete the task yet avoid hidden clues?
Is the task “portable” to competitor
products?
Create a scenario
You have 300 seconds.
Take one of the red routes you created earlier.
Use it to create a scenario — build in the ‘context’ information.
How to Plan and Moderate a Usability Test
How to moderate a usability test
Hats moderators wear in a usability test (based on Carolyn Snyder’s Paper Prototyping)
Flight Attendant- Safeguard the physical, psychological and
emotional wellbeing of the participant.
• Greet participants, hang up their coats, offer a drink, engage in small-talk
• Monitor participants for signs of stress
• Provide reassurance and assistance as needed
• Obtain informed consent
How to be a good moderator
Before starting — let participants know they are going to be listened to and that their opinions are valued.
Observe and read the participant’s non-verbal behaviour — movement, facial expressions, etc.
Reinforce the participant by nodding or paraphrasing
Take some notes, but don’t write all the time
Reflect back what you think you heard and check your understanding.
Ask open-ended questions
Pace your talking speed to match the participant’s
Listen to inconsistencies that may need to be challenged
To be an effective listener, listen to yourself
To what degree does my non-verbal behaviour show that I value the participant (e.g. am I shuffling papers)?
What attitudes am I expressing non-verbally (e.g. am I focusing on the recording technology)?
What attitudes am I expressing verbally (e.g. am I making the person feel like a “subject” rather than a “participant”)?
What is stopping me from giving my full attention to this participant (e.g. noise, tiredness, boredom)?
To be an effective listener, listen to yourself
What am I doing to handle these distractions?
How intently am I listening to what the participant is saying?
What am I doing to demonstrate my willingness to listen to the participant?
How well am I reading the participant’s non-verbal behaviours?
How careful am I not to over-interpret non-verbal behaviour?
What NOT to say to participants (slightly adapted from Rubin & Chisnell’s Handbook of Usability Testing)
- Saying, “Remember, we’re not testing you,” more than three times.
- “Don’t worry, the last participant couldn’t do it either.”
- “No-one’s ever done that before.”
- “HA! HA! HA!”
- “That’s amazing! I didn’t even know it could go in upside down!”
- “Could we stop for a moment? Watching you struggle is making me tired.”
- “I didn’t really mean you should press any button.”
- “Yes, it’s normal for observers to laugh during a test.”
- “Don’t feel bad, many people take 15 or 16 tries.”
- “Are you sure you’ve used a computer before?”
Hats moderators wear in a usability test (based on Carolyn Snyder’s Paper Prototyping)
Sports Commentator- Ensure that observers get as much useful
information from the test as possible.
Getting participants to think aloud
Getting participants to think aloud
Phase 1 — Give instructions
Getting participants to think aloud
Getting participants to think aloud
Phase 2 — Demonstrate and let them practice
Demonstrate and let them practice
Demonstrate thinking aloud with an example:
Turn your phone to silent
Get the participant to practice with a different example:
Adjust the height of his or her chair
Make sure your practice task is easy: you want the user to practice the technique and feel successful!
Project Activity: Quick activity
Practise welcoming a participant- Set expectations and reassure your
participant.
- Ask for your participant’s consent to record the session.
- Demonstrate thinking aloud.
- Get your participant to practise thinking aloud.
Getting participants to think aloud
Getting participants to think aloud
Phase 3 — Probe with open questions
Probe with open questions
Keep
talking…
Tell me
more
about that What are you
thinking
right now?
it was
easy!what was
easy?
is that how
it's supposed
to work?
is that how
you expected
it to work?
PHASE 3: probe with open questions
Probe with open questions
Keep
talking…
Tell me
more
about that What are you
thinking
right now?
it was
easy!what was
easy?
is that how
it's supposed
to work?
is that how
you expected
it to work?
PHASE 3: probe with open questions
Probe with open questions
Keep
talking…
Tell me
more
about that What are you
thinking
right now?
it was
easy!what was
easy?
is that how
it's supposed
to work?
is that how
you expected
it to work?
PHASE 3: probe with open questions
is this a useful
feature?
Mmm, do I think that someone, somewhere
will find some use for this feature?
would this feature, as it's currently presented, be
valuable to the way you choose products today?
Mmm, there are some good things and some bad things, so what do
i think overall?
was there anything you particularly
liked or disliked about the
prototype?what specifically?
Is this a good idea?
don't ask users to extrapolate…
…instead, focus on the present
…focus on the specifics
instead of generic questions…
is this a useful
feature?
Mmm, do I think that someone, somewhere
will find some use for this feature?
would this feature, as it's currently presented, be
valuable to the way you choose products today?
Mmm, there are some good things and some bad things, so what do
i think overall?
was there anything you particularly
liked or disliked about the
prototype?what specifically?
Is this a good idea?
don't ask users to extrapolate…
…instead, focus on the present
…focus on the specifics
instead of generic questions…
crikey, he thinks this option should appear on the home page and he won't like it if i say
otherwise.
don't you think this option would be better if it was available on the
home page?
is there any other place you'd like to see
a feature like this?
do i have my 'surprised' face on again? i better
think of something surprising now.
was that what you expected? what did you
expect?
you seem surprised, were you?
don't be judgemental
a better question would be…
don'T encourage confabulation
but keep the participant talking
crikey, he thinks this option should appear on the home page and he won't like it if i say
otherwise.
don't you think this option would be better if it was available on the
home page?
is there any other place you'd like to see
a feature like this?
do i have my 'surprised' face on again? i better
think of something surprising now.
was that what you expected? what did you
expect?
you seem surprised, were you?
don't be judgemental
a better question would be…
don'T encourage confabulation
but keep the participant talking
Hats moderators wear in a usability test (based on Carolyn Snyder’s Paper Prototyping)
Scientist- Responsible for avoiding test bias and
recording the data.
• Representative users.• Appropriate task
scenarios.• Balanced design• Neutral questioning. • Where bias can’t be
avoided, make a note. • Decide when to intervene
to help a participant• Log the data• Record audio and video of
the session. • Collect accurate
measures of task completion, time on task and satisfaction.
Usability Test Plan Toolkit
Use a checklist to remind you how to phrase unbiassed questions.
You’ll receive a copy of this with your second refresher training e-mail.
!"#$%&'()*+,-*.,/#-0)'12*0*3(04'&')5*)#()*6#&$,.'12*)"#*70-)'$'701)*
! "#$!#%&#'$($)*+,!
!"#$"%&'()'*+"&*,&-"+-.&)*/"+&*,&0*1")2-*).&3*-"&-24")&231&'2)-$5$'23-.&"+-$02-"1&1()2-$*36&
! -#(,,./#!$0#!&(/$)')&(+$!
7*-&2&-"+-&*,&-8"&'2)-$5$'23-.&%23-&-*&/"2)3&-8"&+-)"39-8+&231&%"243"++"+&*,&-8"&1"+$93.&0*1")2-*)&$+&3*-&-8"&1"+$93")6&
8)0-)'12*0*1#9*)0(%*
! 10#'2!3*/!.+4#/,$(+4)+5!
:+4&-8"&'2)-$5$'23-&-*&)"'"2-&-8"&-2+4&$3&8$+&*)&8")&*%3&%*)1+6&
:3-'12*)"#*)0(%*
! 6#$!$0#!&(/$)')&(+$!$(72!
;2(+"&-*&/"-&-8"0&,$3$+8&2&-8*(98-&<&1*3=-&-2/4&*#")&-8"06&
! -#('$!$*!8*49!7(+5.(5#!
;$54&('&*3&'2)-$5$'23-&>*1?&/239(29"&*,&,)(+-)2-$*3&*)&233*?235"&231&21@(+-&$3-")25-$*3&255*)1$39/?6&
! :*+;$!4#3#+4!$0#!4#,)5+!
:5-&2+&2&3"(-)2/&'2)-?&*(-+$1"&*,&-8"&1"+$93&-"20&A&?*(&2)"&-8")"&,*)&)"+"2)58.&3*-&-*&1","31&1"+$93+6&
!<($'0!3*/!,#73='#+,*/)+5!
;)*>"&,()-8")&%8"3&'2)-$5$'23-+&5"3+*)&-8"$)&%*)1+&*)&>/20"&-8"0+"/#"+6&
! :#>#7*&!3/)#+479!/(&&*/$!
B"&%"/5*0$39&231&8"/',(/&-*&-8"&'2)-$5$'23-&%$-8*(-&>"$39&-**&"0'2-8"-$5&$,&'2)-$5$'23-&$+&3"92-$#"&-*%2)1&2&1"+$936&
! ?/*>)4#!+#.$/(7!#+'*./(5#@#+$!
C)"2-&'*+$-$#"&231&3"92-$#"&5*00"3-+&-8"&+20"&%2?6&DC8$+&$+&#")?&8"/',(/6&E*(=)"&1*$39&@(+-&%82-&%"&3""1&?*(&-*&1*F6&
! -#&#($!
!"$-")2-"&5*00"3-+&#")>2-$0&-*&"35*()29"&"/2>*)2-$*36&DE*(&+2$1&?*(&(+(2//?&+-2)-&-8$+&2''/$52-$*3&,$)+-GF&
! 10#'2!9*./!)+$#/&/#$($)*+!
!"'8)2+"&-8"&'2)-$5$'23-=+&+-2-"0"3-&$3&?*()&*%3&%*)1+&-*&"3+()"&255()25?6&DH&20&8"2)$39&?*(&+2?IF&
!A)+)@)B#!@*4#/($*/!4),$/('$)*+,!
B"&2%2)"&*,&23?&233*?$39&*)&)"'"-$-$#"&9"+-()"+&231&")21$52-"&-8"06&
! C+'*./(5#!$0)+2=(7*.4!
!"0$31&-8"&'2)-$5$'23-&-*&-8$34J2/*(16&DK82-=+&9*$39&-8)*(98&?*()&8"21&2-&-8"&0*0"3-GF&
! :*+;$!3*/'#!*&)+)*+,!
:+4&2>*(-&'2)-$5$'23-&*'$3$*3+&%8")"&2'')*')$2-".&>(-&1*3=-&,*)5"&23&*'$3$*3&$,&-8"&'2)-$5$'23-&1*"+3=-&,""/&+-)*39/?6&
! 17(/)39!(!$#/@!*/!'*+'#&$!
DK82-&1*"+&L+?+-"0=&0"23&8")"GF&DK82-&1*&?*(&52//&-8*+"&>(--*3+GF&
;(%'12*<3#()',1(*
! D>*)4!7#(4)+5!E.#,$)*+,!
B21M&DN*%&1$1&?*(&/$4"&-8"&/*9$3&+5)""3GF&O**1M&DK82-&1$1&?*(&-8$34&2>*(-&-8"&/*9$3&+5)""3GF&
! D>*)4!'7*,#4!E.#,$)*+,!
B21M&DP*"+&-8$+&024"&+"3+"GF&O**1M&DK82-&2)"&?*()&$0')"++$*3+&*,&-8$+GF&
! D>*)4!8)(,#4!E.#,$)*+,!
B21M&DH+&-8$+&,"2-()"&8"/',(/&-*&?*(GF&O**1M&H+&-8$+&,"2-()"&8"/',(/&-*&?*(&*)&$+&-8$+&,"2-()"&3*-&8"/',(/&-*&?*(G&K8?GF&
! F*'.,!*+!('$.(7!8#0(>)*./!(+4!(>*)4!5#+#/(7),($)*+!
B21M&DH+&-8$+&2&9**1&$1"2GF&O**1M&DN*%/(2>/"&%*(/1&-8$+&>"&-*&?*(&$3&?*()&@*>GF&
! :*+;$!(,,)5+!87(@#!
B21M&DK8?&1$1&?*(&1*&-82-GF&O**1M&DH&3*-$5"1&?*(&1$1&Q6&C2/4&0"&-8)*(98&?*()&-8*(98-&')*5"++6F&
! D,2!3*/!,&#')3)'!#%(@&7#,!G!#+'*./(5#!,$*/9$#77)+5!
B21M&DP*&?*(&82#"&-)*(>/"&%$-8&+"2)58GF&O**1M&DC"//&0"&2>*(-&2&-$0"&%8"3&?*(&5*(/13=-&,$31&%82-&?*(&%")"&+"2)58$39&,*)6F&
! D>*)4!7*+5!(+4!*>#/79!'*@&7#%!E.#,$)*+,!
B21M&DK8"3&-8$+&+$-"&%2+&1"+$93"1.&-8")"&%2+&1$+5(++$*3&%8"-8")IF&O**1M&DN*%&5*(/1&-8$+&+$-"&>"+-&0""-&?*()&3""1+GF&
! :*+;$!(,2!&(/$)')&(+$,!$*!/#=4#,)5+!$0#!)+$#/3('#!
B21M&DK82-&%*(/1&>"&-8"&>"+-&%2?&-*&1"+$93&-8$+GF&O**1M&DH3&?*()&"R'")$"35".&%82-&*-8")&%">&+$-"+&1*&-8$+&%"//GF&
! D,2!'./)*.,!E.#,$)*+,!$*!#%&7*/#H!8.$!(>*)4!
'*+3/*+$($)*+!
B21M&DP*&?*(&82#"&23?-8$39&"/+"&-*&+2?&2>*(-IGF&O**1M&DS23&?*(&-"//&0"&0*)"&2>*(-IGF&
! I#$!8#0)+4!$0#!8#0(>)*./J!
B21M&DN*%&%*(/1&?*(&)2-"&?*()&"R'")$"35"&%$-8&-8"&+$-"GF&O**1M&DN*%&%*(/1&?*(&)2-"&?*()&"R'")$"35"&%$-8&-8"&+$-"G&K8?GF&
! ".@@(/),#!(+4!4/(K!(!'*+'7.,)*+!
DT*&%82-&H&20&8"2)$39&?*(&+2?&$+&-82-&-8"&%2?&-8"&LB**4&2&,25$/$-?=&$+&$0'/"0"3-"1&8")"&$+&5*3,(+$39.&>(-&-8"&$1"2&$+&+$0'/"6F&
6-077'12*37*
! D,2!3*/!(+!*>#/(77!(,,#,,@#+$!
DS23&?*(&+(002)$U"&?*()&"R'")$"35"&%$-8&Q&-*12?GF&*)&DK82-&%*(/1&?*(&+2?&%")"&-8"&-*'&-%*&+-)"39-8+&231&%"243"++"+6F&
! -#E.#,$!'*@@#+$,!/#5(/4)+5!$*&)',!+*$!/(),#4!89!
@*4#/($*/!
DK82-&5*00"3-+&1*&?*(&82#"&*3&-*'$5+&%"&1$13=-&5*#")&-*12?GF&*)&DK82-&"/+"&+8*(/1&H&82#"&2+4"1GF&
Quick Exercise: Improve these closed questions
For example:
- Closed question: Don’t you think you should try the Back button?
- Open question: How else might you return to the home page?
Are you looking for search?
Did you just click on the Submit button?
Didn’t you see the link to the site map?
Don’t you think you should look in the pull-down menu?
Do the radio buttons make your options clear?
Do you think the design of this page is good?
Do you think people will find this easy to use?
Getting clients and the development team to log usability issues
Ask the people observing the test to write a short description of each usability problem they see on a sticky note.
One problem per sticky note.
Identify the participant associated with the observation (e.g. P5).
At the end of the day’s testing, work with the observers to group, prioritise and label the usability issues.
Why data logging is important
As you watch the test, you should note down the participant’s behaviours and code your observations.
Good note taking saves hours in analysis time.
Why data logging is important
- Data logging ensures you note
all behaviours, not just the ones
that stand out (this helps reduce bias in your observations).
- It helps you concentrate and
focus on the session.
- When lots of observations come at
once, you will be able to just note
the observation code — you can
then review it later on the videotape.
Why data logging is important
- When scanning your notes, the observation codes make it easy to
distinguish one class of
observation (e.g. the usability issues) from other observations.
- It’s one of those things you’ll be glad you did when there are
problems with the video
recording (e.g. when the sound is
poor or when the recording is corrupted).
S Start task
E End task
X Usability problem
* Video highlight — an “Ah-ha!” moment
G Comment from P (general)
P Positive opinion
N Negative opinion
B Bug
F Facial reaction (e.g. surprise)
A Assist from moderator
Q Gives up or wrongly thinks finished
H Help or documentation accessed
M Misc (general observation by logger)
C Clickstream (optional)
How to classify your observation
How to log usability problems
As a rule of thumb, you should average about one observation per minute.
But remember this is an average- Observations are a bit like buses (none for
ages, then three come along at once).
What you should write down:- The time
- The class of observation
- A short description
S Start task B Bug
E End task F Facial reaction (e.g. surprise)
G General comment A Assist from moderator
P Positive opinion Q Gives up or wrongly thinks finished
N Negative opinion H Help or documentation accessed
X Usability problem M Misc (general observation by logger)
* Video highlight — an “Ah-ha!” moment C Clickstream
Time Code Description
Thank You!
Dr David TravisManaging Director
180 Piccadilly, London, W1J 9HFUSERFOCUS
Work 020 7917 9535Mobile 07747 016132Email [email protected] @userfocus