Running user testing

Strategy Research Design Delivery

Running user testingSteve Cable (@steve_cable)Walt Buchan (@silverfoxyboy)

Thursday, 25 November 2010

2

Why user test

Pain points identified from analyticsReported problems Strategic changes

People test their website for different reasons


3

What stage in the design process should you test?

HTMLDesignsWireframesSketches

The earlier you begin testing the better.

Test as many stages of the design as the budget allows

There are issues that testing designs will bring up that wireframes won’t, and issues that testing HTML will bring up that designs won’t


4

Recruitment Getting the right people is vital

the type of recruits usually come from user groups already defined by the client

We like to define user groups by the tasks they will perform on the site/software

for a shopping site it could be split between people who shop for themselves, and people who shop for others

Usually its a good idea to get a mix of existing and new users

We avoid ‘Expert’ users as these people usually only represent a small portion of your user group


5

The test plan

We produce a test plan in collaboration with our client

It defines the overall objectives of the test and what we want to find out on each part of the website

However we don’t use it as a rigid script. We use it as a guide

This means if we find an interesting new line of investigation during a test, we can explore it


6

Who should be involved?


7

Users

6 participants plus 1

The tests are qualitative research, meaning we are not trying to prove something, just identify issues

we find six participants are enough to identify the majority of issues

The extra user is paid to be around all day incase one of the 6 participants does not show up

it means we can avoid the embarrassing situation of a no-show


8

Moderators

That’s us!

We use two moderators for our tests

One moderator can concentrate on the interview

The other can concentrate on the note taking (And looking after the clients)

This means we can do more in depth interviews and have more comprehensive notes to work from afterwards


9

Stakeholders

We always encouraged Clients, developers, designers and anybody who’s involved to come and watch the tests

This is so stakeholders can see any issues users are having for themselves

It’s much more convincing than having a consultant telling them about it a week later


10

The test


11

What devices or media can be tested

Mobile devices e.g. Phones, Media players, Tablets Personal computers e.g. PCs, Apple Macs

Printed material e.g. Brochures, Printed forms Prototypes e.g. Paper based, magnetic stickers

We’ve tested all of these devices and materials for a number of different clients


12

Test structure

Preamble Usability test Summary & Close

Depth interview Usability testSummary & Close

Test focussing on usability

Test focussing on user experience

Examples of additional user testing • Benchmarking exercises• Simple card sorting and ranking exercises• Preferences e.g. Photography styles, Tone of Voice, Visual style

This structure focuses on finding specific usability issues

This structure focuses on the wider context of use

Great for producing mental models and user journeys

Tests can be structured in different ways to achieve different results


13

Interview techniques

• Put the participant at ease ‘It’s not you being tested, it’s the website’

• Probe where necessary, leave space for the participant to think

• Listen, show interest, (Be consistently interested or the participant will learn what you want to find out)

• Don’t lead the participant, be careful with words and body language

• Use open questions: Who, What, When, How, Where, Why

• It’s about what they actually do, not what they might do (or what their mum would do)

There are loads of interview techniques, but these are the six we feel are the most important to remember


14

The set up


15

Why no two way mirror?

Big mirrors can make participants feel uneasy

Most people only know them from police interrogations in films

Anybody watching has to be silent in the dark

It can be hard for anybody to stay alert after four hours of testing in these conditions


16

The set up

Instead we use video link up

This makes the interview room seem a little more private

It also means the people watching can make as much noise as they like

It also gives them a better view of whats going on as they can see:• A direct feed from the users

computer (With live eyetracking data

• A shot of the users face

• A shot of the computer screen (For when the user points at something)


17

The output


18

Eye tracking data

Heatmaps show an aggregated view of what all the participants looked at on a page

It’s great for backing up what we have identified as issues

This example shows how all the users were missing important information on the right of the page


19

User videos We produce full length videos of the user tests for the clients to keep

We also produce highlight clips to highlight particular problems that were identified


20

Recommendation report

Our reports are designed to be easy to use

Very visual with one clear concise point per page

We back up each point with a quote from the participants

We don’t just highlight problems. We also provide clear actionable recommendations

These recommendations are prioritise with the client to provide them with a clear direction for improving their site


Thank you

21

Steve CableUser Experience Consultant

[email protected]

+44 117 930 3541

Walt BuchanSenior User Experience Consultant

[email protected]

+44 117 930 3543


mailto:[email protected]




Technology

Running user testing