Lecture #7 COLLECTION, ANALYSIS AND PRESENTATION OF DATA FROM USABILITY TESTS TUR Winter 2012/2013

Lecture #7COLLECTION, ANALYSIS AND PRESENTATION OF DATA FROM USABILITY TESTS

TUR Winter 2012/2013

Standard Method of Usability Test

User gets an assignment The observer observes Problem !

– It is hard to remember all the user activities

TUR 2010

(2)S. Greenberg

Standard Method of Usability Test

Solution is:data collection,analysis,presentation.

TUR 2012

(3)

DATA COLLECTION

Data collection introduction

When?– During the test execution.

What?– Everything, you are interested in.

How?– By means of standard or specialized applications.

Who?– Each member of test team.

Why?– To use it in later stages (analysis and presentation)

Where?

TUR 2012

(5)

Usability laboratory – controlled env.Observer roomObserver room(UI designers(UI designers, , developersdevelopers, , test organizers)test organizers)

Participant roomParticipant room(test participant(test participant, , moderator)moderator)

Testing in the field – uncontrolled env.

TUR 2012

(7)

Collected Data Sources

A/V recording Notes Custom logs Post-test questionnaire Interview Retrospecting

TUR 2012

(8)



TUR 2012

(10)

Recording the Observation

Video Recording– Can see what the user does– Typical are multi-camera setup

• One camera records the screen

• One camera records the participant face and body

• Problem of synchronization

– “Big Brother problem”

Audio Recording– Can hear what the user does– Good for think-aloud protocol– Audio recording is very important

TUR 2012

(11)

A/V Recording - Setup

Participant PC recording– Allows recording of approx. 2 video sources (Morae Recorder, …)– Video recording can take some processing resources of the PC– Not usable for accelerated graphics!!– You can perform test with 1 computer

TUR 2012

(12)

A/V Recording - Setup

Participant PC recording– Allows recording of approx. 2 video sources (Morae Recorder, …)– Video recording can take some processing resources of the PC– Not usable for accelerated graphics!!– You can perform test with 1 computer

TUR 2012

(13)

A/V Recording – Setup

Observer PC recording– Dedicated computer with multiple data sources (Noldus, …)– Screen capture of all contents with higher FPS (from monitor output)– No problem with accelerated graphics– At least 2 computers necesarry.

Observer/Moderator notes

Notes taken by observer/moderator Electronic or paper/pencil

– Basic solution but can be improved by applications

Single user or collaborative

TUR 2012

(14)

ProsProsEasy to prepareEasy to prepareScalableScalable

ConsConsDistract from observationDistract from observation



TasksTask 1: xx xxx xx x xxx xx xxx.

Lo

gT

ask 1: xx xxx xx x xxx xx xxx.



I am trying to create new project

TasksTask 1: xx xxx xx x xxx xx xxx.

Záp

isÚ

loh

a 1: xx xxx xx x xxx xx xxx.

I expected “New Project” item in menu “File”.

I cannot find particular menu item

Test log

Task 1:12:01 start12:07 endÚloha 2:12:09 start12:10 creates new project12:12 cannot find particular menu item12:13 expects “New Project” item ïn menu “File

TUR 2012

(18)

Think-aloud Protocol

Pros: Provides insights to the user’s thinking Most common method in the Usability Engineering Cons:

– Can alter the mental process of the user– Unnatural– It’s difficult to speak when focusing on the task

TUR 2012

(19)

Example of Hints (Think-aloud)

“Please, speak on” “Please tell me what you think.” “Please tell me what you are trying to do.” “Are you searching for something in particular?” “What do you think that it’s going to happen now?” “What did you mean by that?”

Adapted from Jake Wobbrock

TUR 2012

(20)

Video Annotations - Datalogging

Processing of AV recording takes 4-8 times the length of the recording– E.g.: 6 participants, 1h each video = minimum 24h! ~ 2-4 days

Typical solution:– While capturing: Annotate video– While processing: Focus on important parts only

Problems of video annotations– It is hard to keep up with the tempo of what's going on – You may miss important interaction during annotation – Annotations contain data from different categories

• e.g. user opinions, behavioral observations and demographic notes

Improvement - Markers

Basic Marker Definition

Code Definition Description

X Usability problem

D Duplicate usability problem (described earlier)

V Video highlight — an "Ah-ha!" moment

C Comment (general comment by participant)

P Positive opinion expressed by participant

N Negative opinion expressed by participant

B Bug

F Facial reaction (e.g. surprise)

H Help or documentation accessed

A Assist from moderator

G Gives up or wrongly thinks finished

I Design idea (design insight by logger)

M Misc. (general observation by logger)

TUR 2012

(21)

TUR 2012

(22)

Example of Video Annotations

An actual usability test Website

– Web portal of a university

Task #8 (10 tasks total)– “Find information on life-long education program at the

Faculty of Architecture.”

TUR 2012

(23)

Example of Video Annotations

Time Note Task ID Marker

15:06 Session Starts M[some lines omitted]

15:16 Tries to find “Info for the Students” #8 M

15:17 Can’t find in the left-hand menu X

Goes systematically through all links in this menu X

Wishes to use full text search, does not how C

Found link “Alumni” M

15:20 Life-long Education link NOT FOUND X

15:21 Found link “For applicants” M

FOUND link “Life-long education” M[some lines omitted]

15:49 Session Ends M

Video Annotations Tips

Marker categories are difficult to remember– Start with subset of the markers– Add more markers when the technique became familiar

Multiple observers:– Can increase number of found usability problems– Can put more information into annotations

TUR 2012

(24)

TUR 2012

(25)

Other Collected Data Low level interaction data

– Mouse clicks, mouse movements– Key presses– Can be used for location of interesting parts in video recording

Application data– Generally any information that may be interesting or valuable during

evaluation– Typically requires some hooks, API or functionality inside the

applications– Examples

• Web - url, rendered page, Basket content• Mobile – GPS position, content of search field

Eye-tracking …

(25)

Testing in the field – uncontrolled env.

TUR 2012

(26)

TUR 2012

(28)

After-test Activities

It is important to find out what the participants think Do they find the test easy? Difficult? Conditions of the test: Good, bad? Other comments?

Data collection same as for the usability test– Audio/video recording– Annotations– Rarely other data (low level, application)

TUR 2012

(29)

Post Test Questionnaire

Collection of insights after the test Allows easier data collection compared to

interview Typically uses Likert scale (1-5) Allows also Yes-No questions Allows also open ended questions (limited)

Example of Post Test Questionnaire

An actual usability test– Web portal of a university

TUR 2012

(30)

TUR 2012

(31)

Interview

Good for finding out specific problems– “Set up” the question to match the context– Can focus on the problems as they show up during the

interview– Good for research studies (open-ended questioning)– Leads to specific suggestions

Problems– Statements are subjective– Time-consuming– The interviewer can easily affect the results

TUR 2012

(32)

How to carry out an interview

Plan the list of basic questions– Several good questions can start an interview

• (But avoid leading questions.)

– These questions can focus the interview.– Can be based on the results received from the

observation.

TUR 2012

(33)

Retrospective testing interviews

Post-observation interview– Observe (make the test)– Capture a video recording– User watches the recording and comments on in

• Explains an unclear behavior during the test

• Great for interpretation of the post-test interview

• Avoids misinterpretation

• Can identify particular improvements

Do you know

why you never tried that

option?I didn’t see it. Why don’t you make it look

like a button?

S. Greenberg

TUR 2012

(34)

Critical Situations during an Interview

People speak about problems that have emerged– People vividly speak about marginal problems

• Important only for them

– Problem has emerged that wasnot captured during the test

I can never get my figures in the right

place. Its really annoying. I spent

hours on it and I had to…

Tell me about the last big problem you

had with Word

S. Greenberg

DATA ANALYSIS

Data Analysis Types

Summative studies– Studies conducted in order to generate metrics or measure

formal improvements, such as benchmark studies require more formal data analysis methods since they are collecting data for metrics.

Formative studies– When the focus of a usability study is to diagnose problems and

offer solutions, less formal data analysis techniques are typically used.

TUR 2012

(36)

Source: UPA 2006 Idea Markets - Analyzing usability study results: Is it magic behind the curtain?Activator: Emma J. Rose, University of Washington

Strategies for Usability Data Analysis

Approaches to data analysis– Statistical analysis

• Used less frequently, requires knowledge about statistics

– Calculating metrics• Calculation of success and failure rates and questionnaire data such as

Likert scales

– Analyzing notes for patterns• Looking for trends and patterns, across tasks and users.

– Physical observations• Observing the facial and bodily expressions especially in regards to

frustration or confusion

– Analysis “on-the-go”• Changes to the design are made immediately based on informal notes

during a study

TUR 2012

(37)


TUR 2012

(39)

Calculating metrics (examples)

Time to carry out the task

Number of tasks carried out

Number of errors

Number of used (or unused) commands and functions

Frequency of help access

Frequency of useful help access

Frequency of positive (negative) comments of the participant

Ratio of participants preferring the tested system

etc …

TUR 2010

(40)

Application support – Morae

Graph Visualizations– Mouse Clicks Graph– Count of Markers Graph– SUS Survey Graph– Time on Task Graph– Web Page Changes Graph

TUR 2012

(41)

Analyzing notes for patterns I

Summarize the findings from the collected data– List of all important events

• Positive or negative aspects

– It’s a good idea to link back to the original data– Identify why there were problems

Things you can look for in your data:– Is the UI behaving predictably?

• Have the people behaved in the way you expected?

– Were all necessary functions available?– Wasn’t there too many unnecessary functions?

TUR 2012

(42)

Analyzing notes for patterns II

Read through the notes Look for:

– Repetitions– Things that could be caused by the same underlying

problem Can be done in the whole group of testers Cluster the observations

– By underlying problem• E.g.: Group all problems related to poor structure of the information

– By feature• E.g.: Group all problems related to printing

TUR 2010

(43)

Application support – Morae I

Timeline visualization– Shows annotations in the timeline– Each marker category has its own color

TUR 2012

(43)

Taskstart

Final anchorCurrentposition

Play task Eventmarkers

Editation andlogging tools

Timelinezoom

End oftask

TUR 2010

(44)

Application support – Morae II

Filtering of tasks, participants Annotation list and frequency graph

TUR 2010

(44)

TUR 2012

(45)

Application support – Morae III

Filtering of tasks, participants Annotation list and frequency graph

(45)

TUR 2012

(46)

Analyzing notes for patterns III

Describe the clusters– What was the problem

• E.g.: “5 out of 8 participants could not locate the menu item X”

– The impact of the problem• E.g.: “Function Y could not therefore be accessed.”

– Place where the problem occurred

TUR 2012

(47)

Analyzing notes for patterns IV

(continuation of the university website example)

Searching for course Priority: 2 The participants do not know where to look for

courses. They go to "study programmes." They do not understand the distinction between

"career courses" and "retraining courses" They do not know what the Masaryk Institute of

Advanced Studes is.

Physical observations

TUR 2012

(48)


TUR 2012

(49)


TUR 2012

(50)

ADVANCED DATA VISUALIZATIONS

TUR 2012

(54)

Visualization of detailed behavior

TUR 2012

(55)


TUR 2012

(56)


TUR 2012

(57)


TUR 2012

(58)

Visualization of user motion

TUR 2012

(59)

Visualization of user motion

High cognitive

load

HighROI

MiddleROI

Low ROI

Eye tracking visualization – Heat Map

TUR 2012

(60)

http://blogs.lib.ucdavis.edu/hsl/2010/01/14/peeking-at-jakob-nielsens-eyetracking-web-usability/

Eye tracking visualization – Heat Map

Advantages– Good for still images– Can show interesting information

Disadvantages– Some behavior is know (reading from left to right)– Some eye trackers are obtrusive– Calibration is necessary– There must be pauses during the test for some eye trackers

TUR 2012

(61)

PRESENTATION OF THE DATA

TUR 2012

(63)

Where the Test Results are Used?

Design

Evaluation Implementation

Factors that impact data analysis

Factors suggesting formal data presentation: – Summative studies– Lower level of support in company UCD processes– External audience– Novices

Factors suggesting informal data presentation:– Formative studies– Higher level of support in company UCD processes– Internal audience– Experts

TUR 2012

(64)


TUR 2012

(65)

Presenting the Results

Informal Report– Initial feedback for the

designers– Within 2 days after the end

of the test

– Up to cca. 4 pages– Short descriptions of the

problems– Organized in bullet points

– A preview of the Formal Report

Formal Report– Detailed description with

analysis– Within weeks after the end of

the test

– 10 – 100 pages

TUR 2012

(66)

Contents of the Formal Report Purpose and aim of the test Demographics of the participants

– Anonymized! Overview of the tasks Pre-test questionnaire and responses Post-test questionnaire and responses Qualitative and quantitative results of the tasks

– Using screenshots, photographs, videos, transcripts of the audio recordings, …

All forms for the test Suggestion of further tests Links to A/V recordings (highlights) See also http://www.usability.gov/templates/index.html

TUR 2012

(67)

Priority assessment for each usability problem

Indicate how severe some problems were Questions:

– Does the problem affect big amount of users?– Is the problem difficult for users to overcome?– Is the problem persistent?

Severity levels:– Critical – Prevent from finishing common task – Fix urgently– Serious – User slowdown – Fix as soon as possible– Medium – Frustration and irritation – Fix during update– Low – Cosmetic issue or spelling error – Fix when have time

Source: http://www.userfocus.co.uk/articles/prioritise.html

TUR 2012

(68)

Suggesting the Solutions

Be careful when formulating suggestions– It’s the designers’ responsibility to identify underlying

causes of the problem– Recommendations should serve as guidelines where the

solutions could be found

TUR 2012

(69)

CarDialer Example

TUR 2012

(70)

Audio/video highlights

Good to illustrate problem (for designer, programmer), they need to accept there’s a problem before they can fix it.

Show top 5 issues, show them in 5 videos, 5 minutes each (for problem understand or behavior comparison).

Videos can be misleading when context is missing (e.g. video recording starts too late)

(70)

TUR 2012

(72)

Anonymizing the Results

Make sure that no information that could lead to the identification of the participants appears in the report.– Unless the participant gives an explicit consent

E.g. use visualizationof the test in VR

TUR 2012

(73)

Conclusion

Data collection– Audio/video, video annotations with markers– Optionally Low level data and Application data (difficult to visualize)

Data analysis– Statistical analysis for quantitative data– Search for behavioral patterns in annotations and video

Data presentations– Informal and formal report– Video highlights

Thank you for you attention

Ivo Maly

[email protected]

TUR 2011

(74)



Documents

Lecture #7 COLLECTION, ANALYSIS AND PRESENTATION OF DATA FROM USABILITY TESTS TUR Winter 2012/2013