Data gathering and analysis Class activity Discuss eye tracking assignment

Identify a question statement/hypothesis about what will attract user’s visual attention.

What answers did you find?

What were your overall observations?

Often you must determine the effectiveness of one design or feature or function over another.◦ What interface is more efficient?◦ What device design accommodates users’ hand

movements best.◦ What is the best menu structure?◦ Etc.

Four key issues of data gathering

Data recording

Interviews

Questionnaires

Observation

Choosing and combining techniques

1. Setting goals◦ Asking questions | making predictions◦ Decide how to analyze data once collected

2. Identifying participants Decide who to gather data from

3. Relationship with participants◦ Clear and professional◦ Informed consent when appropriate

4. Triangulation◦ Use more than one approach

5. Pilot studies◦ Small trial of main study

Q. Are Sans Serif fonts easier to read on screen?

Predictions: ◦ Information written in sans serif type will be read

faster than information written in serif type.

◦ Information written in sans serif type will be read faster on Retina display devices than information written in serif type.

T T

Q. Is checkout procedure A more efficient (takes less time) than checkout procedure B.

Collecting Data: Collecting Data: Interviews | Focus GroupsInterviews | Focus Groups

• Unstructured - are not directed by a script. Rich but not replicable.

• Structured - are tightly scripted, often like a questionnaire. Replicable but may lack richness.

• Semi-structured - guided by a script but interesting issues can be explored in more depth. Can provide a good balance between richness and ability to replicate.

Interview questionsInterview questions

• Two types:− ‘closed questions’ have a predetermined answer

format, e.g., ‘yes’ or ‘no’− ‘open questions’ do not have a predetermined

format

Avoid:− Compound sentences - split them into two− Jargon and language that the interviewee may not

understand − Leading questions that make assumptions e.g., why do

you like …?

Compound sentences - split them into two

Do you like the information structure and the navigation labels? NO!

Do you like the information structure? YES Do you like the navigation labels? NO

Interview questionsInterview questions

Compound sentences - split them into two. Do you find the text helpful and navigation

easily to follow? Do you find the text helpful? Do you find navigation easily to follow?

Jargon and language that the interviewee may not understand. Did you find the UX section of the HCI and IDs

category useful? Did the UI implementation of JS work?

Leading questions that make assumptions: I know this screen is crowded but can you find…

Running the interview |Focus groupRunning the interview |Focus group• Introduction –

• introduce yourself, • explain the goals of the interview, • reassure about the ethical issues, • ask to record, • present any informed consent form.

• Warm-up – make first questions easy and non-threatening.

• Main body – present questions in a logical order• A cool-off period – include a few easy questions

to defuse tension at the end• Closure – thank interviewee, signal the end,

e.g., switch recorder off.

• Props - devices for prompting interviewee, e.g., a prototype, scenario

Focus groupsProps

Decide on whether phrases will all be positive, all negative or mixed.

I like this website (SD) 1 2 3 4 5 (SA) Navigation was NOT easy (SD)1 2 3 4 5 (SA) Information was easy to find (SD)1 2 3 4 5

(SA)

Question and response Question and response formatformat

• Rating scales

– Likert scales

– (SA) 1 2 3 4 5 (SD) or (SD) 1 2 3 4 5 (SA)

– semantic scales

(Clear) |_|_|_|_|_|_|_|_| (Confusing)

(Usable) |_|_|_|_|_|_|_|_| (Unusable)

Make sure purpose of study is clear Promise anonymity Offer a short version for those who do not have time

to complete a long questionnaire Follow-up with emails, phone calls, letters Provide an incentive 40% response rate is high, 20% is often acceptable

Responses are usually received quickly No copying and postage costs Data can be collected in database for analysis Time required for data analysis is reduced Errors can be corrected easily

Preventing individuals from responding more than once

Direct observation in the field◦ Degree of participation (insider or outsider)◦ Ethnography

Direct observation in controlled environments

Indirect observation: tracking users’ activities◦ Diaries◦ Interaction logging◦ Analytics

Think-aloud technique◦ Usability studies◦ AB Testing

• Indirect observation• Diaries• Interaction logs

Depends on◦ The focus of the study◦ The participants involved◦ The nature of the technique◦ The resources available

Video Sketching:

http://www.youtube.com/watch?v=3Sk7cOqB9Dk&feature=youtu.be

Source: http://www.uxbooth.com/articles/tools-for-sketching-user-

experiences/

Source: http://boxesandarrows.com/storyboarding-ipad-transitions/

1.Sketch the checkout process (assume the user is buying a loaf of bread

or some other product of your choice). Sketch two alternative designs.

2.Ask yourself questions about the designs.

3.Write your questions down.

4.Make predictions/hypotheses.

5.Collect data.

Question: Will Pop-up selection result in

reduce check-out time and greater customer

satisfaction?

Hypothesis: Users will complete checkouts

with Pop-up selection (Design A) quicker

than link selection (Design B).

Quantitative analysis

Qualitative analysis

Tools to support data analysis

Theoretical frameworks

Presenting the findings

Quantitative data – expressed as numbers

Qualitative data – difficult to measure sensibly as numbers, e.g. count number of words to measure dissatisfaction

Quantitative analysis – numerical methods to ascertain size, magnitude, amount

Qualitative analysis – expresses the nature of elements and is represented as themes, patterns, stories

Averages ◦ Mean: add up values and divide by number of data

points◦ Median: middle value of data when ranked◦ Mode: figure that appears most often in the data

Percentages Graphical representations give overview

of dataNumber of errors made

00.5

11.5

22.5

33.5

44.5

1 3 5 7 9 11 13 15 17

UserN

um

be

r o

f e

rro

rs m

ad

e

Internet use

< once a day

once a day

once a week

2 or 3 times a week

once a month

Number of errors made

0

2

4

6

8

10

0 5 10 15 20

User

Nu

mb

er o

f er

rors

mad

e

Simple qualitative analysis

• Unstructured - are not directed by a script. Rich but not replicable.

• Structured - are tightly scripted, often like a

questionnaire. Replicable but may lack richness.

• Semi-structured - guided by a script but interesting issues can be explored in more depth.

Visualizing log data

Interaction profiles of players in online game

Log of web page activity

Online discussion

Online

discussion

Date

Time

Web Behavior Graph

New York Times

Simple qualitative Simple qualitative analysisanalysis

• Recurring patterns or themes– Emergent from data, dependent on observation

framework if used

• Categorizing data– Categorization scheme may be emergent or pre-specified

• Looking for critical incidents– Helps to focus in on key events

• Spreadsheet – simple to use, basic graphs

• Statistical packages, e.g. SPSS

• Other data analysis tools– Categorization and theme-based analysis

– Quantitative analysis of text-based data

– NuD*IST

– Observer XT

– Morae

– AtLast ti

Theoretical frameworks Theoretical frameworks for qualitative analysisfor qualitative analysis

• Basing data analysis around theoretical frameworks provides further insight

• Three such frameworks are:– Grounded Theory– Distributed Cognition– Activity Theory

Grounded TheoryGrounded Theory

• Aims to derive theory from systematic analysis of data

• Based on categorization approach (called ‘coding’)

• Researchers are encouraged to draw on own theoretical backgrounds to inform analysis

Web Behavior Graph

New York Times

LevenShtien Distance

CodeBehavior Description

Link space

CL Click link

CI Click image link

B Click Back button

PG Page change

Search spaceS1 Set search Set search option to search local site or on the Web.

S Type query in search

Visual scan spaceVT Visually scan text Fixated on screen text or actually reads text aloud

VS Visually scan pageLooks at screen, does not fixate on any specific area, eye movements are large and cover large portions of screen.

SRU Scrolls up May look at screen, does not appear focused on screen and scrolls upSRD Scroll down May look at screen, does not appear focused on screen and scrolls downSRH Scroll horizontally May look at screen, does not appear focused on screen and scrolls horizontallyCS Cursor scan of links Purposefully passes cursor over links within the page or in navigation areasCT Cursor scan of text Purposefully passes cursor over text while readingCA Cursor scan of advertisement Purposefully passes cursor over advertisement, usually while talking about advertisement.CSP Cursor scan of image Purposefully passes cursor over image, usually while talking about image.

CSRD Cursor scans links while scrolling downPurposefully passes cursor over screen areas. Highly focused on the screen, may lean forward to look at screen, may verbalize that he/she is searching while scrolling down.

CSRU Cursor scans links while scrolling upPurposefully passes cursor over screen areas. Highly focused on the screen, may lean forward to look at screen, may verbalize that he/she is searching while scrolling up.

WV Watch video User watches video.

VV Visually scan video popup windowLooks at video popup window, does not fixate on any specific area, eye movements are large and cover large portions of window.

VSRD Visually scan page while scrolling down

Highly focused on screen, may lean forward to look at screen, may verbalize that he/she is searching while scrolling down.

VSRU Visually scan page while scrolling upHighly focused on screen, may lean forward to look at screen, may verbalize that he/she is searching while scrolling up.

Interaction space

TY Typing in form

PH Photo gallery popup User clicks link to display a photo gallery window.VD Video popup User clicks link to display a video window.

CP Close popup window

MP Minimize popup window

MX Maximize popup window

CPN Click next button in photo window

VC Use video controls

Distributed CognitionDistributed Cognition

• The people, environment & artifacts are regarded as one cognitive system

• Used for analyzing collaborative work

• Focuses on information propagation & transformation

Activity TheoryActivity Theory

• Explains human behavior in terms of practical activity with the world.

• When people interact with environment, production of tools results.

•Tools are "exteriorized" forms of mental processes.

• Provides framework that focuses analysis around the concept of an

‘activity’.

• Stresses importance of involving representatives of the target community in

the process of designing any type of computer mediated artifact.

(http://www.slis.indiana.edu/faculty/yrogers/act_theory2/)

Presenting the findingsPresenting the findings

• Only make claims that your data can support

• The best way to present your findings depends on the audience, the purpose, and the data gathering and analysis undertaken

• Graphical representations