How do different early evaluation tools predict final site ... › sites › ... · there were a few minor concerns. The project is on a tight budget and the product owner wonders

How do different early evaluation tools predict final site navigation?In collaboration with Userzoom

David Juhlin | [email protected]

781.891.3142 | bentley.edu/uxc

David JuhlinBill AlbertDani Nordin

July 8, 2015

Table of Contents

Topic Slide

1. Introduction 3

2. Live site base line 9

3. Prototypes 14

4. Screenshot click testing 33

5. Tree testing 45

6. Summary 58

7. Translation into practice 61

Introduction

4

Tools to Investigate

Prototypes

Screenshot click testing

Tree testing

Live site (Benchmark)

?

?

?

5

Questions

Prototype test

• You have created high fidelity prototypes and conducted an unmoderated usability test to gain insight about the navigation. The outcome was primarily positive but there were a few minor concerns. The project is on a tight budget and the product owner wonders if the outcome would have been the same if the test was conducted on the final live design. What is your answer?

?

6

Questions

Screenshot click test

• You conducted a screenshot click test and the result was a success rate of 60%. You present this to the product owner who asks you if this should be a concern. What is your answer?

?

7

Questions

TreeTest

• You have conducted a tree test and made significant enhancements to the IA (the final version has an average success rate of 85%). You are now in a meeting and the group is considering if they need to do any more IA research or if that would be a waste of resources since the IA performed well. What is your answer?

?

8

Data Collection

Website Tested

Wells Fargo’s live website was used to establish the live site’s performance and as a benchmark against the other tools. The artifacts used for the early research tools were reverse engineered from the live website into prototypes and artifacts for screenshot click tests and tree tests.

Participants and Design of the Research Study

Participants were recruited through Amazon’s Mechanical Turk and a between subjects design was used. Every participant performed 7 tasks, which were designed to cover different features of the site and were minimally tweaked to suit the different types of research methods. The sample size for each research method was between 99-195 participants.

Live Site (Benchmark)

10

Live site

Remote Unmoderated Usability Testing

• Participants were allowed to continue until they clicked “Found it” or “I can’t find it.”

11

Tasks - SummarizedTask Purpose

T1 -Overdraft

How much is an overdraft if you have linked your checking account to your savings account?

Impact of mega menu (overdraft link is furthest out to the right)

T2 -Locations

How far away is the closest Wells Fargo branch?

Impact of utility navigation (ATM/Locations link is at the top of all pages)

T3 - Interest rate

What is the interest rate for a Way2Save savings account?

Primary navigation + in-page link navigation

T4 -Insurance

What is covered if you purchase insurance for your apartment?

Primary navigation + in-page tab navigation

T5 - ATM fee What is the ATM fee for a withdrawal in Mexico?

Multiple distracting starting points

T6 -Internship

What majors are desired if you want to apply for an internship in the audit division?

Impact of footer navigation (Careers link at the bottom of the page)

T7 -Retirement

What is the recommended amount you should be placing in your 401k if you are 22 years old?

Impact of separation in the primary navigation

12

Tasks 2

• Abbreviated task: How far away is the closest Wells Fargo branch?

13

Tasks 7

• Abbreviated task: What is the recommended amount you should be placing in your 401k if you are 22 years old?

Prototypes

15




Tree Testing

Prototypes

16

High Fidelity Prototypes

Remote Unmoderated Usability Testing

• Participants were allowed to continue until they clicked “Found it” or “I can’t find it.”

• Total of 64 pages + tab functionality on some pages

• No pages for the Small Business and Commercial areas of the site

17

Phrasing of Tasks was Exactly the Same

Live Site You would like to know more about the overdraft charge for a checking account that is linked to your savings account. Please navigate to where you expect to find this information, and make note of the charge for all overdrafts occurring on the same day, if you link a savings account to your checking account.

High Fidelity Prototype

You would like to know more about the overdraft charge for a checking account that is linked to your savings account. Please navigate to where you expect to find this information, and make note of the charge for all overdrafts occurring on the same day, if you link a savings account to your checking account.

18

Level of Abstraction

• The live site is the reality.

• High fidelity prototypes were very close to the real live site.

AbstractReality

Live SiteHigh Fidelity

Prototype

A Couple Things you Should Know about Statistics

20

p-Values

• Provide insight of how likely the result is due to chance • P<0.05 is considered statistically significant• Example: We believe the GRE can predict student class performance.

– p-value turns out to be 0.001 which is less than 0.05 -> it is very unlikely the result was due to chance -> Schools can continue using GRE as a means of evaluating students

• Does not say how much better a student with a high GRE score will perform

21

Pearson Correlation (r-value)

• Explains how well one variable correlates to another

22

Pearson Correlation (r-value)

• The correlation (r-value) is not the slope

Prototype- Results

24

Success rate is excellent at predicting success rate

• Used prototype success rates to predict live site success rates• Pearson Correlation= .961, p=.001• Low prototype success rates can potentially understate the live site success

y = 0.8261x + 0.1499

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

0.00% 20.00% 40.00% 60.00% 80.00% 100.00%

Live

sit

e s

ucc

ess

rate

Prototype success rate

PrototypePrototypesuccess rate

Live sitepredicted success rate

20% 32%40% 48%

60% 65%80% 81%

Forecasting

25

y = -135.59x + 188.15

0

20

40

60

80

100

120

140

160

180

200

0 0.2 0.4 0.6 0.8 1

Live

sit

e -

Mea

n t

ime


PrototypePredictor of time

PrototypeSuccess rate

Live sitetime (S)

20% 161

40% 13460% 107

80% 80

Forecasting

Success rate is good at predicting time

• Used prototype success rate to predict live site completion time• Pearson Correlation= -.807, p=.028 • Higher success rate results in shorter time

26

y = -6.1403x + 8.0276

0

1

2

3

4

5

6

7

8

9

0 0.2 0.4 0.6 0.8 1

Live

sit

e -

Me

an u

niq

ue

pag

es


PrototypePredictor of Unique Pages


Live sitePages

20% 6.80

40% 5.5760% 4.34

80% 3.12

Forecasting

Success rate is good at predicting unique pages

• Used prototype success rate to predict live site unique page views• Pearson Correlation= -.833, p=.020 • Higher success rate leads to fewer page views

27

Predictor of Clicks


Live siteClicks

20% 13.67

40% 11.6660% 9.64

80% 7.63

Forecasting

y = -10.055x + 15.677

0

2

4

6

8

10

12

14

16

0 0.2 0.4 0.6 0.8 1

Live

sit

e -

Mea

n c

licks


Prototype

Success rate is good at predicting clicks

• Used prototype success rate to predict live site number of clicks• Pearson Correlation= -.796, p=.032• Higher success leads to fewer clicks

28

y = 0.7579x + 10.817

0

20

40

60

80

100

120

140

160

180

200

0 50 100 150 200 250

Live

sit

e M

ean

Tim

e

Prototype Mean Time

TimePrediction of Time (s)

Prototype Live site

80.00 71 120.00 102

160.00 132 200.00 162

Forecasting

Time is excellent at predicting time

• Used prototype ‘completion time’ to predict live site ‘completion time’• Pearson Correlation=0.960, p=.001• The prototypes tend to overstate the time a slight bit compared to the live

site

29

y = 1.0069x + 0.1719

0

1

2

3

4

5

6

7

8

9

0 2 4 6 8

Live

sit

e M

ean

Un

iqu

e P

age

s

Prototype Mean Unique Pages

Unique Pages Forecasting

Unique pages is excellent at predicting unique pages

• Used prototype unique page views to predict live site unique page views• Pearson Correlation=0.943, p=.001• Prototype is slightly understating the number of pages

Prediction of Unique Pages

Prototype Live site3.00 3.19

4.00 4.20 5.00 5.21 6.00 6.21 7.00 7.22

30

y = 0.7518x + 4.0282

0

2

4

6

8

10

12

14

16

0 5 10 15 20

Live

sit

e N

um

be

r o

f C

licks

Prototype Number of Clicks

ClicksPrediction of Clicks

Prototype Live site

6.00 8.54 8.00 10.04

10.00 11.55 12.00 13.05

14.00 14.55

Forecasting

Number of clicks is good at predicting number of clicks

• Used prototype number of clicks to predict live site number of clicks• Pearson Correlation=0.814, p=.026• Number of clicks is understated in the prototypes for lower values

31

Quick Summary Prototypes

Live Site Prototypes-Success

Prototypes-Time

Prototypes-Pages

Prototypes-Clicks

Success ExcellentCorrelation = 0.961p = 0.001

Time GoodCorrelation = -0.807p = 0.028

ExcellentCorrelation = 0.960p = 0.001

Unique Pages GoodCorrelation = -0.833p = 0.020

ExcellentCorrelation = 0.943p = 0.001

Clicks GoodCorrelation = -0.796p = 0.032

GoodCorrelation = 0.814p = 0.026

32

Questions

Prototype test

• You have created high fidelity prototypes and conducted an unmoderated usability test to gain insight about the navigation. The outcome was primarily positive but there were a few minor concerns. The project is on a tight budget and the product owner wonders if the outcome would have been the same if the test was conducted on the final live design. What is your answer?

Answer

• We can be very confident the result would have been the same if the test was conducted on the final live design.

Excellent

Screenshot Click Test

34



Prototypes

Tree Testing

35

Screenshot Click Test

First click

• Screenshot of home page

• Personal tab selected

• Removed search box, customer service link at the top of the page, and login box

36

Phrasing of tasks was slightly different

Live Site You would like to know more about the overdraft charge for a checking account that is linked to your savings account. Please navigate to where you expect to find this information, and make note of the charge for all overdrafts occurring on the same day, if you link a savings account to your checking account

Screenshotclick test

You would like to know more about the overdraft fees for a checking account that is linked to your savings account. Please click where you would go to find this information.

37


• The live site is the reality

• Screenshot click test is a bit more abstract than prototypes, but still very close to the real live site

AbstractReality


PrototypeScreenshot Click Testing

Screenshot Click Test - Results

39

y = 1.0454x + 0.1555

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

0.00% 20.00% 40.00% 60.00% 80.00% 100.00%

Live

sit

e s

ucc

ess

rate

First click success rate

Screenshot click test Forecasting

Success rate is excellent at predicting success rate

• Used screenshot click test success rate to predict live site success rate• Pearson Correlation= .910, p=.004• Screenshot click testing tends to underestimate the success of the final site

First click success rate


20% 36%

40% 57%60% 78%

40

y = -169.3x + 186.35

0

20

40

60

80

100

120

140

160

180

200

0 0.2 0.4 0.6 0.8 1

Live

sit

e -

Mea

n t

ime

First click test success rate

First click test Forecasting


• Used screenshot click test success rate to predict live site completion time• Pearson Correlation= -.754, p=.050 • Higher success rate leads to shorter time

Predictor of Time

First clickSuccess rate

Live siteTime (S)

20% 152

40% 11860% 85

41

Predictor of Unique Pages


Live sitePages

20% 6.69

40% 4.8660% 3.04

y = -9.1173x + 8.5114

0

1

2

3

4

5

6

7

8

9

0 0.2 0.4 0.6 0.8 1

Live

sit

e -

Me

an u

niq

ue

pag

es


First click test Forecasting

Success rate is excellent at predicting unique pages

• Used screenshot click test success rate to predict live site unique page views• Pearson Correlation= -.926, p=.003 • Higher success rate leads to fewer unique pages visited

42

y = -12.438x + 15.498

0

2

4

6

8

10

12

14

16

0 0.2 0.4 0.6 0.8 1

Live

sit

e -

Me

an c

licks


First click testPredictor of Clicks


Live siteClicks

20% 13.0104

40% 10.522860% 8.0352

Forecasting

Success rate is most likely good at predicting number of clicks

• Used screenshot click test success rate to predict live site number of clicks• Pearson Correlation= -.737, p=.059 • Higher success rate leads to fewer clicks

43

Quick Summary Screenshot Click Test

Live Site Screenshot click testing - Success

Screenshot click testing - Time

Screenshot click testing - Pages

Screenshot click testing - Clicks

Success ExcellentCorrelation = 0.910p = 0.004


Unique Pages GoodCorrelation = -0.926p = 0.003

Clicks GoodCorrelation = -0.737p = 0.059

44

Questions

Screenshot click test

• You conducted a screenshot click test and the result was a success rate of 60%. You present this to the product owner who ask you if this should be a concern. What is your answer?

Answer

• You should not be concerned since this translates into a live site success rate of 78% with an average completion time of 85 seconds/task.

Very Good

Tree Test

46



Prototypes


47

Tree Test

Tree Testing

• Reverse engineered the primary navigation (no utility or footer navigation)

• Included some in-page navigation as well (tabs on the page, accounts overview)

• Only the left side of the mega menu was included

ATM Services

48

Phrasing of tasks was slightly different

Live Site You would like to know more about the overdraft charge for a checking account that is linked to your savings account. Please navigate to where you expect to find this information, and make note of the charge for all overdrafts occurring on the same day, if you link a savings account to your checking account

Tree Test You would like to know more about the overdraft charges for a checking account that is linked to your savings account. Please select where you expect to find this information.

49


• The live site is the reality

• Tree testing is more abstract than real and differs more from the other methods

AbstractReality


PrototypeScreenshotClick Testing

Tree Testing

Tree Test - Results

51

y = 0.265x + 0.4395

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

0.00% 20.00% 40.00% 60.00% 80.00% 100.00%

Live

sit

e s

ucc

ess

rate

Prototype Success rate

Tree testTree testsuccess rate


20% 49%40% 55%

60% 60%80% 65%

Forecasting

Success rate is poor at predicting success rate

• Used tree test success rate to predict live site success rate• Pearson Correlation= .302, p=.510 • Cannot make a conclusion due to the high p-value

– (Regardless of high (80% or low (20%) success rates on the tree test, the prediction of the live site performance is only 49%-65%).

52


• Used tree test success rate to predict live site completion time• Pearson Correlation= -.756, p=.049 • Higher success leads to shorter time

y = -129.39x + 180.68

0

20

40

60

80

100

120

140

160

180

200

0 0.2 0.4 0.6 0.8 1

Live

sit

e -

Mea

n t

ime

Tree test success rate

Tree testPredictor of Time

Tree TestSuccess rate

Live siteTime (s)

20% 15540% 12960% 10380% 77

Forecasting

53

y = -4.4077x + 7.0123

0

1

2

3

4

5

6

7

8

9

0 0.2 0.4 0.6 0.8 1

Live

sit

e -

Me

an u

niq

ue

pag

es


Tree test

Success rate is most likely fair at predicting unique pages

• Used tree test success rate to predict live site unique pages visited• Pearson Correlation=- .587, p=.166• The p-value is too high to make a concrete conclusion

Predictor of Unique PagesTree TestSuccess rate

Live sitePages

20% 6.1340% 5.2560% 4.3780% 3.49

Forecasting

54

y = -8.8411x + 14.772

0

2

4

6

8

10

12

14

16

0 0.2 0.4 0.6 0.8 1

Live

sit

e -

Me

an c

licks


Tree test

Predictor of ClicksTree TestSuccess rate

Live siteClicks

20% 13.0040% 11.24

60% 9.4780% 7.70

Forecasting

Success rate is most likely good at predicting clicks

• Used tree test success rate to predict live site number of clicks• Pearson Correlation=- .687, p=.088• Higher success rate may lead to fewer clicks

55

y = 1.4349x + 54.339

0

20

40

60

80

100

120

140

160

180

200

0 20 40 60 80

Live

sit

e M

ean

Tim

e

Tree Test Mean Time

Live Site Time dependent on Tree Test Time

Completion time is most likely a fair predictor of completion time

• Used tree test completion time to predict live site completion time• Pearson Correlation=0.611, p=.145• The p-value is too high to make a concrete conclusion

Prediction of Time

Tree Test (Time) Live site (Time)30.00 97

40.00 112 50.00 126 60.00 140 70.00 155

Forecasting

56

Quick Summary Tree Test

Live Site Tree testing -Success

Tree testing -Time

Tree testing -Pages

Tree testing -Clicks

Success PoorCorrelation = 0.302p = 0.510


FairCorrelation =0.611p=0.145

Unique Pages FairCorrelation = -0.587p = 0.166

Clicks FairCorrelation = -0.687p = 0.088

57

Questions

Tree Test

• You have conducted a tree test and made significant enhancements to the IA (The final version has an average success rate of 85%). You are now in a meeting and the group is considering if they need to do any more IA research or if that would be a waste of resources since the IA performed well. What is your answer?

Answer

• You should do more rounds of IA/navigation testing later on in the design process since there are many other factors that influence final live performance.

Fair

Summary

59

Summary Overview


Excellent

Very good

Fair

Prototypes

60

Summary Overview

Prototypes Screenshot Tree test

Live Site Success Time Pages Clicks Success Success Time

Success Excellent Excellent Poor

Time Good Excellent Good Good Fair

UniquePages

Good Excellent Good Fair

Clicks Good Good Good Fair

Translation into Practice

62

Prototypes

Keep in mind• Used high fidelity prototypes -> Excellent predictor• Don’t know how low fidelity prototypes would perform• Provides good feedback to the participants• Testing late in the design process

Put into practice • High fidelity prototypes performed closely with the live site

– Use high fidelity prototypes to discover usability issues prior to development (especially if it requires a lot of development time)

• Prototypes take longer to create than the other testing techniques – Competitive testing becomes more expensive– If there is uncertainty regarding the performance of the information architecture, it can

become expensive to make multiple rebuilds of the prototypes

63

Screenshot Click Testing

Keep in mind• Used high graphic design screenshot -> Very good predictor • Don’t know how low graphic design screenshot would perform• Lack of feedback to the participants

Put into practice • Screenshot testing can be conducted as soon as there are some screenshot

designs– Can be used when evaluating different design ideas

• Very useful if designs/screenshots already exist– Inexpensive to test – Provides a very good prediction of live site performance

• Don’t skip this – It is cheap and useful

64

Tree Testing

Keep in mind• Early testing method -> less accurate prediction• Many other features influence performance (utility navigation, content, etc. )• This research did not only include tasks focusing on the primary navigation

– Task selection most likely impacted the success

• Lack of feedback to the participants

Put into practice • Test structural concepts of the information architecture

– Great for competitive testing

• Make sure to retest information architecture performance later on as well– Tree tests are not as good predictors as other forms or testing and many other factors

influence final performance

• Don’t skip this– Can be expensive to rebuild prototypes multiple times and even more expensive to rebuild

the live site

Thank You!

David JuhlinUser Experience ConsultantBentley University’s User [email protected]

Bill AlbertExecutive DirectorBentley University’s User Experience [email protected]

Dani NordinSenior UX DesignerHarvard Business Review [email protected]

Connect with us and stay up to date:

Website: www.Bentley.edu/uxc

Twitter: @BentleyUXC

LinkedIn: Join our LinkedIn Group (Bentley UXC)

Documents

How do different early evaluation tools predict final site ... › sites › ... · there were a few minor concerns. The project is on a tight budget and the product owner wonders