How do different early evaluation tools predict final site navigation?In collaboration with Userzoom
David Juhlin | [email protected]
781.891.3142 | bentley.edu/uxc
David JuhlinBill AlbertDani Nordin
July 8, 2015
Table of Contents
Topic Slide
1. Introduction 3
2. Live site base line 9
3. Prototypes 14
4. Screenshot click testing 33
5. Tree testing 45
6. Summary 58
7. Translation into practice 61
Introduction
4
Tools to Investigate
Prototypes
Screenshot click testing
Tree testing
Live site (Benchmark)
?
?
?
5
Questions
Prototype test
• You have created high fidelity prototypes and conducted an unmoderated usability test to gain insight about the navigation. The outcome was primarily positive but there were a few minor concerns. The project is on a tight budget and the product owner wonders if the outcome would have been the same if the test was conducted on the final live design. What is your answer?
?
6
Questions
Screenshot click test
• You conducted a screenshot click test and the result was a success rate of 60%. You present this to the product owner who asks you if this should be a concern. What is your answer?
?
7
Questions
TreeTest
• You have conducted a tree test and made significant enhancements to the IA (the final version has an average success rate of 85%). You are now in a meeting and the group is considering if they need to do any more IA research or if that would be a waste of resources since the IA performed well. What is your answer?
?
8
Data Collection
Website Tested
Wells Fargo’s live website was used to establish the live site’s performance and as a benchmark against the other tools. The artifacts used for the early research tools were reverse engineered from the live website into prototypes and artifacts for screenshot click tests and tree tests.
Participants and Design of the Research Study
Participants were recruited through Amazon’s Mechanical Turk and a between subjects design was used. Every participant performed 7 tasks, which were designed to cover different features of the site and were minimally tweaked to suit the different types of research methods. The sample size for each research method was between 99-195 participants.
Live Site (Benchmark)
10
Live site
Remote Unmoderated Usability Testing
• Participants were allowed to continue until they clicked “Found it” or “I can’t find it.”
11
Tasks - SummarizedTask Purpose
T1 -Overdraft
How much is an overdraft if you have linked your checking account to your savings account?
Impact of mega menu (overdraft link is furthest out to the right)
T2 -Locations
How far away is the closest Wells Fargo branch?
Impact of utility navigation (ATM/Locations link is at the top of all pages)
T3 - Interest rate
What is the interest rate for a Way2Save savings account?
Primary navigation + in-page link navigation
T4 -Insurance
What is covered if you purchase insurance for your apartment?
Primary navigation + in-page tab navigation
T5 - ATM fee What is the ATM fee for a withdrawal in Mexico?
Multiple distracting starting points
T6 -Internship
What majors are desired if you want to apply for an internship in the audit division?
Impact of footer navigation (Careers link at the bottom of the page)
T7 -Retirement
What is the recommended amount you should be placing in your 401k if you are 22 years old?
Impact of separation in the primary navigation
12
Tasks 2
• Abbreviated task: How far away is the closest Wells Fargo branch?
13
Tasks 7
• Abbreviated task: What is the recommended amount you should be placing in your 401k if you are 22 years old?
Prototypes
15
Tools to Investigate
Live site (Benchmark)
Screenshot click testing
Tree Testing
Prototypes
16
High Fidelity Prototypes
Remote Unmoderated Usability Testing
• Participants were allowed to continue until they clicked “Found it” or “I can’t find it.”
• Total of 64 pages + tab functionality on some pages
• No pages for the Small Business and Commercial areas of the site
17
Phrasing of Tasks was Exactly the Same
Live Site You would like to know more about the overdraft charge for a checking account that is linked to your savings account. Please navigate to where you expect to find this information, and make note of the charge for all overdrafts occurring on the same day, if you link a savings account to your checking account.
High Fidelity Prototype
You would like to know more about the overdraft charge for a checking account that is linked to your savings account. Please navigate to where you expect to find this information, and make note of the charge for all overdrafts occurring on the same day, if you link a savings account to your checking account.
18
Level of Abstraction
• The live site is the reality.
• High fidelity prototypes were very close to the real live site.
AbstractReality
Live SiteHigh Fidelity
Prototype
A Couple Things you Should Know about Statistics
20
p-Values
• Provide insight of how likely the result is due to chance • P<0.05 is considered statistically significant• Example: We believe the GRE can predict student class performance.
– p-value turns out to be 0.001 which is less than 0.05 -> it is very unlikely the result was due to chance -> Schools can continue using GRE as a means of evaluating students
• Does not say how much better a student with a high GRE score will perform
21
Pearson Correlation (r-value)
• Explains how well one variable correlates to another
22
Pearson Correlation (r-value)
• The correlation (r-value) is not the slope
Prototype- Results
24
Success rate is excellent at predicting success rate
• Used prototype success rates to predict live site success rates• Pearson Correlation= .961, p=.001• Low prototype success rates can potentially understate the live site success
y = 0.8261x + 0.1499
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
0.00% 20.00% 40.00% 60.00% 80.00% 100.00%
Live
sit
e s
ucc
ess
rate
Prototype success rate
PrototypePrototypesuccess rate
Live sitepredicted success rate
20% 32%40% 48%
60% 65%80% 81%
Forecasting
25
y = -135.59x + 188.15
0
20
40
60
80
100
120
140
160
180
200
0 0.2 0.4 0.6 0.8 1
Live
sit
e -
Mea
n t
ime
Prototype success rate
PrototypePredictor of time
PrototypeSuccess rate
Live sitetime (S)
20% 161
40% 13460% 107
80% 80
Forecasting
Success rate is good at predicting time
• Used prototype success rate to predict live site completion time• Pearson Correlation= -.807, p=.028 • Higher success rate results in shorter time
26
y = -6.1403x + 8.0276
0
1
2
3
4
5
6
7
8
9
0 0.2 0.4 0.6 0.8 1
Live
sit
e -
Me
an u
niq
ue
pag
es
Prototype success rate
PrototypePredictor of Unique Pages
PrototypeSuccess rate
Live sitePages
20% 6.80
40% 5.5760% 4.34
80% 3.12
Forecasting
Success rate is good at predicting unique pages
• Used prototype success rate to predict live site unique page views• Pearson Correlation= -.833, p=.020 • Higher success rate leads to fewer page views
27
Predictor of Clicks
PrototypeSuccess rate
Live siteClicks
20% 13.67
40% 11.6660% 9.64
80% 7.63
Forecasting
y = -10.055x + 15.677
0
2
4
6
8
10
12
14
16
0 0.2 0.4 0.6 0.8 1
Live
sit
e -
Mea
n c
licks
Prototype success rate
Prototype
Success rate is good at predicting clicks
• Used prototype success rate to predict live site number of clicks• Pearson Correlation= -.796, p=.032• Higher success leads to fewer clicks
28
y = 0.7579x + 10.817
0
20
40
60
80
100
120
140
160
180
200
0 50 100 150 200 250
Live
sit
e M
ean
Tim
e
Prototype Mean Time
TimePrediction of Time (s)
Prototype Live site
80.00 71 120.00 102
160.00 132 200.00 162
Forecasting
Time is excellent at predicting time
• Used prototype ‘completion time’ to predict live site ‘completion time’• Pearson Correlation=0.960, p=.001• The prototypes tend to overstate the time a slight bit compared to the live
site
29
y = 1.0069x + 0.1719
0
1
2
3
4
5
6
7
8
9
0 2 4 6 8
Live
sit
e M
ean
Un
iqu
e P
age
s
Prototype Mean Unique Pages
Unique Pages Forecasting
Unique pages is excellent at predicting unique pages
• Used prototype unique page views to predict live site unique page views• Pearson Correlation=0.943, p=.001• Prototype is slightly understating the number of pages
Prediction of Unique Pages
Prototype Live site3.00 3.19
4.00 4.20 5.00 5.21 6.00 6.21 7.00 7.22
30
y = 0.7518x + 4.0282
0
2
4
6
8
10
12
14
16
0 5 10 15 20
Live
sit
e N
um
be
r o
f C
licks
Prototype Number of Clicks
ClicksPrediction of Clicks
Prototype Live site
6.00 8.54 8.00 10.04
10.00 11.55 12.00 13.05
14.00 14.55
Forecasting
Number of clicks is good at predicting number of clicks
• Used prototype number of clicks to predict live site number of clicks• Pearson Correlation=0.814, p=.026• Number of clicks is understated in the prototypes for lower values
31
Quick Summary Prototypes
Live Site Prototypes-Success
Prototypes-Time
Prototypes-Pages
Prototypes-Clicks
Success ExcellentCorrelation = 0.961p = 0.001
Time GoodCorrelation = -0.807p = 0.028
ExcellentCorrelation = 0.960p = 0.001
Unique Pages GoodCorrelation = -0.833p = 0.020
ExcellentCorrelation = 0.943p = 0.001
Clicks GoodCorrelation = -0.796p = 0.032
GoodCorrelation = 0.814p = 0.026
32
Questions
Prototype test
• You have created high fidelity prototypes and conducted an unmoderated usability test to gain insight about the navigation. The outcome was primarily positive but there were a few minor concerns. The project is on a tight budget and the product owner wonders if the outcome would have been the same if the test was conducted on the final live design. What is your answer?
Answer
• We can be very confident the result would have been the same if the test was conducted on the final live design.
Excellent
Screenshot Click Test
34
Tools to Investigate
Live site (Benchmark)
Prototypes
Tree Testing
35
Screenshot Click Test
First click
• Screenshot of home page
• Personal tab selected
• Removed search box, customer service link at the top of the page, and login box
36
Phrasing of tasks was slightly different
Live Site You would like to know more about the overdraft charge for a checking account that is linked to your savings account. Please navigate to where you expect to find this information, and make note of the charge for all overdrafts occurring on the same day, if you link a savings account to your checking account
Screenshotclick test
You would like to know more about the overdraft fees for a checking account that is linked to your savings account. Please click where you would go to find this information.
37
Level of Abstraction
• The live site is the reality
• Screenshot click test is a bit more abstract than prototypes, but still very close to the real live site
AbstractReality
Live SiteHigh Fidelity
PrototypeScreenshot Click Testing
Screenshot Click Test - Results
39
y = 1.0454x + 0.1555
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
0.00% 20.00% 40.00% 60.00% 80.00% 100.00%
Live
sit
e s
ucc
ess
rate
First click success rate
Screenshot click test Forecasting
Success rate is excellent at predicting success rate
• Used screenshot click test success rate to predict live site success rate• Pearson Correlation= .910, p=.004• Screenshot click testing tends to underestimate the success of the final site
First click success rate
Live sitepredicted success rate
20% 36%
40% 57%60% 78%
40
y = -169.3x + 186.35
0
20
40
60
80
100
120
140
160
180
200
0 0.2 0.4 0.6 0.8 1
Live
sit
e -
Mea
n t
ime
First click test success rate
First click test Forecasting
Success rate is good at predicting time
• Used screenshot click test success rate to predict live site completion time• Pearson Correlation= -.754, p=.050 • Higher success rate leads to shorter time
Predictor of Time
First clickSuccess rate
Live siteTime (S)
20% 152
40% 11860% 85
41
Predictor of Unique Pages
First clickSuccess rate
Live sitePages
20% 6.69
40% 4.8660% 3.04
y = -9.1173x + 8.5114
0
1
2
3
4
5
6
7
8
9
0 0.2 0.4 0.6 0.8 1
Live
sit
e -
Me
an u
niq
ue
pag
es
First click test success rate
First click test Forecasting
Success rate is excellent at predicting unique pages
• Used screenshot click test success rate to predict live site unique page views• Pearson Correlation= -.926, p=.003 • Higher success rate leads to fewer unique pages visited
42
y = -12.438x + 15.498
0
2
4
6
8
10
12
14
16
0 0.2 0.4 0.6 0.8 1
Live
sit
e -
Me
an c
licks
First click test success rate
First click testPredictor of Clicks
First clickSuccess rate
Live siteClicks
20% 13.0104
40% 10.522860% 8.0352
Forecasting
Success rate is most likely good at predicting number of clicks
• Used screenshot click test success rate to predict live site number of clicks• Pearson Correlation= -.737, p=.059 • Higher success rate leads to fewer clicks
43
Quick Summary Screenshot Click Test
Live Site Screenshot click testing - Success
Screenshot click testing - Time
Screenshot click testing - Pages
Screenshot click testing - Clicks
Success ExcellentCorrelation = 0.910p = 0.004
Time GoodCorrelation = -0.754p = 0.050
Unique Pages GoodCorrelation = -0.926p = 0.003
Clicks GoodCorrelation = -0.737p = 0.059
44
Questions
Screenshot click test
• You conducted a screenshot click test and the result was a success rate of 60%. You present this to the product owner who ask you if this should be a concern. What is your answer?
Answer
• You should not be concerned since this translates into a live site success rate of 78% with an average completion time of 85 seconds/task.
Very Good
Tree Test
46
Tools to Investigate
Live site (Benchmark)
Prototypes
Screenshot click testing
47
Tree Test
Tree Testing
• Reverse engineered the primary navigation (no utility or footer navigation)
• Included some in-page navigation as well (tabs on the page, accounts overview)
• Only the left side of the mega menu was included
ATM Services
48
Phrasing of tasks was slightly different
Live Site You would like to know more about the overdraft charge for a checking account that is linked to your savings account. Please navigate to where you expect to find this information, and make note of the charge for all overdrafts occurring on the same day, if you link a savings account to your checking account
Tree Test You would like to know more about the overdraft charges for a checking account that is linked to your savings account. Please select where you expect to find this information.
49
Level of Abstraction
• The live site is the reality
• Tree testing is more abstract than real and differs more from the other methods
AbstractReality
Live SiteHigh Fidelity
PrototypeScreenshotClick Testing
Tree Testing
Tree Test - Results
51
y = 0.265x + 0.4395
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
0.00% 20.00% 40.00% 60.00% 80.00% 100.00%
Live
sit
e s
ucc
ess
rate
Prototype Success rate
Tree testTree testsuccess rate
Live sitepredicted success rate
20% 49%40% 55%
60% 60%80% 65%
Forecasting
Success rate is poor at predicting success rate
• Used tree test success rate to predict live site success rate• Pearson Correlation= .302, p=.510 • Cannot make a conclusion due to the high p-value
– (Regardless of high (80% or low (20%) success rates on the tree test, the prediction of the live site performance is only 49%-65%).
52
Success rate is good at predicting time
• Used tree test success rate to predict live site completion time• Pearson Correlation= -.756, p=.049 • Higher success leads to shorter time
y = -129.39x + 180.68
0
20
40
60
80
100
120
140
160
180
200
0 0.2 0.4 0.6 0.8 1
Live
sit
e -
Mea
n t
ime
Tree test success rate
Tree testPredictor of Time
Tree TestSuccess rate
Live siteTime (s)
20% 15540% 12960% 10380% 77
Forecasting
53
y = -4.4077x + 7.0123
0
1
2
3
4
5
6
7
8
9
0 0.2 0.4 0.6 0.8 1
Live
sit
e -
Me
an u
niq
ue
pag
es
Tree test success rate
Tree test
Success rate is most likely fair at predicting unique pages
• Used tree test success rate to predict live site unique pages visited• Pearson Correlation=- .587, p=.166• The p-value is too high to make a concrete conclusion
Predictor of Unique PagesTree TestSuccess rate
Live sitePages
20% 6.1340% 5.2560% 4.3780% 3.49
Forecasting
54
y = -8.8411x + 14.772
0
2
4
6
8
10
12
14
16
0 0.2 0.4 0.6 0.8 1
Live
sit
e -
Me
an c
licks
Tree test success rate
Tree test
Predictor of ClicksTree TestSuccess rate
Live siteClicks
20% 13.0040% 11.24
60% 9.4780% 7.70
Forecasting
Success rate is most likely good at predicting clicks
• Used tree test success rate to predict live site number of clicks• Pearson Correlation=- .687, p=.088• Higher success rate may lead to fewer clicks
55
y = 1.4349x + 54.339
0
20
40
60
80
100
120
140
160
180
200
0 20 40 60 80
Live
sit
e M
ean
Tim
e
Tree Test Mean Time
Live Site Time dependent on Tree Test Time
Completion time is most likely a fair predictor of completion time
• Used tree test completion time to predict live site completion time• Pearson Correlation=0.611, p=.145• The p-value is too high to make a concrete conclusion
Prediction of Time
Tree Test (Time) Live site (Time)30.00 97
40.00 112 50.00 126 60.00 140 70.00 155
Forecasting
56
Quick Summary Tree Test
Live Site Tree testing -Success
Tree testing -Time
Tree testing -Pages
Tree testing -Clicks
Success PoorCorrelation = 0.302p = 0.510
Time GoodCorrelation = -0.756p = 0.049
FairCorrelation =0.611p=0.145
Unique Pages FairCorrelation = -0.587p = 0.166
Clicks FairCorrelation = -0.687p = 0.088
57
Questions
Tree Test
• You have conducted a tree test and made significant enhancements to the IA (The final version has an average success rate of 85%). You are now in a meeting and the group is considering if they need to do any more IA research or if that would be a waste of resources since the IA performed well. What is your answer?
Answer
• You should do more rounds of IA/navigation testing later on in the design process since there are many other factors that influence final live performance.
Fair
Summary
59
Summary Overview
Live site (Benchmark)
Excellent
Very good
Fair
Prototypes
60
Summary Overview
Prototypes Screenshot Tree test
Live Site Success Time Pages Clicks Success Success Time
Success Excellent Excellent Poor
Time Good Excellent Good Good Fair
UniquePages
Good Excellent Good Fair
Clicks Good Good Good Fair
Translation into Practice
62
Prototypes
Keep in mind• Used high fidelity prototypes -> Excellent predictor• Don’t know how low fidelity prototypes would perform• Provides good feedback to the participants• Testing late in the design process
Put into practice • High fidelity prototypes performed closely with the live site
– Use high fidelity prototypes to discover usability issues prior to development (especially if it requires a lot of development time)
• Prototypes take longer to create than the other testing techniques – Competitive testing becomes more expensive– If there is uncertainty regarding the performance of the information architecture, it can
become expensive to make multiple rebuilds of the prototypes
63
Screenshot Click Testing
Keep in mind• Used high graphic design screenshot -> Very good predictor • Don’t know how low graphic design screenshot would perform• Lack of feedback to the participants
Put into practice • Screenshot testing can be conducted as soon as there are some screenshot
designs– Can be used when evaluating different design ideas
• Very useful if designs/screenshots already exist– Inexpensive to test – Provides a very good prediction of live site performance
• Don’t skip this – It is cheap and useful
64
Tree Testing
Keep in mind• Early testing method -> less accurate prediction• Many other features influence performance (utility navigation, content, etc. )• This research did not only include tasks focusing on the primary navigation
– Task selection most likely impacted the success
• Lack of feedback to the participants
Put into practice • Test structural concepts of the information architecture
– Great for competitive testing
• Make sure to retest information architecture performance later on as well– Tree tests are not as good predictors as other forms or testing and many other factors
influence final performance
• Don’t skip this– Can be expensive to rebuild prototypes multiple times and even more expensive to rebuild
the live site
Thank You!
David JuhlinUser Experience ConsultantBentley University’s User [email protected]
Bill AlbertExecutive DirectorBentley University’s User Experience [email protected]
Dani NordinSenior UX DesignerHarvard Business Review [email protected]
Connect with us and stay up to date:
Website: www.Bentley.edu/uxc
Twitter: @BentleyUXC
LinkedIn: Join our LinkedIn Group (Bentley UXC)