How do different early evaluation tools predict final site ... › sites › ... · there were a...

How do different early evaluation tools predict final site navigation?In collaboration with Userzoom

David Juhlin | djuhlin@bentley.edu

781.891.3142 | bentley.edu/uxc

David JuhlinBill AlbertDani Nordin

July 8, 2015

Table of Contents

Topic Slide

1. Introduction 3

2. Live site base line 9

3. Prototypes 14

4. Screenshot click testing 33

5. Tree testing 45

6. Summary 58

7. Translation into practice 61

Introduction

Tools to Investigate

Prototypes

Screenshot click testing

Tree testing

Live site (Benchmark)

Questions

Prototype test

• You have created high fidelity prototypes and conducted an unmoderated usability test to gain insight about the navigation. The outcome was primarily positive but there were a few minor concerns. The project is on a tight budget and the product owner wonders if the outcome would have been the same if the test was conducted on the final live design. What is your answer?

Questions

Screenshot click test

• You conducted a screenshot click test and the result was a success rate of 60%. You present this to the product owner who asks you if this should be a concern. What is your answer?

Questions

TreeTest

• You have conducted a tree test and made significant enhancements to the IA (the final version has an average success rate of 85%). You are now in a meeting and the group is considering if they need to do any more IA research or if that would be a waste of resources since the IA performed well. What is your answer?

Data Collection

Website Tested

Wells Fargo’s live website was used to establish the live site’s performance and as a benchmark against the other tools. The artifacts used for the early research tools were reverse engineered from the live website into prototypes and artifacts for screenshot click tests and tree tests.

Participants and Design of the Research Study

Participants were recruited through Amazon’s Mechanical Turk and a between subjects design was used. Every participant performed 7 tasks, which were designed to cover different features of the site and were minimally tweaked to suit the different types of research methods. The sample size for each research method was between 99-195 participants.

Live Site (Benchmark)

Live site

Remote Unmoderated Usability Testing

• Participants were allowed to continue until they clicked “Found it” or “I can’t find it.”

Tasks - SummarizedTask Purpose

T1 -Overdraft

How much is an overdraft if you have linked your checking account to your savings account?

Impact of mega menu (overdraft link is furthest out to the right)

T2 -Locations

How far away is the closest Wells Fargo branch?

Impact of utility navigation (ATM/Locations link is at the top of all pages)

T3 - Interest rate

What is the interest rate for a Way2Save savings account?

Primary navigation + in-page link navigation

T4 -Insurance

What is covered if you purchase insurance for your apartment?

Primary navigation + in-page tab navigation

T5 - ATM fee What is the ATM fee for a withdrawal in Mexico?

Multiple distracting starting points

T6 -Internship

What majors are desired if you want to apply for an internship in the audit division?

Impact of footer navigation (Careers link at the bottom of the page)

T7 -Retirement

What is the recommended amount you should be placing in your 401k if you are 22 years old?

Impact of separation in the primary navigation

Tasks 2

• Abbreviated task: How far away is the closest Wells Fargo branch?

Tasks 7

• Abbreviated task: What is the recommended amount you should be placing in your 401k if you are 22 years old?

Prototypes

Tree Testing

Prototypes

High Fidelity Prototypes

Remote Unmoderated Usability Testing

• Participants were allowed to continue until they clicked “Found it” or “I can’t find it.”

• Total of 64 pages + tab functionality on some pages

• No pages for the Small Business and Commercial areas of the site

Phrasing of Tasks was Exactly the Same

Live Site You would like to know more about the overdraft charge for a checking account that is linked to your savings account. Please navigate to where you expect to find this information, and make note of the charge for all overdrafts occurring on the same day, if you link a savings account to your checking account.

High Fidelity Prototype

You would like to know more about the overdraft charge for a checking account that is linked to your savings account. Please navigate to where you expect to find this information, and make note of the charge for all overdrafts occurring on the same day, if you link a savings account to your checking account.

Level of Abstraction

• The live site is the reality.

• High fidelity prototypes were very close to the real live site.

AbstractReality

Live SiteHigh Fidelity

Prototype

A Couple Things you Should Know about Statistics

p-Values

• Provide insight of how likely the result is due to chance • P<0.05 is considered statistically significant• Example: We believe the GRE can predict student class performance.

– p-value turns out to be 0.001 which is less than 0.05 -> it is very unlikely the result was due to chance -> Schools can continue using GRE as a means of evaluating students

• Does not say how much better a student with a high GRE score will perform

Pearson Correlation (r-value)

• Explains how well one variable correlates to another

Pearson Correlation (r-value)

• The correlation (r-value) is not the slope

Prototype- Results

Success rate is excellent at predicting success rate

• Used prototype success rates to predict live site success rates• Pearson Correlation= .961, p=.001• Low prototype success rates can potentially understate the live site success

y = 0.8261x + 0.1499

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

0.00% 20.00% 40.00% 60.00% 80.00% 100.00%

Prototype success rate

PrototypePrototypesuccess rate

Live sitepredicted success rate

20% 32%40% 48%

60% 65%80% 81%

Forecasting

y = -135.59x + 188.15

0 0.2 0.4 0.6 0.8 1

PrototypePredictor of time

PrototypeSuccess rate

Live sitetime (S)

20% 161

40% 13460% 107

80% 80

Forecasting

Success rate is good at predicting time

• Used prototype success rate to predict live site completion time• Pearson Correlation= -.807, p=.028 • Higher success rate results in shorter time

y = -6.1403x + 8.0276

0 0.2 0.4 0.6 0.8 1

PrototypePredictor of Unique Pages

Live sitePages

20% 6.80

40% 5.5760% 4.34

80% 3.12

Forecasting

Success rate is good at predicting unique pages

• Used prototype success rate to predict live site unique page views• Pearson Correlation= -.833, p=.020 • Higher success rate leads to fewer page views

Predictor of Clicks

Live siteClicks

20% 13.67

40% 11.6660% 9.64

80% 7.63

Forecasting

y = -10.055x + 15.677

0 0.2 0.4 0.6 0.8 1

Prototype

Success rate is good at predicting clicks

• Used prototype success rate to predict live site number of clicks• Pearson Correlation= -.796, p=.032• Higher success leads to fewer clicks

y = 0.7579x + 10.817

0 50 100 150 200 250

Prototype Mean Time

TimePrediction of Time (s)

Prototype Live site

80.00 71 120.00 102

160.00 132 200.00 162

Forecasting

Time is excellent at predicting time

• Used prototype ‘completion time’ to predict live site ‘completion time’• Pearson Correlation=0.960, p=.001• The prototypes tend to overstate the time a slight bit compared to the live

y = 1.0069x + 0.1719

0 2 4 6 8

Prototype Mean Unique Pages

Unique Pages Forecasting

Unique pages is excellent at predicting unique pages

• Used prototype unique page views to predict live site unique page views• Pearson Correlation=0.943, p=.001• Prototype is slightly understating the number of pages

Prediction of Unique Pages

Prototype Live site3.00 3.19

4.00 4.20 5.00 5.21 6.00 6.21 7.00 7.22

y = 0.7518x + 4.0282

0 5 10 15 20

Prototype Number of Clicks

ClicksPrediction of Clicks

Prototype Live site

6.00 8.54 8.00 10.04

10.00 11.55 12.00 13.05

14.00 14.55

Forecasting

Number of clicks is good at predicting number of clicks

• Used prototype number of clicks to predict live site number of clicks• Pearson Correlation=0.814, p=.026• Number of clicks is understated in the prototypes for lower values

Quick Summary Prototypes

Live Site Prototypes-Success

Prototypes-Time

Prototypes-Pages

Prototypes-Clicks

Success ExcellentCorrelation = 0.961p = 0.001

Time GoodCorrelation = -0.807p = 0.028

ExcellentCorrelation = 0.960p = 0.001

Unique Pages GoodCorrelation = -0.833p = 0.020

ExcellentCorrelation = 0.943p = 0.001

Clicks GoodCorrelation = -0.796p = 0.032

GoodCorrelation = 0.814p = 0.026

Questions

Prototype test

• You have created high fidelity prototypes and conducted an unmoderated usability test to gain insight about the navigation. The outcome was primarily positive but there were a few minor concerns. The project is on a tight budget and the product owner wonders if the outcome would have been the same if the test was conducted on the final live design. What is your answer?

Answer

• We can be very confident the result would have been the same if the test was conducted on the final live design.

Excellent

Screenshot Click Test

Prototypes

Tree Testing

Screenshot Click Test

First click

• Screenshot of home page

• Personal tab selected

• Removed search box, customer service link at the top of the page, and login box

Phrasing of tasks was slightly different

Live Site You would like to know more about the overdraft charge for a checking account that is linked to your savings account. Please navigate to where you expect to find this information, and make note of the charge for all overdrafts occurring on the same day, if you link a savings account to your checking account

Screenshotclick test

You would like to know more about the overdraft fees for a checking account that is linked to your savings account. Please click where you would go to find this information.

• The live site is the reality

• Screenshot click test is a bit more abstract than prototypes, but still very close to the real live site

AbstractReality

PrototypeScreenshot Click Testing

Screenshot Click Test - Results

y = 1.0454x + 0.1555

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

0.00% 20.00% 40.00% 60.00% 80.00% 100.00%

First click success rate

Screenshot click test Forecasting

Success rate is excellent at predicting success rate

• Used screenshot click test success rate to predict live site success rate• Pearson Correlation= .910, p=.004• Screenshot click testing tends to underestimate the success of the final site

First click success rate

20% 36%

40% 57%60% 78%

y = -169.3x + 186.35

0 0.2 0.4 0.6 0.8 1

First click test success rate

First click test Forecasting

• Used screenshot click test success rate to predict live site completion time• Pearson Correlation= -.754, p=.050 • Higher success rate leads to shorter time

Predictor of Time

First clickSuccess rate

Live siteTime (S)

20% 152

40% 11860% 85

Predictor of Unique Pages

Live sitePages

20% 6.69

40% 4.8660% 3.04

y = -9.1173x + 8.5114

0 0.2 0.4 0.6 0.8 1

First click test Forecasting

Success rate is excellent at predicting unique pages

• Used screenshot click test success rate to predict live site unique page views• Pearson Correlation= -.926, p=.003 • Higher success rate leads to fewer unique pages visited

y = -12.438x + 15.498

0 0.2 0.4 0.6 0.8 1

First click testPredictor of Clicks

Live siteClicks

20% 13.0104

40% 10.522860% 8.0352

Forecasting

Success rate is most likely good at predicting number of clicks

• Used screenshot click test success rate to predict live site number of clicks• Pearson Correlation= -.737, p=.059 • Higher success rate leads to fewer clicks

Quick Summary Screenshot Click Test

Live Site Screenshot click testing - Success

Screenshot click testing - Time

Screenshot click testing - Pages

Screenshot click testing - Clicks

Success ExcellentCorrelation = 0.910p = 0.004

Unique Pages GoodCorrelation = -0.926p = 0.003

Clicks GoodCorrelation = -0.737p = 0.059

Questions

Screenshot click test

• You conducted a screenshot click test and the result was a success rate of 60%. You present this to the product owner who ask you if this should be a concern. What is your answer?

Answer

• You should not be concerned since this translates into a live site success rate of 78% with an average completion time of 85 seconds/task.

Very Good

Tree Test

Prototypes

Tree Test

Tree Testing

• Reverse engineered the primary navigation (no utility or footer navigation)

• Included some in-page navigation as well (tabs on the page, accounts overview)

• Only the left side of the mega menu was included

ATM Services

Phrasing of tasks was slightly different

Live Site You would like to know more about the overdraft charge for a checking account that is linked to your savings account. Please navigate to where you expect to find this information, and make note of the charge for all overdrafts occurring on the same day, if you link a savings account to your checking account

Tree Test You would like to know more about the overdraft charges for a checking account that is linked to your savings account. Please select where you expect to find this information.

• The live site is the reality

• Tree testing is more abstract than real and differs more from the other methods

AbstractReality

PrototypeScreenshotClick Testing

Tree Testing

Tree Test - Results

y = 0.265x + 0.4395

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

0.00% 20.00% 40.00% 60.00% 80.00% 100.00%

Prototype Success rate

Tree testTree testsuccess rate

20% 49%40% 55%

60% 60%80% 65%

Forecasting

Success rate is poor at predicting success rate

• Used tree test success rate to predict live site success rate• Pearson Correlation= .302, p=.510 • Cannot make a conclusion due to the high p-value

– (Regardless of high (80% or low (20%) success rates on the tree test, the prediction of the live site performance is only 49%-65%).

• Used tree test success rate to predict live site completion time• Pearson Correlation= -.756, p=.049 • Higher success leads to shorter time

y = -129.39x + 180.68

0 0.2 0.4 0.6 0.8 1

Tree test success rate

Tree testPredictor of Time

Tree TestSuccess rate

Live siteTime (s)

20% 15540% 12960% 10380% 77

Forecasting

y = -4.4077x + 7.0123

0 0.2 0.4 0.6 0.8 1

Tree test

Success rate is most likely fair at predicting unique pages

• Used tree test success rate to predict live site unique pages visited• Pearson Correlation=- .587, p=.166• The p-value is too high to make a concrete conclusion

Predictor of Unique PagesTree TestSuccess rate

Live sitePages

20% 6.1340% 5.2560% 4.3780% 3.49

Forecasting

y = -8.8411x + 14.772

0 0.2 0.4 0.6 0.8 1

Tree test

Predictor of ClicksTree TestSuccess rate

Live siteClicks

20% 13.0040% 11.24

60% 9.4780% 7.70

Forecasting

Success rate is most likely good at predicting clicks

• Used tree test success rate to predict live site number of clicks• Pearson Correlation=- .687, p=.088• Higher success rate may lead to fewer clicks

y = 1.4349x + 54.339

0 20 40 60 80

Tree Test Mean Time

Live Site Time dependent on Tree Test Time

Completion time is most likely a fair predictor of completion time

• Used tree test completion time to predict live site completion time• Pearson Correlation=0.611, p=.145• The p-value is too high to make a concrete conclusion

Prediction of Time

Tree Test (Time) Live site (Time)30.00 97

40.00 112 50.00 126 60.00 140 70.00 155

Forecasting

Quick Summary Tree Test

Live Site Tree testing -Success

Tree testing -Time

Tree testing -Pages

Tree testing -Clicks

Success PoorCorrelation = 0.302p = 0.510

FairCorrelation =0.611p=0.145

Unique Pages FairCorrelation = -0.587p = 0.166

Clicks FairCorrelation = -0.687p = 0.088

Questions

Tree Test

• You have conducted a tree test and made significant enhancements to the IA (The final version has an average success rate of 85%). You are now in a meeting and the group is considering if they need to do any more IA research or if that would be a waste of resources since the IA performed well. What is your answer?

Answer

• You should do more rounds of IA/navigation testing later on in the design process since there are many other factors that influence final live performance.

Summary

Summary Overview

Excellent

Very good

Prototypes

Summary Overview

Prototypes Screenshot Tree test

Live Site Success Time Pages Clicks Success Success Time

Success Excellent Excellent Poor

Time Good Excellent Good Good Fair

UniquePages

Good Excellent Good Fair

Clicks Good Good Good Fair

Translation into Practice

Prototypes

Keep in mind• Used high fidelity prototypes -> Excellent predictor• Don’t know how low fidelity prototypes would perform• Provides good feedback to the participants• Testing late in the design process

Put into practice • High fidelity prototypes performed closely with the live site

– Use high fidelity prototypes to discover usability issues prior to development (especially if it requires a lot of development time)

• Prototypes take longer to create than the other testing techniques – Competitive testing becomes more expensive– If there is uncertainty regarding the performance of the information architecture, it can

become expensive to make multiple rebuilds of the prototypes

Screenshot Click Testing

Keep in mind• Used high graphic design screenshot -> Very good predictor • Don’t know how low graphic design screenshot would perform• Lack of feedback to the participants

Put into practice • Screenshot testing can be conducted as soon as there are some screenshot

designs– Can be used when evaluating different design ideas

• Very useful if designs/screenshots already exist– Inexpensive to test – Provides a very good prediction of live site performance

• Don’t skip this – It is cheap and useful

Tree Testing

Keep in mind• Early testing method -> less accurate prediction• Many other features influence performance (utility navigation, content, etc. )• This research did not only include tasks focusing on the primary navigation

– Task selection most likely impacted the success

• Lack of feedback to the participants

Put into practice • Test structural concepts of the information architecture

– Great for competitive testing

• Make sure to retest information architecture performance later on as well– Tree tests are not as good predictors as other forms or testing and many other factors

influence final performance

• Don’t skip this– Can be expensive to rebuild prototypes multiple times and even more expensive to rebuild

the live site

Thank You!

David JuhlinUser Experience ConsultantBentley University’s User Experiencedjuhlin@bentley.edu781-891-3142

Bill AlbertExecutive DirectorBentley University’s User Experience Centerwalbert@bentley.edu781-891-2608

Dani NordinSenior UX DesignerHarvard Business Review GroupDanielle.nordin@hbr.org

Connect with us and stay up to date:

Website: www.Bentley.edu/uxc

Twitter: @BentleyUXC

LinkedIn: Join our LinkedIn Group (Bentley UXC)

How do different early evaluation tools predict final site ... › sites › ... · there were a...

Documents

Natural wonders

English Language Arts Wonders Unit 5: Wonders of Nature ... 5... · English Language Arts Wonders Unit 5: Wonders of Nature Grade K Long Beach Unified School District

Wildlife wonders

Water wonders

Winged Wonders

1 Nacht Weerwolven en waaghalzen5 Dice small 5211 Azul 7 Wonders 7 Wonders Armada 7 Wonders Cities 7 Wonders Duel 7 Wonders Duel Agora NL 7 Wonders Duel Pantheon 7 Wonders Leaders

Works Wonders

New Wonders

1:18 maisto chevy - Toy Wonders, Inc. - Toy Wonders, Inc

World wonders

My7 Wonders!

CISER Wonders

A method to predict the resistivity index for tight

City Wonders

Wine Wonders

Brain wonders

Geographical Wonders

Small Wonders

New Wonders

Natural Wonders and Fetures & Human Wonders