Lessons Learned From 1,001 Web Site Reviews...•Traffic drivers and analytics tools evaluate your conversion path – not your landing page •Most landing pages don’t do a good

www.forrester.com ©2007, Forrester Research, Inc.

1

Alan E. WebberSenior Analyst, Customer ExperienceForrester Research

Lessons Learned From 1,001 Web Site Reviews

2 Entire contents © 2007 Forrester Research, Inc. All rights reserved.

↑ Customer Experience =

↑ Sales


2


Customer experience

► An individual’s perception of your organization based on the collection of direct and indirect interactions

► How customers feel about you and your organization

∑n=1

(Pn - En)

Where:P = Perception of the nth interactionE = Expectation for the nth interaction

∞►


“Neo, sooner or later you're going to realize, just as I did, there's a difference between knowing the path and walking the path.”

- Morpheus (The Matrix)


3


Agenda

• What is a Web Site Review?

• What can we learn from data on 1,001 Web Site Reviews?

• What are the key insights from more than eight years of doing reviews?

• The future? Experience-based differentiation


Agenda






4


What is an expert review?

• Also known as heuristic review or scenario review

Trained experts

. . . attempt to achieve

user goals

. . . and evaluate the experience

based on criteria


Forrester’s Web Site Review process

1. Develop a description of the target users.

2. Identify specific goals that those users want to accomplish.

3. Analysts attempt to accomplish those goals.

4. Evaluate the experience across 25 criteria that span four categories: Value, Presentation, Navigation, and Trust.

5. Grade each criteria on a modified pass/fail scale:

• +2 (strong pass), +1 (pass), -1 (fail), -2 (strong fail)

(Read Forrester’s report “Best And Worst Of B2B Site Design, 2006”)


5


Forrester’s 25 Web Site Review criteria


Agenda






6


Distribution of Web Site Review scores, versions 2-6

Source: Forrester’s 1,001 Web Site Reviews, versions 2-6

Num

ber o

f site

s

Number of sites passed: 35


Distribution Of B2B Web Site Review Scores


7

© 2007, Forrester Research, Inc. Reproduction Prohibited

B2B Site Experience Lags B2C Site Experience

May 2007, “Lessons Learned From 1,001 Web Site Reviews”


Top 10 Usability Failures On B2B Web SitesJuly 2007 “The Three Keys To Improving B2B Site Experience”


8


Top 10 Usability Failures On B2B Web Sites (Cont.)July 2007 “The Three Keys To Improving B2B Site Experience”


Agenda






9


No. 1: Scenario Design remains the key ingredientOur advice since 1999: Always ask — and answer — three questions:

2) What aretheirgoals?

3) How can you help them accomplishtheir goals?

1) Whoare yourusers?


No. 2: Good personas fuel great results

Composite representation of a target audience based on primary user research

• A model of key user attributes and goals

• Distilled from observing real people

• Presented as a vivid, narrative description

• Of a single “person”who represents a customer segment

• Used to guide the design of products, channels, and messaging


10


No. 3: The ROI from a Web site redesign can be outrageous

Example for a manufacturer

Source: March 17, 2006, "The ROI Of Web Redesigns Made Simple”


No. 4: Online experiences need a cross-channelcontext

Results from 16 cross-channel reviews

Source: February 16, 2007, “Best And Worst Of Cross-Channel Design, 2007”

Source: December 18, 2006, "Best Practices In Multichannel Retailing"


11


No. 5: Excellent usability is necessary, but not sufficient

Source: November 20, 2006, “Humanizing The Digital Experience”


Agenda






12



Companies can compete based on customer

experience


13


Companies are already doing it


B2B companies are too


14


Customer experience needs to be embraced, not implemented

Treatcustomer

experience as a competence

Obsess about customer

needs

Reinforce brands

with every interaction

Experience-Based

Differentiation


Experience-Based Differentiation

Typical customer-centric activities

Experience-Based Differentiation requires a significant change

Translate into requirements for all interactions

Drive marketing communications

Use of brand attributes

Customer experience embedded in core strategy

Actively support customer experience efforts

Senior executive commitment

CulturalOrganizationalFocus of changes

CultivatedMandated Employee involvement

Combination of products, services, and digital

interfacesProduct featuresDevelopment of

offerings

How customer accomplishes goals

How company interacts with customer

Key customer insights


15


Principle No. 1: Obsess about customer needs

• Clearly identified target customer segments.

• Customer insights drawn from ethnographic and qualitative research.

• Customer needs examined across entire life cycle.

• Offerings defined across product and service boundaries.


Principle No. 2: Reinforce brands with every interaction, not just communications

• Brand is well defined and widely communicated as a set of promises to customers.

• Promises are translated into requirements for interactions.

• Efforts are focused on moments of truth.


16


Principle No. 3: Treat customer experience as a competence, not a function

• Leadership is involved, not just “bought-in.”

• Employees are engaged in, not mandated into, the process.

• Customer feedback drives continuous improvement efforts.


“B2B companies that expect to continue to differentiate purely on products are going the way of the dinosaur – soon to be extinct.”

CMO from Fortune 500 manufacturing firm


17


Summary

• The B2B experience is lagging.

• Don’t forget to continuously ask and answer three questions:

» Who are your users?

» What are their goals?

» How can you help them achieve those goals?

• Differentiate based on the customer experience


Alan E. Webber

+1 703.245.6657

[email protected]

www.forrester.com

Thank you


18


Selected bibliography

• May 11, 2007, “Lessons Learned From 1,001 Web Site Reviews”

• March 30, 2007, “Best And Worst Of B2C Site Design, 2007”

• March 17, 2006, “The ROI Of Web Redesigns Made Simple”

• May 6, 2005, “Executive Q&A: Web Site Reviews”

• July 19, 2004, “Scenario Design: A Disciplined Approach To Customer Experience”


Appendices


19


Find the Web Site Review tools on the Customer Experience site


Find the Web Site Review tools on the Customer Experience site


20


Download the Web Site Review scorecard and Reviewer’s Guide

Scorecard Reviewers Guide

INTERACTIVE CONVERSATION

Interactive ConversationsBreathing Life into Interactivity

A Web Analogy

2

3

SOURCE OF THE PROBLEMAn example from one of the best e-tailers on the planet…

4

5

6

7

8

9

SOURCE OF THE PROBLEM

A confusing, impersonal, unaware, unhelpful customer experience

Billions of dollars spent driving prospectsto marginally interactive brochures.

10

A NEW APPROACH…What if instead of a landing page

your customers were greeted by a helpful advisor?

Run this: www.jellyvision.com/kitchen/computersales

11

The Interactive Conversation Interface

• Creates interaction that feels like a real person is just behind the machine

• Incorporates seamless pacing and personality though writing, performance and audio sequencing (vs. artificial intelligence)

• A form of mass communication for any platform

ANOTHER EXAMPLE

12

iCi EMPATHY

13

ConnectEDUwww.connectedu.net

HOW TO DO ITThe Jack Principles of the Interactive Conversation Interface

• A set of design concepts to allow users to forget that they are interacting with a machine and feel like the character in the computer is really talking to them– Maintain Pacing– Create the Illusion of Awareness– Maintain the Illusion of Awareness

1

1Google Confidential and Proprietary

Making Your Web Assets Work Harder for You

Jennifer HowardHead of B2B Markets, Central RegionOctober 1, 2007


•Users move fast – 8 second rule

•Traffic drivers and analytics tools evaluate your conversion path – not your landing page

•Most landing pages don’t do a good job of selling.

•Increasing landing page effectiveness can halve your acquisitioncosts.

*MarketingSherpa, Landing Page Handbook, 2005

Website Optimization - Why is it important?

2


Website Optimizer Product Benefits

• Free multivariate and A/B split testing application

• Works with all traffic to your website (not just AdWords traffic)

• Marketers have full control over the testing and creatives

• Intuitive, self-service application does not require consulting or professional services to implement

• Integrated directly into AdWords

• Works with all site analytics solutions

• Provides detailed graphical reports


Multivariate: Test Multiple Sections and Combinations

3





2 different headlines

x 3 different images

= 6 possible combinations

4


A/B: Or test whole pages


Landing Page with 5% Conversion Rate

5


Landing Page with 15% Conversion Rate


Landing Page with 8.3% Conversion Rate

6


Landing Page with 17.9% Conversion Rate


Gadgets

7


Five Pearls of Wisdom

1. Gadget Ads Are Not Just Rich MediaPosition as an evolution towards the componentized and sharable web.

2. The Creative Is Everything

Performance will vary greatly based on the design and concept.

3. Distribution Across Ad Networks Is More Powerful Than You Think

Use the custom placements team to help determine site selections.

4. Optimize with Cost Per Interaction (CPI)

Lower CPIs are a result of a high degree of interactions and a clear CTA.

5. Use the Flexible Power of Gadget Ad Technology

Maintain user interest by keeping content fresh, relevant and useful.


Increased sales by 40%

Don’t, Don’t, Don’t, Don’t Believe the Hype

Video and gadget ads are ONLY for “branding.”FALSE

Drove more qualified users

Source: http://adwords.blogspot.com/2007/05/case-study-using-video-advertising-to.html

8


Product Demonstration5Website Extension4

Can You Think of Others?6

Brand Engagement & Extension3Asset Connections Come to Life2Professional & User-Generated Content Converge1

Use These Kinds of Concepts


Website Extension - Peugeot 207 GTi

Opening Image Racetrack Video 360 Spin

Choose Color

Test Drive

• Automotive and lifestyle sites

• Over 2300 interactions

• Engaged user on averaged interacted 2.5 times with ad

1

Improving Customer Experiences:Tracking Customers

Panel:Is Your Website Getting the Job Done Right? At MarketingProfs Business-to-Business Forum, 2007 October 1, 2007

Prepared by User Centric:Robert Schumacher, PhDManaging Director

User Centric, Inc.

Web Metrics – The StoryWhere customers visit,Where they come from, andWhen they visit

But they can’t tell you …

Why Customers VisitIf They Are Satisfied

2

User Centric, Inc.

Automated User Testing

You’ve got questions that Web Analytics can’t answer…

Who’s coming to my site?

How do customers really use the website?

Do different customer groups use the site differently?

What are the roadblocks to success and satisfaction?

Why don’t customers buy more, sign-up more, respond to ads?

Where do we stand compared to the competition?

How do people use multiple websites over a period of time to address their needs?

User Centric, Inc.

Basic Approach – Automated User Testing

Measure behavior and ask questions while user surfs– Customers opt-in– Measure behavior (navigation, clicks, mouse movement, scrolling)– Ask questions (before, during, and after surfing)– Variety of ways to report the data – depends on your needs

Easy customer participation and invitation– Recruit from your site, no plug-in required (1 session, 1 site, today)– Or, recruit from panel and use plug-in (many weeks, many sites)– Multiple ways to invite customers to opt-in

Large “N“ research

Complementary to other approaches– Usability testing, surveys, analytics (e.g. Omniture)

3

User Centric, Inc.

Types of Automated User Testing

True Intent / Experience Report – No tasks

Benchmarking – Your site and one or more competitors with tasks

Variant testing – Which design variation is best: success and satisfaction

Industry monitor – Behavior and opinions of a customer segment in a market space over time. e.g. How do Latin Americans consume business news?

User Centric, Inc.

Automated User Testing – Searching

How do customers use Search?

Which elements or rather functions stimulate or obstruct purchase processes?

Questions

425 respondents were recruited offline.– While surfing they could report feedback

LEOtrace™ recorded all of the customers' actions as well as the pages they visited.

– As a result valuable indications regarding optimization measures could be generated.

Method and Implementa-

tion

4

User Centric, Inc.

Automated User Testing - Example

Idea: Automated-

Testing

Website-usage at home in the "natural environment"

Real involvement

"Natural" tasks (just what people actuallywant to do)

Ability to "observe" and "measure" (not only "ask")

User Centric, Inc.

Automated User Testing - Process

Online-recruitment via layer

Invitation, data privacy statement

Short pre-exploration

Remote-Session

In-between-feedback

Post-exploration

5

User Centric, Inc.

Customer Goes to Website to search for information on Java

Customer Gets Invited to Participate in Panel

Clicks Link and Gets Instructions

User Centric, Inc.

Instruct the Panel and Get Crucial Customer Data

Gather Information About the CustomerPanel Instructions Appear

6

User Centric, Inc.

Collect Data

Customer Goes Back to Searching…Tracking is

“ON”

Customer’s Can Also Choose to Provide

Feedback at Any Time

When “Done”Click Here

User Centric, Inc.

Close with Survey and Turn Off Tracking

Customer Completes Post Tracking Survey

Customer Completes Post Tracking Survey

Back to Site…Tracking is

“OFF”

7

User Centric, Inc.

Automated User Testing – The Guts

Participants

UsabilityExpert

Recalls all recorded actions of the respondents

Survey Server

Give free and scaled feedback

WebsiteProxy-Server

Natural use of the target web site

Retrieves the participants' comments and evaluations

Passes inquiries on to

An adjustment of the remote tool regarding the target website is necessary for the version without Plug-in.

User Centric, Inc.

So What Do You Learn?

8

User Centric, Inc.

Individual Customer Behaviors

User Centric, Inc.

Rolled Up Customer Behavior

Timeline of all pages visited for each customer

All customer feedback and

analytics filtered by demographic

data

9

User Centric, Inc.

Overlays of Clicks by Page Areas

User Centric, Inc.

Automated User Testing - Heatmaps

10

User Centric, Inc.

Automated User Testing – User Experience Scorecard

User Centric, Inc.

Search Study - Background

1844 people clicked the “Participate” link. (We know each search term used)

425 (~ 23%) gave demographic data (Frequency, Role, Purpose)

100 Used the Pos/Neg buttons – have qualitative data

325 (~18%) hit the “Done” button on the feedback bar

65 (~4% of 1844 or 15% of 425) completed the final questionnaire.

11

User Centric, Inc.

A Peek at the Findings…Framework: Web Analytics tell “what” not “why”

– Because we have session data we can follow user around– Web Analytics provides “points”, not whole trajectory through the UI

A couple of learnings…– Search terms of three or more terms yielded lower satisfaction ratings– Users searching for technical assistance issues also had lower satisfaction

ratings

How do you know?– Customers report on what they are searching and– We can examine their click path and follow through their experience and tie to

rating and qualitative feedback

So what did we do?– Issue was with user interface users went to wrong location

• Change user interface in wrong locations to add “road signs” because you know where customer went and why

User Centric, Inc.

Automated User Testing - Comparison

Obtain valid, high-quality data in the private or work environments of respondents

Detailed analysis ranges from click-paths and heat maps over silent movies on to the log files of competitors' pages.

Automated User Testing is suitable for international target groups that are regionally scattered and/or difficult to reach, such as B2B.

Inexpensive testing including great samples is possible.

Advantage compared to

Lab-Tests

12

User Centric, Inc.

So Let’s Switch Gears

+

User Centric, Inc.

What Can Eye Tracking Do For You?

13

User Centric, Inc.

Why Do Eye Tracking?

Why eye tracking?– Gives insight into customers’ cognitive processes– Provides more information than observable behavior (e.g., clicks)– Provides more objective information than self-report

Eye tracking applied in a variety of projects including:– Commercial and informational Web sites– Major search engines and portals– Product packaging– Medication labels– Email and online advertising

Eye Tracking at User Centric

User Centric, Inc.

Eye Tracking Clients


14

User Centric, Inc.

Eye Tracking Lab

Moderator(administers tasks & controls the eye-tracking software)

Tested interface/stimuli

Tobii 1750 remote eye-tracking systemand 17”monitor

Infrared camerasintegrated into the monitor (track participant’s eye movements)

Participant


TEST ROOM

User Centric, Inc.

Basic FactsEye tracker captures location of the eyes when user is looking at a stimulus

Most common type of eye movements “Saccadic eye movements”

Saccadic eye movements consist of:– Saccades (“jumps”)– Fixations (“stops”)

Assumption: fixation = attention

Introduction to Eye Tracking

15

User Centric, Inc.

What Can Eye Tracking Tell You…and How?

Introduction to Eye Tracking

Cognitive processing demands or Customer mental workload Pupil diameter

If you want to know about the… Then measure the…

Prominence or perceived importance of an area

Percentage of users fixating on an area, the order of 1st fixation on an area, or number of visits to area

Layout effectiveness or visual search demands

Number of fixations before target found, time to 1st fixation on targetOr scanpath complexity

Information clarity or density or information processing demands Fixation length

Informativeness of an area or customer interest in the area Number fixations on an area

User Centric, Inc.

Applications of Eye Tracking

16

User Centric, Inc.

ComparisonsEye tracking provides additional measures to assess and compare alternative designs:

– Us vs. competition– Design vs. redesign (before and after)– Redesign concepts– Design with and without element X

This assessment can help us choose the design which:– Ensures that users notice all key areas– Facilitates the most effective and efficient search (i.e., users can find things and

can find them easily)– Facilitates the most effective and efficient information processing (i.e., users can

easily understand information presented)– Poses the lowest cognitive processing demands on the users

Applications of Eye Tracking: For Comparisons

User Centric, Inc.

Example: Compare Attention Allocation on Key AreasYou are considering changes to your homepage

– Suppose: • You’ve added new branding, new product info, and new ad space• Creative has come up with five new concepts • But…

– How do you know which will ensure that customers notice the new information?

– Task: • Your friend emailed you a link to this Web site and you decided to see what

the site is about

– Measures: • Percentage of customers fixating on area• Number of visits to a page area• Order of 1st fixation on area


17

User Centric, Inc.

Compare Attention Allocation on Key Areas


AREA C: New Product Intro

AREA B: Ad Space

AREA A: New Branding

KEY AREAS OF INTEREST

100%

62%

77%

CONCEPT 3

45%

73%

82%

CONCEPT 2

100%

45%

82%

CONCEPT 4

45%

77%

77%

CONCEPT 5

100%

92%

100%

CONCEPT 1

Areas fixated by at least 80% of users.

B A

C A

AA

A

B

B

B

C

CC

C

B

User Centric, Inc.

Applications of Eye TrackingFor Individual Evaluations

18

User Centric, Inc.

Individual EvaluationsOn a per design basis, eye tracking can help:

– Assess decision making processes– Diagnosing search and labeling efficiency– Determine search strategies and user expectations– Explain ineffective or inefficient interaction (usability issues)

Applications of Eye Tracking: For Individual Evaluations

User Centric, Inc.

Assess Decision Making ProcessesWhen multiple relevant targets are present, how do users decide where to click?

Online search example– Which search results are

considered prior to the click?

– Which elements of a result matter the most? (Title? Description? URL?)

Online retailer example– Task: Purchase a gift for …

on … .com– Which options do users

consider and which do they miss/ignore before deciding where to click?


19

User Centric, Inc.

Diagnosing Search and Labeling EfficiencyTask: You are building a home for a customer in Pennsylvania who wants to keep their heating & cooling costs low in their new house. Find a part of this website that demonstrates how Company products address this issue.


User Centric, Inc.

Diagnosing Search and Labeling EfficiencyThe white rectangle shows where the Built on Science link is located. It received minimal attention, and no clicks.


20

User Centric, Inc.

Diagnosing Search and Labeling Efficiency86% of the participants clicked on US Residential first.

However, none of the participants tried clicking Built on Science. The terminology did not resonate with them.

Participants were looking for the term "energy" and eventually selected the Energy Saving Center.

Other places that participants searched for this information include:

– THERMAX

– WEATHERMATE

– Choose the Right Insulation

– Story of Styrofoam

– Performance at the Core

"Built on science – I don’t know what that means. I imagine it is scientific background on the insulation."

-P6, builder

Unassisted Task CompletionStruggled Task CompletionFailure


User Centric, Inc.

Determine Search Strategies and User ExpectationsIf there is a specific target, how do users look for it?

Where did they look first/second?

Helps determine whether placement of elements matches user expectations and if not, where they should be moved.

Example– Task: Purchase a gift

card– User expects the link to

be either on top of the page or at the bottom


Gift Card

21

User Centric, Inc.

Explain Ineffective or Inefficient InteractionEye tracking can help explain why certain usability problems occurred.

Example– Very few users were able

to chat on AIM without downloading it

– Did they not see the “AIM Express” link?

– Eye movements revealed that users saw it. They must have not known what the label meant.

Packaging example– Why do users not know

what’s in the box? Do they miss the info or do they read it, but without comprehending?


52

AIM Express

User Centric, Inc.

Eye Tracking: SummaryWe use eye tracking in combination with other techniquesWe do not use eye tracking to impress but to answer questionsThere are inherent limitations of the methodUser Centric has created guidelines on appropriateness of use

– When to use eye tracking and when not to use it– What measures are appropriate for what type of problem– What does eye tracking data mean when combined with other data, etc.

Applications of Eye Tracking

22

User Centric, Inc.

Robert M. Schumacher, Ph.D.+1.630.376.1180

[email protected]

www.UserCentric.com

Services: Usability Testing User Experience Design Eye Tracking

Industries: e-Commerce, Telecom, Travel, Finance, etc.Areas: Web, Software, Small Screen Devices, MedicalOffices: Chicago & Beijing

Thank You!

Documents

Lessons Learned From 1,001 Web Site Reviews...•Traffic drivers and analytics tools evaluate your conversion path – not your landing page •Most landing pages don’t do a good