Effectiveness Metrics for the Web

Effectiveness Metrics for the WebPrepared for the University of Washington

May 17th, 2001

What Will We Be Discussing Today?

Overview of Online Metrics

Measuring Web Site Effectiveness

Wells Fargo Case Study

Cap’n Crunch Case Study

Q&A

Introduction

Online Metrics – Art vs. Science

What is the Value of Online Metrics?

Prove the value of a campaign/initiative

Evolve your campaigns, products, etc.

Understand your users better

Forecast and respond to trends

Uncover new business opportunities


Perspective

Process Tools

• User-Centric

• Results Oriented

• Seek Affirmation

• Web Reporting Tools

• Campaign Tracking

• Sales Data

• Dynamic Marketing• Online Focus Grouping

• Surveys

• Periodic Analyses


The Art & Science of Measuring Effectiveness

User Experience (UXP) is:

Whatever the user sees, thinks, or feels in the interaction with a brand, product, or company.

The UXP is an accumulation of interactions through Web, brick and mortar, call center, wireless, direct marketing, kiosks, DITV, PDAs, etc.

The UXP should drive transactions, learn from the customer, and positively impact brand perception

UXP is successful through understanding expectations, quality of execution, and ability to adapt quickly to ever-changing user needs.

TransactionsCustomer

Insight

User

Business

GoalsAttraction Expectations

Functionality ContentEnvironment

The User Experience

BrandImpact


Our View on Measuring Web Effectiveness

Site Diagnostics

Online Focus Groups

Web Reporting & Analysis - eMetrics

Natural Language Query and Search

Setting Performance Models and Driving Business Value

DareStep Perspective on Evaluating Web Site Effectiveness

Professionals

Technology &Process

Real Users

Technology


Information Architecture: structure of the site, navigation, and labeling

Content: text, images, and multimedia content

Functionality: such as transactional capabilities, personalization, etc.

Brand Impact: consistent message and visual representation of brand

Design: effectiveness of the visual layout, use of color, type, image, etc., in conveying the brand image and enabling users to accomplish their goals

Marketing: can users easily find the site? Are their expectations set correctly?

Community: social interaction and creation of intellectual capital like customer book reviews


Site Diagnostics: Evaluation Criteria

Usability: how easy is it for users to accomplish their tasks?

Support: What kinds of support (email, phone, etc.) are available and how effective are they?

Consumer Confidence: will users trust this site to do business? Includes security, privacy, credibility, and reliability


Site Diagnostics: Evaluation Criteria

• Target customer profile

• Large sample size

• Panel proactively managed

Invite sample of testers

Invite sample of testers

Testers login from natural Web settingTesters login from

natural Web setting

• Natural setting

• Minimize “demand characteristics”

• Anonymity encourages candor

Testers evaluate site(s)

Testers evaluate site(s)

• Intent-based context

• Expectations and interaction interpreted together

• Objective and subjective

indicators

Online Focus Groups: Methodology


Online Focus Groups: User Response


Natural Language Input (Query /Search)

- Responses tailored to explicit Query and Search

- Enhancement Opportunities for Knowledge Base

- Improved perception of customer service

Keyword Search

- Access through Indexed Content Management

Service Effectiveness Measures / User Experience

- Percent Questions Answered Correctly

- Speed to Correct Response

- Content Gaps – Enhancement Opportunities

Search


Natural Language: Query and Search


Web Reporting & Analysis: eMetrics

Site Analysis: Operations & Design- Volume and Capacity Planning

- Response Time and Latency

- Dead Ends – Content Augmentation – Link Integrity

Customer Analysis: User Behavior & History- Customer Profiles

- Behavioral

- Lifetime Value Analysis

Business Analysis: Financial & Marketing- Return on Investment

- Campaign and Partnership Effectiveness

- Cost of Business and Efficiencies

Web Reporting & Analysis: Range of Reports


Basic Reports – Metrics based on traffic data

eBusiness Reports – Reports based on integrated traffic data, transactional data, registration data, email

Behavioral Profile Reports – Reports generating customer profiles and segmentation, based on integrated traffic data, transactional data, registration data, email

Web Reporting & Analysis: Basic Report

Number of Visits by Visit Duration

This report shows the amount of time visitors spent on your Web site during visits.

ResultsVisit Duration # of Visits

1 No Duration 12,5592 1 second - 1 minute 2,8373 1 minute - 2 minutes 3,5044 2 minutes - 3 minutes 3,5985 3 minutes - 4 minutes 2,9506 4 minutes - 5 minutes 2,5887 5 minutes - 6 minutes 2,2438 6 minutes - 7 minutes 1,8419 7 minutes - 10 minutes 3,45610 >= 10 minutes 2,093

Summary# of Visits

Total # of Visits 37,669Avg. Visit Duration (minutes) 4.2

Basic report category is fundamental to all business functions and forms the building block of Web Site Analysis.

Types of reports:• Hits• Page views• Visits• Click-through• URL of origin


Source: NetGenesis

Web Reporting & Analysis: eBusiness Report

Purchase Status

This shows what percent of users are browsing vs. buying.

ResultsBought Status # of Visits % of Total Cumulative %

1 Did Not Buy During Visit 30,728 86.8 86.82 Bought During Visit 4,659 13.2 100.0

Total 35,387

eBusiness report category uses various data to depict visitor activity on the Web, and is increasingly required to support higher level of business.

Types of reports:• Conversion rate• Customer Lifetime value• ROI reports• Site performance


Source: NetGenesis

Web Reporting & Analysis: eBusiness Report

171,511 (14.3 %) Entered Registration Page

42.1%

56.5%

Registration Process Step 1

Step 2

Step 3

Confirmed

100%

64.5%

1,200,500 Visitors accessed the Everyday Law site

110,636 (9.2% ) Reached Step 2

96,913 (8.1 %) Reached Step 3

72,035 (6.0% ) Registered

35.5% Dropout at Step 1



85.7% Never Entered

Registration Page

57.9% Abandoned the Registration Process

eBusiness report category uses pre-generated segmentation of identified visitors, to answers higher level business questions.

Types of reports:• Visitor form abandonment• Visitor shopping cart

abandonment• Repeat purchasers• Repeat items purchased by

type of visitor

Source: Personify


Web Reporting & Analysis: Behavioral Profile Report

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Criminal Law

P rofessional Section

Ask the Expert

Small Businesses

Consumer Law

Resources

News & Tips

Cursory Visitors Attorneys Small Businesses Community Seekers News fans

6%

9%

11%

16%

18%

22%

29%

32%

45%

International Law

Resources

Criminal Law

IP Law

Small Businesses

Employment Law

News & Tips

Ask the Expert

Professional Section

Source: Personify


How many visitors who viewed the different content areas also subscribed to the Newsletter?

Case Study: Wells Fargo

Wells Fargo’s Objectives

Convert existing customers to online banking and bill payment

Engage account holders in other relationships with the bank

Lower the costs of mining customer relationships and building new accounts


Approach

Establish an ad serving system to track the delivery and effectiveness of campaigns

Capture the “low hanging fruit” – target existing account holders with high internet usage on their favorite sites

Target potential and existing customers on lifestyle sites and on wellsfargo.com


Potential Metrics?

Click through rates

Total Accounts

Cost per click (CPM + Agency Costs)

Cost per transaction

Cost per new account

Potential total customer value


Wells Fargo: Execution


User Sees Banner on Web

User Clicks Through (cookied)

User Visits Landing Page User Fills Out Application

User Exits – Browses

Elsewhere

User Exits – Browses

Elsewhere

# of views

# of clicks

# of visits,time

# of exits

# of apps

# of returns

Brand Impression



Old Banner Creative

• Average Click-through rate of 0.x%• y number of applications per month• z number of “branded impressions”

New Banner Creative

• Same average click-through rate• ~10% higher number of applications per month• ~20% higher “branded impressions”

Champion

Challenger



Old Style Creative

• Average click-through rate of 0.x%

New Style Creative

• ~10% higher click-through rate

Champion

Challenger



468x60

Banner Creative

160x40

Landing Page

Overall Results

Installed Campaign Management tools - adapted creative, placements on the fly

Optimized campaigns for optimal results, resulting in a lower cost-per-transaction

Drove considerably higher numbers of new accounts across all products


Case Study: Cap’n Crunch

Cap’n Crunch’s Objectives

Create brand relevancy with the 9-12 year old market

Invigorate the aging brand and create a sense of excitement among youth

Increase sales with new customers

Recapture market share


Cap’n Crunch: Approach


Potential Metrics?

Page Views

Time Spent on Site

Registration

Clue Entries / Offline Interactions

Sales

Branding Value


Cap’n Crunch: Campaign Highlights








Overall Results

Week 1 gave us 600,000 impressions, 70,000 user sessions

After the first half of the campaign, the web site recorded:

- 4 million impressions

- 500,000 user sessions

- 100,000 registered users

Quaker beat their projected sales


Overall Results

Avg. time on Disney’s site: 15 min.

Avg. time on Nickelodeon’s site: 19 min.

Avg, time on Cap’n Crunch: 20 min.

Kids see an average of 10 – 20 ads per year, for a total of 10 minutes of exposure to the brand – the web far outpaced old media


Overall Results

7MM CD-ROMs distributed in special packs

Avg. play time of 5-7 hours (initial)

Cap’n Crunch increased their market share from 2% to 4.5% (previous promo increased share by 1%)


Discussion

Documents

Effectiveness Metrics for the Web