Microsoft Case Study:

40% Conversion Rate Increase of SEM Traffic using

Multivariate Testing

Michael Schultz Director Product Management & Marketing, Microsoft Office Unmanaged Services

Frans KeylardDirector of Optimization, Widemile

Agenda

• A/B & Multivariate Testing Overview

• Microsoft Office Live Small Business Case Study

– Business Overview

– Case Study Details

• Top 10 Page Optimization Tips

Cus

tom

er

Sal

es F

unne

l

• Email marketing• Loyalty Programs• Newsletters• Personalization• Coupons /Surveys

Widemile Focus

• Key web pages• Sales funnels• Landing pages• Registration pages

• Search marketing• Display advertising• Affiliate marketing• Contextual ads• Ad targeting

Conversion Optimization

Conversion

Acquisition

Retention

*Source: eMarketer – April 2009

Investment recommendation:5% to 15% of acquisition budget

Testing and segmentation can improve retention performance

Automated, data-driven action that increases revenue by optimizing a user’s online experience.

$37 Billion annual US spend by 2013. $23 Billion spent in 2008.*

Testing Options – Approach Overview

Basic Analytics

Ideal use: > Validates traffic and conversion rates

Limitation: > Simple measurement; no optimization

A/B Split Testing

Ideal use: > Best for testing two or more radically different pages <i.e. template testing>

Limitation: > No insight into why one page is better, just which won among traffic tested

Multivariate Testing (MVT)

Ideal use: > Test different content, simultaneously determining most effective combination> Segmentation can be applied for more granular optimization in follow-on tests

Limitation: > Requires sufficient traffic & conversions

MVT Testing Options - Full & Fractional Factorial

Fractional FactorialFull Factorial

5

0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170

Full Factorial

Fractional Factorial

Number of Days to Complete Test

Expertise

Requirements for Success

6

User Interface Proven Architecture Reporting Interface

Analyst Creative Web Developer

Technology

2009 – Year for Conversion Optimization

Microsoft Office Live Small Business

Audience:

• Small businesses, 1-10 employees(real estate agent, local bakery, video store, restaurant)

• Non-profits/Organizations(community groups, soccer clubs, PTSAs)

Benefits:

• Free website and email

• Free services to view, store and share documents online

Goals Summary

Goals:

• Improve sign up conversions

• Better understand audience

• Identify highest impact page factors

Testing Strategy:

• Leverage proven landing page template

• Execute multivariate test

• Fractional factorial approach to minimize time

Microsoft Case Study

Getting started…1. Define page to be tested

2. Analyze outside factors

1. In-bound traffic

2. Traffic volume

3. UX - Funnel

3. Review page & conversion goals

Original web page

Testing Considerations

What is the best hero shot?

Does the headline grab the customer’s attention?

Do these resource links belong here?

Does the button maximize conversions?

Does this copy help or hurt?

Indentifying Factors to Test

1

2

3

4

5

Defining a Test Matrix

No. Factor (Variable) Levels (versions)

1 Hero Shot 4

2 Title 4

3 Description 4

4 Call-To-Action Button 4

5 Resource Links 2

Audience Participation – Best CTA Button

Level 2

Level 3

Level 4

Level 1

Efficiency Through Technology

1 of 16 2 of 16

3 of 16 4 of 16

0 100 200 300 400 500

Full Factorial

Fractional Factorial

Comparison - # of Experiments Required for Test by Approach

512

16

And the Winner is…

40%Improvement

Which Factors Made the Biggest Difference

High Influence

Medium Influence

Low Influence

Hero Shot

Title

Description

Call-To-Action

Resource Link Location

High Influence Factor - Key Learning

Factor/level insights

• “Get Started Free” won

• The free aspect of the service is very important to get people to try it out

• More task focused terminology like “Set up my business!” or “Build My Online Business” lost

• “Sign Up Free!” which is the typical text found on so many CTA buttons also lost out.

• Future testing around the positioning of the FREE service

Case Study Summary

Microsoft Office Live Small Business’ testing and optimizing investment:

1. Delivered a 40% lift in conversion

2. Reduced customer acquisition cost

3. Maximized SEM performance

Testing and optimizing content is a fast and effective way to maximize the performance of your SEM spend.

10 Optimization Tips

1. Offer Clarityo Make sure the offer is clear, relevant, and compelling

o Keep your pages simple, clear, and appealing

2. Path to Conversiono Make sure the path to conversion is simple and straightforward

o Keep conversion paths efficient, focused, uncluttered

3. Performanceo Prospects will not tolerate slow pages or links

o Google penalizes slow page loads

4. Test Design and User Experienceo Avoid confusing landing page designs and difficult to read text

o Make transitions seamless and keep the consistency high (both messaging and look & feel)

5. Seasonalityo Seasonally optimize your pages and PPC campaign

o Leverage historical SEM in planning current and future campaigns

20

10 Optimization Tips

6. Target Audienceo Optimize your content based on audience insight

o Capture contact information early to better inform your marketing

7. Segment Audienceo Optimize for distinct audiences to convert

o Segment your traffic and test to see if unique landing pages would perform better

8. Offers and Goalso Create and test offers that have a high perceived value but a low cost to fulfill

o Force your page to have only one clear goal and link ads to landing pages

9. Consistencyo Build consistency and relevancy between PPC campaigns and where you send your traffic

o Reinforce the call to action/offer on order and lead forms

10. Competitiono Look at what your competition is doing, but also what they are not doing

o Differentiation can often help, so test new approaches, offers, and incentives

4 Additional Tips for Marketers

1. Drink the Kool-Aid and start now!

2. Think of multivariate testing as a “fixed” marketing expense and allocate budget appropriately

3. Ensure collaboration between Creative, Optimization Testing and SEM

4. Make sure Optimization vendor learns the business and understands the audience

22

Questions?

PresentersFrans Keylard – frans@widemile.com

Michael Schultz – mschultz@microsoft.com

Content Informationwww.widemile.com

www.officelive.com