TESTING, TESTING: IS THIS THING ON? Integrating Testing & Optimization into your Digital Marketing Efforts
Kate McRoberts, Evantage Consulting
1© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
2© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
I AM A HERO!
3© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTINGhttp://s183.photobucket.com/profile/gruaig_rua/index
@kmcroberts4© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
@www.evantageconsulting.com
Let’s start with a case study….y
How one organization made $60 Million by running a simple experiment. y g p p
5© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
The Question.
How can we convert site visitors to give us their email address?
The Key Metric.
Email signup rate.g p
6© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
Experiment tested two parts of the splash page:splash page:
7© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
Elements Tested
Media areaMedia area• Images and Videos
Buttons
6 media x 4 buttons = 24 combinations
8© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
The Winner
Family image with y g“Learn More” button.• Sign up rate: 11.6% (vs.
8.26% original)g )
• 40.6% sign up rate improvement
9© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
Res ltsResults
• “Roughly 10 million people signed up on the splash page during the campaign. If we hadn't run this experiment and just stuck with the original page that number would be closer to 7,120,000 signups. That's a difference of 2,880,000 email addresses”
• “Each email address that was submitted through our splash page ended up donating an average of $21 during the length of the campaign. The additional 2,880,000 email addresses on our email list translated into an additional $60 million in donations.”
http://blog optimizely com/how-obama-raised-60-million-by-running-an-exp
10© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
http://blog.optimizely.com/how-obama-raised-60-million-by-running-an-exp
The Lessons.
• Simple tests can make a BIG impact
• What YOU think is going to win, may not ACTUALLY win
11© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
H d I t t t d ti i ti ?How do I get started optimization?
12© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
Who took Psych 101?13© 2011 EVANTAGE CONSULTING
Who took Psych 101?
MASLOW
14© 2011 EVANTAGE CONSULTING
morality, creativity, spontaneity problem
self ‐actualization
spontaneity, problem solving, lack of prejudice
esteem(confidence, respect)
love/belonging(friendship, family)
Safety(security of body, resources, employment)
Physiological(air, water, food)
15© 2011 EVANTAGE CONSULTINGHEIRARCHY OF NEEDS
Testing & Optimization!
self ‐actualization
Optimization!
esteem(confidence, respect)
love/belonging(friendship, family)
Safety(security of body, resources, employment)
Physiological(air, water, food)
16© 2011 EVANTAGE CONSULTINGHEIRARCHY OF NEEDS
Hierarchy of Needs for the Modern Marketer testing & optimization style
17© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
But first, a few words about data…
18© 2011 EVANTAGE CONSULTING
Consider this staggering statistic…
“gg g
W t h i f ti“We now create as much information every two days as we did from the dawn of civilization until 2003.
- Eric Schmidt, CEO, Google
19© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTINGhttp://techcrunch.com/2010/08/04/schmidt-data/
20© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
21© 2011 EVANTAGE CONSULTING
22© 2011 EVANTAGE CONSULTING
http://www.mamapop.com/2011/07/facebook-skype-chat-feature-debuts-to-thunderous-shrugs.html/sad-trombone
Marketing organizations everywhere are struggling ith h t t d ith ll thi d twith what to do with all this data.
1,700 CMO’s consistently ranked “Data Explosion” as the #1 “pervasive, universal, game changer” challenge to be addressedbe addressed.
IBM Global CMO Study 2011
23© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTINGhttp://www-935.ibm.com/services/us/cmo/cmostudy2011/cmo-registration.html
Data was once the domain of tech geeks and direct‐marketing “ gurus, while chief marketing officers focused on loftier things like shaping brand perception. But those days are over.
Thanks to an explosion of data …CMOs who want a seat at the table will have to harness customer data and leverage it ‐‐or risk being relegated to chief promotions officer.
‐ AdAge 2/13/2012“When CMO’s Love to Learn Data, They will be VIP’s in the C‐Suite”
24© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTINGhttp://adage.com/article/cmo-strategy/cmos-learn-love-data-c-suite-vips/232699/
i h f d f h d kHierarchy of Needs for the Modern Marketer testing & optimization style
25© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
at
I have a website. I send emails.
26© 2011 EVANTAGE CONSULTING
Hierarchy of Needs for the Modern Marketertesting & optimization style
I have web analytics. I see standard reports.
I have a website. I send emails.
27© 2011 EVANTAGE CONSULTING
Hierarchy of Needs for the Modern Marketertesting & optimization style
We have defined KPIs. We track them regularly. We can spot issues.
I have web analytics. I see standard reports.
I have a website. I send emails.
28© 2011 EVANTAGE CONSULTING
Hierarchy of Needs for the Modern Marketertesting & optimization style
Analysis. Insights. We take action on the data.
We have defined KPIs. We track them regularly. We can spot issues.
I have web analytics. I see standard reports.
I have a website. I send emails.
29© 2011 EVANTAGE CONSULTING
Hierarchy of Needs for the Modern Marketertesting & optimization style
Optimization.
Analysis. Insights. We take action on the data.
We have defined KPIs. We track them regularly. We can spot issues.
I have web analytics. I see standard reports.
I have a website. I send emails.
30© 2011 EVANTAGE CONSULTING
Hierarchy of Needs for the Modern Marketertesting & optimization style
A/B and Multivariate Testing/ g
31© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
A/B and Multivariate Testing/ g
How it works.How it works.
1
Add a line of code to your site.
32© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
A/B and Multivariate Testing/ g
How it works.How it works.
1 2
Add a line of code to your site.
Set up rules you want your test to follow.
33© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
A/B and Multivariate Testing/ g
How it works.How it works.
1 2 3
Add a line of code to your site.
Set up rules you want your
Set up challengers.
test to follow.
34© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
A/B and Multivariate Testing/ g
How it works.How it works.
1 2 3 4
Add a line of code to your site.
Set up rules you want your
Set up challengers.
Launch test!
test to follow.
35© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
A/B and Multivariate Testing/ g
How it works.How it works.
1 2 3 4 5
Add a line of code to your site.
Set up rules you want your
Set up challengers.
Launch test! Run until statistical significance
test to follow.
achieved.
36© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
Bonus question!q
Who knows the formula for statistical significance?Who knows the formula for statistical significance?
*
* May or may not be the actual formula. But it sure looks scary right? Good thing the tool
37© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
it sure looks scary, right? Good thing the tool calculates this for us.
A/B and Multivariate Testing/ g
How it works.How it works.
1 2 3 4 5 6
Add a line of code to your site.
Set up rules you want your
Set up challengers.
Launch test! Run until statistical significance
Full scale winners
test to follow.
achieved.
38© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
A/B vs. MVT ‐‐When to use which kind./
Use A/B testing when you want to…Use A/B testing when you want to…• Get started with testing
• Get results quickly ‐ smaller sample size to achieve statistical significance
• Test dramatic changes/radical redesigns – e.g. does design A work better than design B?
Use Multivariate testing when you want to…• Increase your level of sophistication• Increase your level of sophistication
• Test multiple elements on the same page
• To understand how different page elements interact with each other
• To isolate impact of different elementsp
• Drive incremental improvements, “fine tune“
39© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
Examples. p
40© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
Examples.p
What if…Instead of this style of navigationy g
41© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
A/B/
Let’s try a fly out menu
42© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
A/B/
What if….Instead of manually moving your cursor over to find new topics
43© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
A/B/
Let’s try automatically rotating th bthe banners
44© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
A/B/
I t d f h iInstead of a hero image that is this size….
45© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
A/B/
We tried a smaller one that introducesWe tried a smaller one that introduces more content above the page fold
46© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
Or what if we tested them all at the same time?
Fly‐out menu
Using a Multivariate Testing
Rotating banner
Smaller hero image
tool allows us to see how each of these elements contributes to the most effective combinationRotating banner effective combination.
47© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
HealthSavingsCheckup.comg p
48© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
How does it work?
Step 1 – Enter basic information
St 2 H lth hi tStep 2 – Health history
Step 3 – Financial information
Step 4 – View resultsStep 4 View results
Step 5 – Take action
49© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
What’s driving abandonment? H d t d ti ?How can we deepen engagement and action?
Analytics review and usability testing to diagnose issues and understand reasons why. y
Findings showed opportunity for A/B testing: • Call to action on myUHC.com• Field labels• Field‐level validation• Results page redesignResults page redesign
50© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
Tools – Entry Levely
These tools can be used for free!
51© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
Tools – Mid Tier
52© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
Tools – Enterprisep
53© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTINGhttp://www.whichmvt.com/
W t t t t th t f th id?Want to get to the top of the pyramid?
We have seen people get stuck at the various levels. Why? y
CHANGECHANGE
54© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
Change. g
This is a mindset shift. And a commitment to a newThis is a mindset shift. And a commitment to a new process. Individual advocates alone won’t succeed.
Change tips: • Create compelling vision for the future
• Have clear and visible sponsorship to champion it
• Create small wins and communicate the heck out of them
• Find your change advocates and embrace themFind your change advocates and embrace them
•Demonstrate tangible benefits to the business
55© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING
56© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTINGhttp://s183.photobucket.com/profile/gruaig_rua/index
Thank you!Thank you!
@kmcroberts
Want a copy of the deck?
57© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING