Integrating Testing & Optimization into your Digital Marketing Efforts

TESTING, TESTING: IS THIS THING ON? Integrating Testing & Optimization into your Digital Marketing Efforts

Kate McRoberts, Evantage Consulting

1© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING


I AM A HERO!

3© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTINGhttp://s183.photobucket.com/profile/gruaig_rua/index

@kmcroberts4© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTING

@www.evantageconsulting.com

Let’s start with a case study….y

How one organization made $60 Million by running a simple experiment. y g p p


The Question.

How can we convert site visitors to give us their email address?

The Key Metric.

Email signup rate.g p


Experiment tested two parts of the splash page:splash page:


Elements Tested

Media areaMedia area• Images and Videos

Buttons

6 media x 4 buttons = 24 combinations


The Winner

Family image with y g“Learn More” button.• Sign up rate: 11.6% (vs.

8.26% original)g )

• 40.6% sign up rate improvement


Res ltsResults

• “Roughly 10 million people signed up on the splash page during the campaign. If we hadn't run this experiment and just stuck with the original page that number would be closer to 7,120,000 signups. That's a difference of 2,880,000 email addresses”

• “Each email address that was submitted through our splash page ended up donating an average of $21 during the length of the campaign. The additional 2,880,000 email addresses on our email list translated into an additional $60 million in donations.”

http://blog optimizely com/how-obama-raised-60-million-by-running-an-exp


http://blog.optimizely.com/how-obama-raised-60-million-by-running-an-exp

The Lessons.

• Simple tests can make a BIG impact

• What YOU think is going to win, may not ACTUALLY win


H d I t t t d ti i ti ?How do I get started optimization?


Who took Psych 101?13© 2011 EVANTAGE CONSULTING

Who took Psych 101?

MASLOW

14© 2011 EVANTAGE CONSULTING

morality, creativity, spontaneity problem

self ‐actualization

spontaneity, problem solving, lack of prejudice

esteem(confidence, respect)

love/belonging(friendship, family)

Safety(security of body, resources, employment)

Physiological(air, water, food)

15© 2011 EVANTAGE CONSULTINGHEIRARCHY OF NEEDS

Testing & Optimization!

self ‐actualization

Optimization!

esteem(confidence, respect)

love/belonging(friendship, family)

Safety(security of body, resources, employment)

Physiological(air, water, food)

16© 2011 EVANTAGE CONSULTINGHEIRARCHY OF NEEDS

Hierarchy of Needs for the Modern Marketer testing & optimization style


But first, a few words about data…


Consider this staggering statistic…

“gg g

W t h i f ti“We now create as much information every two days as we did from the dawn of civilization until 2003.

- Eric Schmidt, CEO, Google

19© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTINGhttp://techcrunch.com/2010/08/04/schmidt-data/




http://www.mamapop.com/2011/07/facebook-skype-chat-feature-debuts-to-thunderous-shrugs.html/sad-trombone

Marketing organizations everywhere are struggling ith h t t d ith ll thi d twith what to do with all this data.

1,700 CMO’s consistently ranked “Data Explosion” as the #1 “pervasive, universal, game changer” challenge to be addressedbe addressed.

IBM Global CMO Study 2011

23© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTINGhttp://www-935.ibm.com/services/us/cmo/cmostudy2011/cmo-registration.html

Data was once the domain of tech geeks and direct‐marketing “ gurus, while chief marketing officers focused on loftier things like shaping brand perception. But those days are over.

Thanks to an explosion of data …CMOs who want a seat at the table will have to harness customer data and leverage it ‐‐or risk being relegated to chief promotions officer.

‐ AdAge 2/13/2012“When CMO’s Love to Learn Data, They will be VIP’s in the C‐Suite”

24© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTINGhttp://adage.com/article/cmo-strategy/cmos-learn-love-data-c-suite-vips/232699/

i h f d f h d kHierarchy of Needs for the Modern Marketer testing & optimization style


at

I have a website. I send emails.


Hierarchy of Needs for the Modern Marketertesting & optimization style

I have web analytics. I see standard reports.




We have defined KPIs. We track them regularly. We can spot issues.





Analysis. Insights. We take action on the data.






Optimization.

Analysis. Insights. We take action on the data.






A/B and Multivariate Testing/ g



How it works.How it works.

1

Add a line of code to your site.




1 2


Set up rules you want your test to follow.




1 2 3


Set up rules you want your

Set up challengers.

test to follow.




1 2 3 4



Set up challengers.

Launch test!

test to follow.




1 2 3 4 5



Set up challengers.

Launch test! Run until statistical significance

test to follow.

achieved.


Bonus question!q

Who knows the formula for statistical significance?Who knows the formula for statistical significance?

*

* May or may not be the actual formula. But it sure looks scary right? Good thing the tool


it sure looks scary, right? Good thing the tool calculates this for us.



1 2 3 4 5 6



Set up challengers.

Launch test! Run until statistical significance

Full scale winners

test to follow.

achieved.


A/B vs. MVT ‐‐When to use which kind./

Use A/B testing when you want to…Use A/B testing when you want to…• Get started with testing

• Get results quickly ‐ smaller sample size to achieve statistical significance

• Test dramatic changes/radical redesigns – e.g. does design A work better than design B?

Use Multivariate testing when you want to…• Increase your level of sophistication• Increase your level of sophistication

• Test multiple elements on the same page

• To understand how different page elements interact with each other

• To isolate impact of different elementsp

• Drive incremental improvements, “fine tune“


Examples. p


Examples.p

What if…Instead of this style of navigationy g


A/B/

Let’s try a fly out menu


A/B/

What if….Instead of manually moving your cursor over to find new topics


A/B/

Let’s try automatically rotating th bthe banners


A/B/

I t d f h iInstead of a hero image that is this size….


A/B/

We tried a smaller one that introducesWe tried a smaller one that introduces more content above the page fold


Or what if we tested them all at the same time?

Fly‐out menu

Using a Multivariate Testing

Rotating banner

Smaller hero image

tool allows us to see how each of these elements contributes to the most effective combinationRotating banner effective combination.


HealthSavingsCheckup.comg p


How does it work?

Step 1 – Enter basic information

St 2 H lth hi tStep 2 – Health history

Step 3 – Financial information

Step 4 – View resultsStep 4 View results

Step 5 – Take action


What’s driving abandonment? H d t d ti ?How can we deepen engagement and action?

Analytics review and usability testing to diagnose issues and understand reasons why. y

Findings showed opportunity for A/B testing: • Call to action on myUHC.com• Field labels• Field‐level validation• Results page redesignResults page redesign


Tools – Entry Levely

These tools can be used for free!


Tools – Mid Tier


Tools – Enterprisep

53© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTINGhttp://www.whichmvt.com/

W t t t t th t f th id?Want to get to the top of the pyramid?

We have seen people get stuck at the various levels. Why? y

CHANGECHANGE


Change. g

This is a mindset shift. And a commitment to a newThis is a mindset shift. And a commitment to a new process. Individual advocates alone won’t succeed.

Change tips: • Create compelling vision for the future

• Have clear and visible sponsorship to champion it

• Create small wins and communicate the heck out of them

• Find your change advocates and embrace themFind your change advocates and embrace them

•Demonstrate tangible benefits to the business


56© 2011 EVANTAGE CONSULTING© 2012 EVANTAGE CONSULTINGhttp://s183.photobucket.com/profile/gruaig_rua/index

Thank you!Thank you!

@kmcroberts

Want a copy of the deck?


@[email protected]

Business

Integrating Testing & Optimization into your Digital Marketing Efforts