Upload
sratkinson
View
151
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Offers some tips and techniques for running marketing experiments ("ab-tests", "multivariate tests") based on experience at two major consumer internet companies
Citation preview
Running Experiments for Online and Offline Marketing
for marketing
(a.k.a “A/B” testing)
© 2013 Steven Atkinson
This is a personal presentation. Any views or opinions represented in this presentation are personal and belong solely to the author and do not represent those of people, institutions or organizations that the author may or may not be associated with in professional or personal capacity, unless explicitly stated. Any views or opinions are not intended to malign any religion, ethnic group, club, organization, company, or individual.
All content provided on this presentation is for informational purposes only. The author of this presentation makes no representations as to the accuracy or completeness of any information on this presentation or found by following any link on this presentation. The author will not be liable for any errors or omissions in this information nor for the availability of this information. The owner will not be liable for any losses, injuries, or damages from the display or use of this information.
Overview
• what are experiments?• how can existing marketing tools inform experiments?• case studies: linkedin email tests, commerce site• but how does the math work?
– sample low-sample-size analysis• caveats vendor
Long, windy road driving with experiments…
types of experiments
• online and offline – online: website ux,
online-clickthru, ad partner testing
– offline: (e)mailings, surveys, media spots
• channel
experiment terminology• tests…• … have cells
– control or test cells• visitors, customers,
members• conversion events for the
business• business metrics
– e.g. time-based rates of events (download/day)
– facets: stuff tracked for everyone to slice by during analysis
Bonus points: Track Test Meta-Data and Resultsin a Journal / Database so you can review past experiments later.
tools for experimental marketing
• site tracking– gives site engagement, page
falloff, time spent– can help answer questions
about why A/B test cells fail, was there a systemic problem poisoning a test cell?
– can help show where your site is fighting your goals -> design new tests that change page flow for example
tools for experimental marketing
• online surveys– gives qualitative data
without hosting qual sessions
– use as pre-flight gut check for future ideas to experiment with• e.g. “what video game
system do you own”
tools for experimental marketing
• Online Marketing / Creative Experiments– channels are a great way to slice test cell
results and tests are a way to know how effective a channel is for a particular task
– send multiple creative to same partner to run in parallel
– channels must be tracked throughout visitor sessions
• When site has more traffic: Consider excluding those arriving by ONLINE or PAID SEARCH channels from your tests (they are not walk-ons / potentially different intent)
before you set up an experiment
• make your metric choices upfront– know what you are measuring for each test– have a clear hypothesis stated
• know what your conversion event is
email testing
• you are in control of volume
• measurable metrics (open, click-thru)
• Invitation emails subject to rigorous testing over 12 months
• Then new marketing folks tested the simple “Please join my professional network on LinkedIn.”
• Major increase in accepted invitations
Does this work for low-traffic?
• statistical test design– Choose 3 inputs:
• confidence level (often 95%)• #cells (start with 2)• conversion difference of interest
– (e.g. >= 1% difference interests me)
– Get back:• required volume per cell• time required for that volume to be
reached
• it’s OK to have smaller sample sizes for each cell if the observed conversion difference is big enough – so be bold in variation design for situations where sample size is smaller – stop running stupid tests
sample test for analysis
Example: Changed Home Page on Commerce Site
“conversion” == a click on a email or call link indicating interest in ordering
analysis: assuming binomial approximation to normal distribution
• Control Existing Homepage cell: – 145 visitors, 35 clicks conversion: 24% (0.24)– standard error for 95% confidence +-6.9%– range: 0.069 == 1.96*square root (0.24 * (1-0.24) / 145)
• Menu-as-Homepage cell:– 123 visitors / 49 clicks conversion: 40% (0.40)– standard error for 95% confidence +-8.6%– range: 0.086 == 1.96*square root (0.40 * (1-0.40) / 123)
• Analysis: No Confidence overlap:– 17.1%...30.9% vs 31.4%...48.6%– So Menu-as-Homepage cell is a clear winner– Moral: Even with low traffic you can get significant results with big enough
difference in conversion rate.– Moral 2: if results are not conclusive go with the variation best for long term
strategy of the business
(statistical numbers based on http://visualwebsiteoptimizer.com/split-testing-blog/using-ab-split-testing-reduce-bounce-rate-ecommerce-store/)
experiment procedure: be careful• start simple• not too many variations• wait until you have large enough sample size for
your desired conversion improvement before doing analysis
• accept that sometimes it will not be possible to give a definitive answer
• think about other factors that could affect your metrics / poison your customer– customer or visitor sees > 1 experience
• kick visitors out of test during analysis when detected
– seasonal effects– site performance
• assign lower %ages of traffic if you are worried about downside effects of testing
poisoned tests: bad performance
• Your cells may ALL fail to convert– If your site or a cell is not performant, you
will dampen all possible resultsof AB tests
• Need a fair playing field for cells– Google: Speed Matters
• Need to have SOME traffic to get results– Make sure site doesn’t fail basic perf optimizations– Run through GooglePageSpeed or Y-Slow.
summary• Perform Visitor (and later
Customer) testing• Consider Online and Offline testing• Decide on metrics/facets business
cares about• Test clear understandable
hypotheses• Perform careful analysis with
statistics• Journal what was tested and the
results• Many online tools to assist – no
need for in-house inventions.
references• Online tools
– Free tool set– http://www.testsignificance.com/
• Companies:– optimizely.com – built by original product managers from Chrome browser, members of
Obama’s “smart” online fundraising team– foresee.com – surveys built in to your site– omniture.com – site tracking (expensive)
• Reading Articles– Basic mathematics guide
• Example case study showing low traffic ok
– Advanced mathemetics slideshow– AB Testing for Executives– “Stop running stupid tests” advice