Upload
craig-sullivan
View
106
Download
0
Tags:
Embed Size (px)
DESCRIPTION
In this presentation, Craig shows the big mistakes people are making with their testing. If you're getting poor results, stuff that doesn't make sense or just a bunch of failed tests - this deck offers advice for all the mistakes to avoid. Craig has personally made all these f***ups and so humbly begs you to not do the same.
Citation preview
1
13 Easy split testing f***ups
@OptimiseOrDie
Top F***ups for 20131. Testing in the wrong place2. Your hypothesis inputs are crap3. No analytics integration4. Your test will finish after you die5. Not testing for long enough6. No QA for your split test7. Opportunities are not prioritised8. Testing cycles are too slow9. Your test fails10. The result is ‘about the same’11. Test flips or moves around12. Nobody ‘feels’ the test13. You forgot you were responsive
@OptimiseOrDie
@OptimiseOrDie
• UX and Analytics (1999)
• User Centred Design (2001)
• Agile, Startups, No budget (2003)
• Funnel optimisation (2004)
• Multivariate & A/B (2005)
• Conversion Optimisation (2005)
• Persuasive Copywriting (2006)
• Joined Twitter (2007)
• Lean UX (2008)
• Holistic Optimisation (2009)
Was : Group eBusiness Manager, BelronNow : Consulting
@OptimiseOrDie
Timeline
- 1998 1999 - 2004
2004-2008 2008-2012
#1 : You’re doing it in the wrong place
@OptimiseOrDie
#1 : You’re doing it in the wrong place
There are 4 areas a CRO expert always looks at:
1. Inbound attrition (medium, source, landing page, keyword, intent and many more…)
2. Key conversion points (product, basket, registration)3. Processes and steps (forms, logins, registration, checkout)4. Layers of engagement (search, category, product, add)
5. Use visitor flow reports for attrition – very useful.6. For key conversion points, look at loss rates & interactions7. Processes and steps – look at funnels or make your own8. Layers and engagement – make a model
Let’s look at an example I’ve used recently@OptimiseOrD
ie
Examples – Concept
Bounce
Engage
Outcome
@OptimiseOrDie
Examples – Shoprush.com
Bounce
Search or Category
Product Page
Add to basket
View basket
Checkout
Complete
@OptimiseOrDie
Examples – 16-25Railcard.co.uk
Bounce
Login to Account
Content Engage
Start Application
Type and Details
Eligibility
Photo
Complete
@OptimiseOrDie
6.3 – Examples – Guide Dogs
Bounce
Content Engage
Donation Pathway
Donation Page
Starts process
Funnel steps
Complete
@OptimiseOrDie
6.3 – Within a layer
Page 1
Page 2
Page 3
Page 4
Page 5
Exit
Deeper Layer
LikeContact
Wishlist
Micro Conversions
@OptimiseOrDie
#1 : You’re doing it in the wrong place
• Get to know the flow and loss (leaks) inbound, inside and through key processes or conversion points.
• Once you know the key steps you’re losing people at and how much traffic you have – make a money model.
• Let’s say 1,000 people see the page a month. Of those, 20% (200) convert to checkout.
• Estimate the influence your test can bring. How much money or KPI improvement would a 10% lift in the checkouts deliver?
• Congratulations – you’ve now built the worlds first IT plan with a return on investment estimate attached!
• I’ll talk more about prioritising later – but a good real world analogy for you to use:
@OptimiseOrDie
Think like a store owner!
If you can’t refurbish the entire store, which floors or departments will you invest in optimising?
Wherever there is:
• Footfall• Low return• Opportunity
@OptimiseOrDie
Insight - Inputs
#FAIL
Competitor copying
GuessingDice rolling
An article the CEO
read
Competitor change
Panic
Ego
OpinionCherished
notions Marketing whims Cosmic rays
Not ‘on brand’ enough
IT inflexibility
Internal company
needs
Some dumbass
consultant
Shiny feature
blindnessKnee jerk reactons
#2 : Your hypothesis is crap!
@OptimiseOrDie
Insight - Inputs
Insight
Segmentation
SurveysSales and
Call Centre
Session Replay
Social analytics
Customer contact
Eye tracking
Usability testing
Forms analytics Search
analytics Voice of Customer
Market research
A/B and MVT testing
Big & unstructured
data
Web analytics
Competitor evalsCustomer
services
#2 : These are the inputs you need…
@OptimiseOrDie
#2 : Solutions• You need multiple tool inputs
– Tool decks are here : www.slideshare.net/sullivac
• Usability testing and User facing teams– If you’re not doing these properly, you’re hosed
• Session replay tools provide vital input– Get vital additional customer evidence
• Simple page Analytics don’t cut it– Invest in your analytics, especially event tracking
• Ego, Opinion, Cherished notions – fill gaps– Fill these vacuums with insights and data
• Champion the user– Give them a chair at every meeting
@OptimiseOrDie
We believe that doing [A] for People [B] will make outcome [C] happen.
We’ll know this when we observe data [D] and obtain feedback [E]. (reverse)
@OptimiseOrDie
#3 : No analytics integration
• Investigating problems with tests• Segmentation of results• Tests that fail, flip or move around• Tests that don’t make sense• Broken test setups• What drives the averages you see?
@OptimiseOrDie
20
A B B A
These Danish porn sites are so hardcore!
We’re still waiting for
our AB tests to finish!
#4 : The test will finish after you die• Use a test length calculator like this one:• visualwebsiteoptimizer.com/ab-split-test-duration/
22
#5 : You don’t test for long enough• The minimum length
– 2 business cycles (comparison)– Always test ‘whole’ not partial cycles– Don’t self stop!– Usually a week, 2 weeks, Month– Be aware of multiple cycles
• How long after that– 95% confidence or higher is my aim (p values are unreliable)– I aim for a minimum 250 outcomes, ideally 350+ for each ‘creative’– If you test 4 recipes, that’s 1400 outcomes needed– You should have worked out how long each batch of 350 needs before you start!– If you segment, you’ll need more data – It may need a bigger sample if the response rates are similar*– Use a test length calculator but be aware of minimums– Important insider tip – watch the error bars! The +/- stuff – let’s explain
* Stats geeks know I’m glossing over something here. That test time depends on how the two experiments separate in terms of relative performance as well as how volatile the test response is. I’ll talk about this when I record this one! This is why testing similar stuff sux.
#2 : The tennis court– Let’s say we want to estimate, on average, what height Roger Federer
and Nadal hit the ball over the net at. So, let’s start the match:
@OptimiseOrDie
First Set Federer 6-4– We start to collect values
62cm+/- 2cm
63.5cm+/- 2cm
@OptimiseOrDie
Second Set – Nadal 7-6– Nadal starts sending them low over the net
62cm+/- 1cm
62.5cm+/- 1cm
@OptimiseOrDie
Final Set Nadal 7-6– We start to collect values
61.8cm+/- .3cm
62cm+/- .3cm
Let’s look at this a different way
62.5cm+/- 1cm
@OptimiseOrDie
9.1% ± 0.3
9.3% ± 0.3
62.5cm+/- 1cm
@OptimiseOrDie
9.1% ± 0.5
9.3% ± 0.5
9.1% ± 0.2
9.3% ± 0.2
9.1% ± 0.1
9.3% ± 0.1
Graph is a range, not a line:
9.1 ± 0.3%
#5 : Summary• The minimum length:
– 2 business cycles minimum, regardless of outcomes– 250+, prefer 350+ outcomes in each– 95%+ confidence – Error bar separation between creatives
• Pay attention to:– Time it will take for the number of ‘recipes’ in the test– The actual footfall to the test – not sitewide numbers– Test results that don’t separate – makes the test longer– This is why you need brave tests – to drive difference– The error bars – the numbers in your AB testing tool are not precise –
they’re fuzzy regions that depend on response and sample size.– Sudden changes in test performance or response– Monitor early tests like a chef!
@OptimiseOrDie
#6 : No QA testing for the AB
test?
#6 - QA Test or Die!Email testing www.litmus.com
www.returnpath.comwww.lyris.com
Browser testing www.crossbrowsertesting.comwww.browserstack.comwww.spoon.netwww.cloudtesting.comwww.multibrowserviewer.comwww.saucelabs.com
Mobile devices www.perfectomobile.comwww.deviceanywhere.comwww.mobilexweb.com/emulatorswww.opendevicelab.com
@OptimiseOrDie
#6 : What QA testing should I do?
• Cross Browser Testing• Testing from several locations (office, home, elsewhere)• Testing the IP filtering is set up• Test tags are firing correctly (analytics and the test tool)• Test as a repeat visitor and check session timeouts• Cross check figures from 2+ sources • Monitor closely from launch, recheck, watch• WATCH FOR BIAS!
@OptimiseOrDie
#7 : Opportunities are not prioritised
Once you have a list of potential test areas, rank them by opportunity vs. effort.
The common ranking metrics that I use include:
• Opportunity (revenue, impact)
• Dev resource• Time to market • Risk / Complexity
Make yourself a quadrant diagram and plot them
#8 : Your cycles are too slow
0 6 12 18
Months
Conversion
@OptimiseOrDie
#8 : Solutions• Give Priority Boarding for opportunities
– The best seats reserved for metric shifters
• Release more often to close the gap– More testing resource helps, analytics ‘hawk eye’
• Kaizen – continuous improvement– Others call it JFDI (just f***ing do it)
• Make changes AS WELL as tests, basically!– These small things add up
• RUSH Hair booking – Over 100 changes– No functional changes at all – 37% improvement
• Inbetween product lifecycles?– The added lift for 10 days work, worth 360k
@OptimiseOrDie
#8 : Make your own cycles“Rather than try and improve one thing by 10% - which would be very, very difficult to do,
We go and find 1,000 things and improve them all by a fraction of a per cent, which is totally do-able.”Chris Boardman
@OptimiseOrDie
38
#9 : Your test fails
@OptimiseOrDie
#9 : Your test fails• Learn from the failure! If you can’t learn from the failure, you’ve
designed a crap test. • Next time you design, imagine all your stuff failing. What would you
do? If you don’t know or you’re not sure, get it changed so that a negative becomes insightful.
• So : failure itself at a creative or variable level should tell you something.• On a failed test, always analyse the segmentation and analytics• One or more segments will be over and under• Check for varied performance• Now add the failure info to your Knowledge Base:• Look at it carefully – what does the failure tell you? Which element do
you think drove the failure?• If you know what failed (e.g. making the price bigger) then you have
very useful information• You turned the handle the wrong way• Now brainstorm a new test
@OptimiseOrDie
#10 : The test is ‘about the same’• Analyse the segmentation• Check the analytics and instrumentation• One or more segments may be over and under• They may be cancelling out – the average is a lie• The segment level performance will help you (beware of
small sample sizes)• If you genuinely have a test which failed to move any
segments, it’s a crap test – be bolder• This usually happens when it isn’t bold or brave enough in
shifting away from the original design, particularly on lower traffic sites
• Get testing again!
@OptimiseOrDie
• There are three reasons it is moving around– Your sample size (outcomes) is still too small– The external traffic mix, customers or reaction has
suddenly changed or – Your inbound marketing driven traffic mix is
completely volatile (very rare)
• Check the sample size• Check all your marketing activity• Check the instrumentation• If no reason, check segmentation
#11 : The test keeps moving around
@OptimiseOrDie
• Something like this can happen:
• Check your sample size. If it’s still small, then expect this until the test settles.
• If the test does genuinely flip – and quite severely – then something has changed with the traffic mix, the customer base or your advertising. Maybe the PPC budget ran out? Seriously!
• To analyse a flipped test, you’ll need to check your segmented data. This is why you have a split testing package AND an analytics system.
• The segmented data will help you to identify the source of the shift in response to your test. I rarely get a flipped one and it’s always something changing on me, without being told. The heartless bastards.
#11 : The test has flipped on me
#12 : Nobody feels the test
• You promised a 25% rise in checkouts - you only see 2%• Traffic, Advertising, Marketing may have changed• Check they’re using the same precise metrics• Run a calibration exercise• I often leave a 5 or 10% stub running in a test• This tracks old creative once new one goes live• If conversion is also down for that one, BINGO!• Remember – the AB test is an estimate – it doesn’t
precisely record future performance• This is why infrequent testing is bad• Always be trying a new test instead of basking in the
glory of one you ran 6 months ago. You’re only as good as your next test.
@OptimiseOrDie
#13 : You forgot about Mobile & Tablet
• If you’re AB testing a responsive site, pay attention• Content will break differently on many screens• Know thy users and their devices• Use bango or google analytics to define a test list• Make sure you test mobile devices & viewports• What looks good on your desk may not be for the user• Harder to design cross device tests• You’ll need to segment mobile, tablet & desktop response
in the analytics or AB testing package• Your personal phone is not a device mix• Ask me about making your device list• Buy core devices, rent the rest from deviceanywhere.com
@OptimiseOrDie
Top F***ups for 20131. Testing in the wrong place2. Your hypothesis inputs are crap3. No analytics integration4. Your test will finish after you die5. Not testing for long enough6. No QA for your split test7. Opportunities are not prioritised8. Testing cycles are too slow9. Your test fails10. The result is ‘about the same’11. Test flips or moves around12. Nobody ‘feels’ the test13. You forgot you were responsive
@OptimiseOrDie
46
Is there a way to fix this then?Conversion Heroes!
@OptimiseOrDie
47
:@OptimiseOrDie
:linkd.in/pvrg14
Get in touch! Slides here : slidesha.re/1jGTQ0n
Best of Twitter
@OptimiseOrDie
@danbarker Analytics@fastbloke Analytics@timlb Analytics@jamesgurd Analytics@therustybear Analytics@carmenmardiros Analytics@davechaffey Analytics@priteshpatel9 Analytics@cutroni Analytics@avinash Analytics@AschottmullerAnalytics, CRO@cartmetrix Analytics, CRO@Kissmetrics CRO / UX@Unbounce CRO / UX@Morys CRO / Neuro@UXFeeds UX / Neuro@Psyblog Neuro@Gfiorelli1 SEO / Analytics
@PeepLaja CRO@TheGrok CRO@UIE UX@LukeW UX / Forms@cjforms UX / Forms@axbom UX@iatv UX@Chudders Photo UX@JeffreyGroks Innovation@StephanieRieger Innovation@BrianSolis Innovation@DrEscotet Neuro@TheBrainLadyNeuro@RogerDooley Neuro@Cugelman Neuro@Smashingmag Dev / UX@uxmag UX@Webtrends UX / CRO
Best of the Web
@OptimiseOrDie
Whichtestwon.comUnbounce.comKissmetrics.comUxmatters.comRogerDooley.comPhotoUX.comTheTeamW.comBaymard.comLukew.comPRWD.comMeasuringusability.comConversionXL.comSmartinsights.comEconsultancy.comCutroni.com
www.GetMentalNotes.com
Best of Books
@OptimiseOrDie
51
Some CRO and Testing resources• 101 Landing page tips : slidesha.re/8OnBRh • 544 Optimisation tips : bit.ly/8mkWOB• 108 Optimisation tips : bit.ly/3Z6GrP• 32 CRO tips : bit.ly/4BZjcW• 57 CRO books : bit.ly/dDjDRJ• CRO article list : bit.ly/nEUgui• Smashing Mag article : bit.ly/8X2fLk