[Elite Camp 2016] Craig Sullivan - Elite Camp Summary Session

The AB Testing Hype CycleBudapest - Netbooster – 11 Nov 2015

@OptimiseOrDie

It’s a Wrap!Elite Camp 2016

How was your trip to Estonia, my dear?

@OptimiseOrDie

Peep Laja

@OptimiseOrDie

• Peeeeep - A car sound

• Piip - Estonian Grocery Store

• Pap - Common blog problem

• Peahp - Texan pipe

• Peppie - Food for small dogs

• Payp - Micro payment system

World CRO Dude Dickens Character

PEP

PIP

The solution? Less Vowels!

@OptimiseOrDie

• Uusaastaöövastuvõtuhommikuidüll• Sünnipäevanädalalõpupeopärastlõunaväsimus

• Peep needs to lose a fucking vowel!

Pe¨p

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

Roundup Outline

@OptimiseOrDie

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

Roundup Outline

@OptimiseOrDie

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

1. This is F****ing Hard!

@OptimiseOrDie

• After 15 years, I see no easy shortcuts

• The tools, knowledge and techniques are better

• The foundations are still the same

• It takes dedication, perspiration, determination and skill!

“The boundary layer between the mind of the customer and your product is where all

the best stuff is always found."

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

2. Humility and Ignorance

@OptimiseOrDie

• Paul Rouke talked about this

• The best optimisers show humility and empathy

• This is one of the most important traits

• Everyone thinks they know the answer

• It’s good to admit you don’t know!

• Acquire a better quality of ignorance…

• Your customers are not the same

• Your site is not the same

• Your advertising and traffic aren’t the same

• Your UX is not the same

• Your X-Device Mix is not the same

• Use them to inform or suggest approaches

• They’re like the picture on meal packets

• “Serving Suggestion Only”

• There are obvious BEST PRACTICES but these are usually in the category of ‘bugs’ or ‘UX problems’ or ‘D’oh’

Just fix that now!

We have no clue

Reading Best Practice Bulls**t

@OptimiseOrDie

• Further reading:

bit.ly/BestPUnicornsbit.ly/BPSuccessbit.ly/BestpFAIL

Just fix that now!

We have no clue

Reading Best Practice Bulls**t

@OptimiseOrDie

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

Better Quality Ignorance

@OptimiseOrDie

• Let me explain!

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

3. Foundations of Data

@OptimiseOrDie

• Building layers of bullshit starts with collection!

• If you get the fundamentals wrong, all the metrics, reports and dashboards will be broken

• 3 years, 127 configurations = 122 were broken

• Even Google’s GA config was completely biased

• Get a health check!

• Speak to me about this service

• Get the basics right, then worry about fripperies!

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

4. Organisational Design

@OptimiseOrDie

• Recently working on Organisational Design

• Working with several large organisations

• Designing scalable growth models and teams

• Why does it nearly always fail?

• Because the entire method of building, improving and growing customer use of products HAS to change

• Without this, it’s another layer of bullshit

• Fundamental reworking is required

• Andre, the FT and Rockboost!

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

4. Organisational Design

@OptimiseOrDie

• Finnair slide

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

5. Failure is Great!

@OptimiseOrDie

• Embrace failure!

• Velocity (cycle time) and Volume (complete per month) are far more important than success rate

• 20% success rate on 1000 tests far better than 80% on 100!

• If you can’t handle being proved wrong or you can’t deal with counter intuitive results, change careers!

• Failure is an integral part of this work

• Everything around you & Scientific Experiments

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

6. Mature your metrics!

@OptimiseOrDie

• This came up repeatedly…

• Moving along the maturity curve from short term metrics towards longer term value

• For example, not just measuring conversion but repeat business, customer satisfaction scores, lifetime value

• If you are not continually challenging, maturing and improving your core metrics, you are standing still

• The impact of product change or testing may be deeper and longer term than you realise!

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

7. Prioritisation Scoring

@OptimiseOrDie

• If you aren’t prioritising, you’re wasting time

• If you are doing discovery for CRO properly, you will expose hundreds of issues, bugs, places to test and things to try!

• You have to prioritise around the effort vs. the potential reward

• There are many scoring systems (PIE, ICE, TIR) but I find that the best design is client specific

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

7. Prioritisation Scoring

@OptimiseOrDie

All my clients have core dimensions of:

• Effort / Cost / Difficulty

• Potential Uplift / Reward / ROI

Secondary dimensions can include:

• Time to market

• Risk of change

• Complexity or Dependency

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

8. Quant & Qual balance

@OptimiseOrDie

• You need Qualitative and Quantitative methods

• Data is great but it tells you what’s happening, where it’s happening and how big the issue is.

• It will not tell you the why

• You need qualitative tools (polls, surveys, feedback, UX research, interviews, session replay etc.)

• Arrogance comes from thinking you know the solution before you understand the problem

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

9. Motivation vs. Usability

@OptimiseOrDie

• Usability vs. Motivation?

• Seems fairly clear – work the motivation layer!

• There are 3 areas you can work on though:

• Motivation

• Usability

• Stuff that’s completely broken?

@OptimiseOrDie

• Clothing Brand 1M per month, 2 bugs

• European Travel site 2M per month, 13 bugs

• Telecoms operator 1.4M per month, 3 bugs

• Travel Aggregator 0.6M per month, 5 bugs

• Event and Ticket Site 2.5M per month, 11 bugs

• Total developer time to fix ALL of these = < 5 days!

• At a cost of £8000 to find and fix, the ROI was 91,150%

• In many cases, savings larger than the ENTIRE IT budget

How Much do these bugs cost?

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

10. You Need a F***ING PROCESS

@OptimiseOrDie

• Andre and others talked about this

• You need a process, methodology and org structure

• Tools and Enthusiasm are not enough

• You need to design your own

• Use an existing methodology and then customise

• Find your own way of working – beware of being a slave!

• Methodologies are good slaves but bad masters!

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

11. Psychology, Emotion, Copywriting

@OptimiseOrDie

• Thank you Amy!

• This is one of the secrets of CRO (not to Amy)

• You MUST have copywriting, persuasion and psychology skills embedded in or available to CRO teams

• Over 65% of my successes involve copy changes

• Words are the lens through which your brand, product, form, interaction or message is read.

• Your brand IS this copy and interactive element

• I love you Estonia

• Funny opener x 2

• Summary Slides

• Tool Slides

• Wrap

• See you next year!

12. Cross Device Optimisation (XDO)

@OptimiseOrDie

• One of my secrets that hardly anyone else uses

12. Cross Device Experience Map

MOBILE

iOS Resolution (Model)

Android Mobile Device Info

Windows Mobile Device Info

TABLET

iOSNOTHING

AndroidMobile Device Info

WindowsResolution (Model)

DESKTOP

Windows Mac Linux

Browser Version

Browser version

Browser version

12. CRO Grids

• An automated tool to pull a CRO grid from any site

• Identify key segments that are over and under performing

• Exploit Device and Browser knowledge to stem leakage

• Create dynamic ‘on the fly’ funnels using events, regex, segments

• A multi-dimensional automated funnel generator

• Uses 10 years of knowledge optimising sites

• Tuned and improved by all my optimisation buddies!

• Alpha is live – www.profitgrid.io

• Don’t complain about UX – tell me about the data!

• Any questions, feedback – email me sullivac@gmail.com

5. We’ve built a tool to do this!

Summary

@OptimiseOrDie

• Growth Optimisation is hard work Get used to it!

• Humility and Ignorance Be humble and accept your ignorance

• Data Foundations Regular health checks & collection checks

• Organisational Design If it isn’t changing you, it isn’t working

• Failure is good! Embrace and love failure

• Mature your metrics Signals for long term value and profit

Summary

@OptimiseOrDie

• Always be Prioritising Without scoring, you are burning rubber

• Qual and Quant balance A balance of where, how big and why!

• Motivation vs. Usability Functionality and Device comes FIRST!

• You need a PROCESS Borrow, Steal, Adapt & Optimise this

• Psychology, Emotion, Copywriting Copywriting is vital for CRO

• Cross Device Optimisation Read my article

Try out our free tool for pulling CRO summaries from any site:

www.profitgrid.io

Summary

@OptimiseOrDie

@OptimiseOrDie

Hacking efficiency/time/

delight/productivity/fun

Tell your friends good

Lovable Empathetic Delightful Self actualisation Life changing Rewiring

FUNCTIONAL

ACCESSIBLE

USABLE

USEFUL

PERSUASIVE

COMPELLING

TRANSFORMATIONAL

Device Browser Mobile Audience

Performance Accessibility Platforms

Comprehension Affordance Simplicity Relevance Clarity Context

Utility Tasks Goals Steps Completion Progression

Resonant Authentic Compelling Consistent Benefits Value Trust

Social Integration and Proof Habits Nudges Addictions Lifecycle Promotions Personalisation Brand

Belonging Trusted Emotive

@OptimiseOrDie

35

WE’RE ALL WINGING IT

Guessaholics Anonymous

Thank You!

Email me sullivac@gmail.com

Slides Will be sent to you!

LinkedIn linkd.in/pvrg14

Chris Stucchio

@OptimiseOrDie

@OptimiseOrDie

STATS FOR A/B

TESTING

The AB Testing Hype CycleBudapest - Netbooster – 11 Nov 2015

@OptimiseOrDie

It’s a Wrap!#CH2015

What About the Organisation?

@OptimiseOrDie

@OptimiseOrDie

Burn Down the Silos

• Non agile, non iterative design

• Silos work on product separately

• No ‘One Team’ per product/theme

• Large teams, unwieldy coordination

• Pass the product around

• More PMs and BAs than a conference

• Endless sucking signoff

• AB testing done the same way!

@OptimiseOrDie

FT, Booking, Skyscanner, ABN, Expedia + more

• Small teams (6-15) with direct access to publish

• Ability to set and get metrics data directly

• Tools, Autonomy, Lack of interference

• No Project Managers or Business Analysts

• Business defines ‘outcomes’ – teams deliver

• No long signoff chain

• No pesky meddling fools

• 18 Month projects over budget?

@OptimiseOrDie

10. FT Example

• 100s of releases a day!

• Iterative MVP approach

• Launch as alpha, beta, pilot, phased rollout

• Like getting in a shower

• Read more at labs.ft.com

CRO

47

@OptimiseOrDie

10. Positive Attributes

• Rapid, Iterative, User Centred & Agile Design. No Silos.

• Small empowered autonomous teams

• Polymaths and Overlap

• Toolkit & Analytics investment

• Persuasive copywriting & Psychology

• Great Testing & Optimisation Tools

@OptimiseOrDie

• Agile, Lean, Iterative x silo teams• Ability to get and set metrics• Autonomy, Control, Velocity• Iterative MVP approach• Work on outcomes, not features

!Burn Down the Silos!

50

Time

ROI

51

Summary Grid

@OptimiseOrDie

Do Your Research Prioritise Hypothesise

• Heuristic analysis• Usability testing• Heatmap tools• Session replay• Web Analytics• Qualitative surveys• Polls• Chat & Text analysis• Interviews• Device compatibility• Performance

• Use TIR or PIE• Prioritise around

opportunity vs. cost• Estimate political

difficulty or friction as well as resource costs

• There is no best practice on prioritisation

• Continually re-prioritise against new discoveries

• Use the collective brain• Assemble and circulate

your research• Set clear goals and have

a process for sketching• Sketch the Hypothesis• Build W/F or mockup• Set ONE primary

(possibly one secondary) success KPI

Summary Grid

@OptimiseOrDie

Pre Flight Checks Run Analyse & Report

• QA testing• Device & Browser

compatibility• Analytics tracking• Ecommerce tracking• Launch

• Set your test length with a calculator

• Run it, stop it, analyse it• Don’t run over time

unless your sample is smaller than expected

• Learn about the statistics

• Watch tests closely!

• Wash up analysis• Share findings widely• Don’t share useless stuff• What is useful across

the organisation?• Store in knowledgebase• Intellectual Property• Use as training!

DISCOVERY CYCLE

@OptimiseOrDie

Discover Ideas

Prioritise

Build & QA

Test or LiveMeasure

Data

Learn

TEST CYCLE

@OptimiseOrDie

Hypothesis Design

Sketch

WireFrame

Mockup / Prototype

Signoff (multiple)

Build

QA

Soft and Hard

Launch

Analyse & Publish

Learn

What did I learn personally?

@OptimiseOrDie

• That best practice is emerging

• It’s about how you can do this properly!

• Methodology & Frameworks – not tactics

• Shared mindset across CRO practitioners

• Stories beat statistics, Data beats opinion

• I always have lots to learn

• “Hold strong opinions, but loosely”

Go Forth and Optimise!

Email me sullivac@gmail.com

Slides Will be mailed to you!

Linkedin linkd.in/pvrg14