Lean Analytics: How to find & test innovative Growth Hacks using Analytics

  • Published on
    17-Jul-2015

  • View
    413

  • Download
    1

Embed Size (px)

Transcript

  • Lean Analytics

    Workout

    DAA Hub

    Phil Pearce

    April 2014

    Fitness Consultant

    linkedin.com/in/philpearce

  • Harder, Better, Faster, Stronger

    Leaner!

  • Who have we got in the room?

    1. Entrepreneur Started company 2. Corporation Work in big company 3. Agency Help startups

  • Have you read any of these?

  • Quick Quiz

    https://www.youtube.com/watch?v=usdeiJP7xh0 and http://prezi.com/lw-arulaenh4/copy-of-lean-startup-buzz-words/

  • START

    START

    GH

    PvT

    PMF

    LS

    Cdev

    FINISH

    FINISH

    FINISH

    IyBITwC

    TwCIyBi

    P2 E4

    MVP

    BML

    VM

  • Web Analytics Exchange

    mentor 750 GA

    questions answered

    Tracking protection group

    About Me

    Phil Pearce Analytics Consultant

    linkedin.com/in/philpearce

  • I`m not an entrepreneur

    Apart from

    this one

  • AdWords

    But... I have done alot of agency consulting & I worked for some innovative startups

    Sold for 16m

    Pivoted

    Changed business

    model IPO in ~1yrs

    Funded by

    Gwyneth Paltrow

    Sold for 37m

    Crazy growth & IPO plans

    IPO soon

    Metrics Plan

    Massive Revenue

    understanding own sites digital value

    to understand investments

    Grew Taxi booking

    Revenue by 10m in 2yrs

  • Intrapreneur & Technical marketer

    1. Build PPC reporting platform MS access

    2. Enabled KW level ROI bidding in 2007.

    3. Managed 600K pm Adwords account & out-

    performed market leader.

    4. Built end-to-end affiliate tracking system.

    5. Reverse engineered Adwords Algo.

    6. Built mathematical ClickFraud detection tool for

    mobile

    7. Built free version of SpeedPPC

    8. Building 4clicks SaaS for Magneto (KPIs,

    dataLayer, Dashboards, Remarketing -> all auto-

    enabled)

  • GH GH

    GH GH

    GH GH

    GH GH

    GH GH

    ...and closet growth hacker FINISH

    GH

  • ... I have author-ed a book on Amazon

  • Agenda Start: 9:30am-12:30am

    Introduce Lean Analytics terminology

    (e.g. MVP, Iterations, Agility)

    Explain why obsessing over the performance of one key metric is vital

    Describe the difference between website and product innovating and

    testing?

    Look at some examples of successful (and unsuccessful) analytics hacks

    Develop a super analytics hack for your business

    Define a process for testing and refining your hacks

  • If you build it they will come.

  • They will come ... if you build it

  • Because Fast Feedback = Build what customers want

    Favourite Food

  • Most startups dont know what their customers will consume

    (or what they are good at making)

    Hotmail

    was a

    database

    company

    Flickr

    was going to

    be an Video

    Game platform

    Twitter

    was a

    podcasting

    company

    Autodesk

    made

    desktop

    automation

    Paypal

    first built for

    Palmpilots

    Freshbooks

    was invoicing

    for a web

    design firm

    Wikipedia

    was to be

    written by

    experts only

    Mitel

    was a

    lawnmower

    company

  • Fast iterations/sprints using Build > Measure > Learn

    BML

    Build

    (products)

    Measure

    (data)

    Learn

    (ideas)

  • Fast iterations/sprints using Build > Measure > Learn (repeat)

    BML

    Build

    (products)

    Measure

    (data)

    Learn

    (ideas)

  • Even the book uses lean

    principles 1. 5th edition in 8months (new

    edition every built

    1.5months!)

    2. We liked to hear from you feedback section in front &

    online blog comments

    encouraged.

    3. Learnings have spawned

    start-up conferences

  • Build > Measure > Learn

    Measure

  • Problem:

    We lie to ourselves

    We are amazing!

  • Reality check

  • Analytics to the rescue

  • Analytics is the measurement of

    movement towards your business

    goals.

  • In a startup, the purpose of analytics is

    to iterate to product/market fit

    before the money runs out.

  • I have two coins.

    Atleast one of them is heads.

  • What is the % probability

    that the other is tails?

  • Guess

  • Tails

    Tails

    Heads

    Tails

    Tails

    Heads

    Heads

    Heads

  • Heads

    Tails

    Tails

    Heads

    Heads

    Heads

    2 of 3 (66%)

    are tails.

  • Some fundamentals.

  • A good metric is:

    Understandable

    If youre busy explaining the

    data, you wont be busy acting

    on it.

    Comparative

    Comparison is

    context.

    A ratio or rate

    The only way to

    measure

    change and roll

    up the tension

    between two

    metrics

    (Miles Per Hour)

    Behavior

    changing

    If youre busy explaining the

    data, you wont be busy acting

    on it.

  • simplest

    rule

    Not a good metric.

    If metric wont change how you

    behave, its

  • Metrics help you know yourself.

    You are

    just like

    Customers that

    buy >1x in 90d

    Your customers

    will buy from you

    Then you are

    in this mode

    Acquisition 70% of retailers

    Once 1-15% Low acquisition

    cost, high

    checkout

    Hybrid 20% of retailers

    2-2.5 per year

    15-30% Increasing return

    rates, market share

    Focus on

    Loyalty 10% of retailers

    >2.5 per year

    >30% Loyalty, selection,

    inventory size

    (Thanks to Kevin Hillstrom for this.)

  • Qualitative

    Unstructured, anecdotal,

    revealing, hard to

    aggregate, often too

    positive & reassuring.

    Warm and fuzzy.

    Quantitative

    Numbers and stats.

    Hard facts, less insight,

    easier to analyze; often

    sour and disappointing.

    Cold and hard.

  • Exploratory

    Speculative. Tries to find

    unexpected or

    interesting insights.

    Source of unfair

    advantages.

    Cool.

    Reporting

    Predictable. Keeps you

    abreast of the normal,

    day-to-day operations.

    Can be managed by

    exception.

    Necessary.

  • Rumsfeld on Analytics

    (Or rather, Avinash Kaushik channeling Rumsfeld)

    Things we

    know

    dont

    know

    we know Are facts which may be wrong and

    should be checked against data.

    we dont

    know

    Are questions we can answer by

    reporting, which we should baseline

    & automate.

    we know Are intuition which we should

    quantify and teach to improve

    effectiveness, efficiency.

    we dont

    know

    Are exploration which is where

    unfair advantage and interesting

    epiphanies live.

  • May

    A/B test:

    Changing one thing

    (i.e. color) and

    measuring the

    result (i.e. revenue.)

    Apr Mar

    0

    Jan Feb

    Segment:

    Cross-sectional

    comparison of all

    people divided by

    some attribute (age,

    gender, etc.)

    Slicing and dicing data 5,000

    Acti

    ve

    users

    Cohort:

    Comparison of

    similar groups

    along a timeline. (this is the April cohort)

    Multivariate

    analysis

    Changing several

    things at once to

    see which correlates

    with a result.

  • Which of these two companies

    is doing better?

  • Is this company

    growing or stagnating?

    Which of these two companies has the best

    Revenue/Customer?

    January February March April May

    Rev/customer $5.00 $ 4.50 $4.33 $4.25 $4.50

    Cohort January February March April May Averages Cohort

    group5 5.00 6.00 7.00 8.00 9.00 7.00

    group4 3.00 4.00 6.00 7.00 5.00

    group3 2.00 2.00 5.00 3.00

    group2 1.00 1.00 1.00

    group1 0.50 0.50

  • Lagging

    Historical. Shows you

    how youre doing; reports the news. Example: sales.

    Explaining the

    past.

    Leading

    Forward-looking.

    Number today that

    predicts tomorrow;

    reports the news. Example: pipeline.

    Predicting the

    future.

  • A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya)

    If someone comes back to Zynga a day after signing up for a game, theyll probably become an engaged, paying user (Nabeel Hyatt)

    A Dropbox user who puts at least one file in one folder on one device (ChenLi Wang)

    Twitter user following a certain number of people, and a certain percentage of those people following the user back

    (Josh Elman)

    A LinkedIn user getting to X connections in Y days (Elliot Schmukler)

    Some examples

    (From the 2012 Growth Hacking conference. http://growthhackersconference.com/)

  • Which means its time to talk about correlation.

  • Number of Analysts

    Chess StarTrek correlated Liked Maths

    causal

    Number of Analysts

    Correlated vs Causal P2 E4

  • Correlated vs Causal

    But it is not the cause!

    Strong Correlation

  • Correlated

    Two variables that are

    related (but may be

    dependent on

    something else.)

    Ice cream &

    drowning.

    Causal

    An independent variable

    that directly impacts a

    dependent one.

    Summertime &

    drowning.

  • A leading, causal metric

    is a superpower. h" p ://www.ickr.com/photos/bloke_with_camera/401812833/sizes/o/in/photostream/

  • Growth hacking, demystified.

    Find

    correlation

    Test

    causality

    Optimize the

    causal factor

    Pick a metric

    to change

  • Why is Nigerian spam so badly

    written?

  • Aunshul Rege of Rutgers University, USA in 2009

    Experienced scammers expect a strike rate of 1 or 2 replies per 1,000 messages emailed; they expect to land 2 or 3 Mugu (fools) each week.

    One scammer boasted When you get a reply its 70% sure youll get the money By

    sending an email that repels all but the most gullible, says [Microsoft Researcher

    Corman] Herley, the scammer gets the most promising marks to self-select, and tilts the true to false positive ratio in his favor.

    1000 emails

    1-2 responses

    1 fool and their money, parted.

    Bad language (0.1% conversion)

    Gullible (70% conversion)

    1000 emails

    100 responses

    1 fool and their money, parted.

    Good language (10% conversion)

    Not-gullible (.07% conversion)

    This would be horribly inecient since

    humans are involved.

  • Turns out the word Nigeria is the best way to identify promising prospects.

  • Nigerian spammers

    really understand their target market.

    They see past vanity metrics.

  • The Lean Analytics framework.

  • Erics three engines of growth

    Virality

    Make people

    invite friends.

    How many they

    tell, how fast they

    tell them.

    Price

    Spend money to

    get customers.

    Customers are

    worth more than

    they cost.

    Stickiness

    Keep people

    coming back. Approach

    Get customers

    faster than you

    lose them.

    Math that

    matters

  • @agatestudio

    Lean Analytics Stages

    Empathy Ive found a real, poorly-met need & reachable market faces

    Stickiness Ive figured out how to solve the problem, in a way they will adore and pay for!

    Virality Ive built the right product/features/functionality that keeps users around.

    Revenue The users and features fuel growth organically and artificially.

    Scale Ive found a sustainable, scalable business with right margin in a healthy ecosystem.

  • 1. Ecommerce

    2. Two sided marketplace

    3. SaaS

    4. Mobile app

    5. Media/Publishing

    6. User generate content

    Six business model types

  • Model + Stage = One Metric That Matters.

    One Metric

    That Matters.

    The business youre in

    E-Com 2-Sided SaaS Mobile Media UCG

    Empathy

    Stickiness

    Virality

    Revenue

    Scale

    Th

    e s

    tag

    e y

    ou

    re a

    t

  • Really? Just one?

  • Yes, one!

  • Because In a startup`s focus" is hard to achieve.

  • Having only one metric

    resolves this problem.

  • www.theeastsiderla.com

    Prevents distraction

  • Metrics are like squeeze toys.

    http://www.flickr.com/photos/connortarter/4791605202/

  • Revenue stage:

    CompareAndSave.com

    (2-sided marketplace)

    Focus on one metric of CTR

    Reduced CPC

    Increased RPC (Effected of reverse economies of scale &

    tiered cpa volumes)

    Marketplace: Consumers + Banks

    Technically a comparison engine

  • Empathy

    Stickiness

    Virality

    Revenue

    Scale

    E-

    commerce SaaS Media

    Mobile

    app

    User-gen

    content

    2-sided

    market

    Loyalty,

    conversion

    CAC, shares,

    reactivation

    Transaction,

    CLV

    Affiliates,

    white-label

    Engagement,

    churn

    Inherent

    virality, CAC

    Upselling,

    CAC, CLV

    API, magic #,

    mktplace

    Content,

    spam

    Invites,

    sharing

    Ads,

    donations

    Analytics,

    user data

    Inventory,

    listings

    SEM, sharing

    Transactions,

    commission

    Other

    verticals

    (Money from transactions)

    Downloads,

    churn, virality

    WoM, app

    ratings, CAC

    CLV,

    ARPDAU

    Spinoffs,

    publishers

    (Money from active users)

    Traffic, visits,

    returns

    Content

    virality, SEM

    CPE, affiliate

    %, eyeballs

    Syndication,

    licenses

    (Money from ad clicks)

  • Workshop Task: 1. Select business type

    (E-Com, 2-Sided, SaaS, Mobile, Media, UCG) 2. Determine Stage

    (Empathy, Stickiness, Virality, Revenue, Scale) 3. Pick one metric 4. Set line in the sand (benchmark)

    Useful sheet bit.ly/BigLeanTable

  • Other measurement models bit.ly/kpishake

  • What other metrics

    do you want to know about?

  • Drawing some lines in the sand.

  • A company loses a quarter of its

    customers every year.

    Is this good or bad?

  • Baseline:

    10% visitor engagement/day

    30% of users/month use web or mobile app

    10% of users/day use web or mobile app

    1% of users/day use it concurrently Fred Wilsons social ratios

  • Baseline:

    2-5% monthly churn

    The best SaaS get 1.5% - 3% a month. They have multiple Ph.Ds on the job.

    Get below a 5% monthly churn rate before you know youve got a business thats ready to grow (Mark MacLeod) and around 2% before you really step on the gas (David Skok)

    Last-ditch appeals and reactivation can have a big impact. Facebooks dont leave reduces attrition by 7%.

  • Who is worth more?

    Lifetime:

    $200

    Lifetime:

    $200

    Today

    A

    Roberto Medri, Etsy

    B

    Visits

  • The Lean Analytics cycle

  • Did we move the

    needle?

    Make changes

    in production

    Hypothesis

    Design a test

    Make changes

    in production

    Measure the

    results

    Success!

    Pivot or give

    up

    Pick a KPI

    Find a potential

    improvement

    Draw a line

    With data: find

    a commonality

    Without data:

    make a good

    guess

    Draw a new line

    Repeat test Did we

    move the

    needle?

  • Do AirBnB hosts

    get more business

    if their property is

    professionally

    photographed?

  • Gut instinct (hypothesis)

    Professional photography helps AirBnBs business

    Candidate solution (MVP)

    20 field photographers posing as employees

    Measure the results

    Compare photographed listings to a control group

    Make a decision

    Launch photography as a new feature for all hosts

  • 5,000 shoots per month

    by February 2012

  • Draw a new line

    Pivot or

    give up Find a potential

    improvement Try again

    Success!

    Did we move the

    needle?

    Measure

    the results

    Make changes

    in production

    Design a test

    Hypothesis

    With data:

    find a

    commonality

    Without data:

    make a good

    guess

    Draw a line Pick a KPI

  • G e e, t ho s e

    ho u se s t ha t d o

    w e ll l o o k r e a l l y

    n i c e.

    Ma y be i t s t he

    c a m er a .

    C o m pu ter : Wha t

    d o a l l t he

    h ig hl y r e n te d

    ho u se s ha v e i n

    c o m m o n ?

    C a m er a m o d e l .

    With data:

    find a commonality

    Without data: make a

    good guess

  • Some non-tech

    examples.

  • I lied. Everyone is a tech company.

  • http://www.flickr.com/photos/puuikibeach/4789015423

    Cost of attention: way up.

    http://www.flickr.com/photos/elcapitanbsc/3936927326

    Cost of experiments:down.

  • Lets pick on restaurants for a while.

  • A line in the sand

    Labor costs

    Gross revenue

    30%

    20%

    Just right

    Understaffed?

    = 24%

    Too costly?

  • A leading indicator

    http://www.flickr.com/photos/avlxyz/4889656453 http://www.flickr.com/photos/mysticcountry/3567440970

    50 reservations

    at 5PM

    250 covers

    that night

    (Varies by

    restaurant.

    McDonalds

    Fat Duck.)

  • http://www.flickr.com/photos/southbeachcars/6892880699

    Restaurant MVP

  • Is tip amount a leading indicator of long-

    term revenue?

  • Why does every table get the same

    menu?

  • Is purple ink better? http://tippingresearch.com/uploads/managing_tips.pdf

  • Growth hacking

    (is a word you should hate but will hear a lot a