Lean Analytics: How to find & test innovative Growth Hacks using Analytics

Lean Analytics

Workout

DAA Hub

Phil Pearce

April 2014

Fitness Consultant

linkedin.com/in/philpearce

Harder, Better, Faster, Stronger

Leaner!

Who have we got in the room?

1. Entrepreneur – Started company 2. Corporation – Work in big company 3. Agency – Help startups

Have you read any of these?

Quick Quiz

https://www.youtube.com/watch?v=usdeiJP7xh0 and http://prezi.com/lw-arulaenh4/copy-of-lean-startup-buzz-words/

FINISH

IyBITwC

TwCIyBi

Web Analytics Exchange

mentor 750 GA

questions answered

Tracking protection group

About Me

Phil Pearce Analytics Consultant

linkedin.com/in/philpearce

I`m not an entrepreneur

Apart from

this one

AdWords

But... I have done alot of agency consulting & I worked for some innovative startups

Sold for €16m

Pivoted

Changed business

model IPO in ~1yrs

Funded by

Gwyneth Paltrow

Sold for €37m

Crazy growth & IPO plans

IPO soon

Metrics Plan

Massive Revenue

understanding own sites digital value

to understand investments

Grew Taxi booking

Revenue by €10m in 2yrs

Intrapreneur & Technical marketer

1. Build PPC reporting platform MS access

2. Enabled KW level ROI bidding in 2007.

3. Managed £600K pm Adwords account & out-

performed market leader.

4. Built end-to-end affiliate tracking system.

5. Reverse engineered Adwords Algo.

6. Built mathematical ClickFraud detection tool for

mobile

7. Built free version of SpeedPPC

8. Building “4clicks” SaaS for Magneto (KPIs,

dataLayer, Dashboards, Remarketing -> all auto-

enabled)

...and closet growth hacker FINISH

... I have author-ed a book on Amazon

Agenda • Start: 9:30am-12:30am

• Introduce Lean Analytics terminology

– (e.g. MVP, Iterations, Agility)

• Explain why obsessing over the performance of one key metric is vital

• Describe the difference between website and product innovating and

testing?

• Look at some examples of successful (and unsuccessful) analytics hacks

• Develop a super analytics hack for your business

• Define a process for testing and refining your hacks

If you build it … they will come.

They will come ... if you build it

Because…

Fast Feedback = Build what customers want

Favourite Food

Most startups don’t know what their

customers will consume

(or what they are good at making)

Hotmail

database

company

Flickr

was going to

be an Video

Game platform

Twitter

podcasting

company

Autodesk

desktop

automation

Paypal

first built for

Palmpilots

Freshbooks

was invoicing

for a web

design firm

Wikipedia

was to be

written by

experts only

lawnmower

company

Fast iterations/sprints using…

Build > Measure > Learn BML

• (products)

Measure

• (data)

• (ideas)

Fast iterations/sprints using…

Build > Measure > Learn (repeat) BML

• (products)

Measure

• (data)

• (ideas)

Even the book uses lean

principles… 1. 5th edition in 8months (new

edition every built

1.5months!)

2. “We liked to hear from you”

feedback section in front &

online blog comments

encouraged.

3. Learnings have spawned

start-up conferences

Build > Measure > Learn

Measure

Problem:

We lie to ourselves

“We” are amazing!

Reality check…

…Analytics to the rescue

Analytics is the measurement of

movement towards your business

goals.

In a startup, the purpose of analytics is

to iterate to product/market fit

before the money runs out.

I have two coins.

Atleast one of them is heads.

What is the % probability

that the other is tails?

Guess…

2 of 3 (66%)

are tails.

Some fundamentals.

A good metric is:

Understandable

If you’re busy

explaining the

data, you won’t

be busy acting

on it.

Comparative

Comparison is

context.

A ratio or rate

The only way to

measure

change and roll

up the tension

between two

metrics

(Miles Per Hour)

Behavior

changing

If you’re busy

explaining the

data, you won’t

be busy acting

on it.

simplest

Not a good metric.

If metric won’t

change how you

behave, it’s…

Metrics help you know yourself.

You are

just like

Customers that

buy >1x in 90d

Your customers

will buy from you

Then you are

in this mode

Acquisition 70% of retailers

Once 1-15% Low acquisition

cost, high

checkout

Hybrid 20% of retailers

2-2.5 per year

15-30% Increasing return

rates, market share

Focus on

Loyalty 10% of retailers

>2.5 per year

>30% Loyalty, selection,

inventory size

(Thanks to Kevin Hillstrom for this.)

Qualitative

Unstructured, anecdotal,

revealing, hard to

aggregate, often too

positive & reassuring.

Warm and fuzzy.

Quantitative

Numbers and stats.

Hard facts, less insight,

easier to analyze; often

sour and disappointing.

Cold and hard.

Exploratory

Speculative. Tries to find

unexpected or

interesting insights.

Source of unfair

advantages.

Reporting

Predictable. Keeps you

abreast of the normal,

day-to-day operations.

Can be managed by

exception.

Necessary.

Rumsfeld on Analytics

(Or rather, Avinash Kaushik channeling Rumsfeld)

Things we

don’t

we know Are facts which may be wrong and

should be checked against data.

we don’t

Are questions we can answer by

reporting, which we should baseline

& automate.

we know Are intuition which we should

quantify and teach to improve

effectiveness, efficiency.

we don’t

Are exploration which is where

unfair advantage and interesting

epiphanies live.

A/B test:

Changing one thing

(i.e. color) and

measuring the

result (i.e. revenue.)

Apr Mar

Jan Feb

Segment:

Cross-sectional

comparison of all

people divided by

some attribute (age,

gender, etc.)

Slicing and dicing data 5,000

Cohort:

Comparison of

similar groups

along a timeline. (this is the April cohort)

Multivariate

analysis

Changing several

things at once to

see which correlates

with a result.

Which of these two companies

is doing better?

Is this company

growing or stagnating?

Which of these two companies has the best

Revenue/Customer?

January February March April May

Rev/customer $5.00 $ 4.50 $4.33 $4.25 $4.50

Cohort January February March April May Averages Cohort

group5 € 5.00 € 6.00 € 7.00 € 8.00 € 9.00 € 7.00

group4 € 3.00 € 4.00 € 6.00 € 7.00 € 5.00

group3 € 2.00 € 2.00 € 5.00 € 3.00

group2 € 1.00 € 1.00 € 1.00

group1 € 0.50 € 0.50

Lagging

Historical. Shows you

how you’re doing;

reports the news. Example: sales.

Explaining the

Leading

Forward-looking.

Number today that

predicts tomorrow;

reports the news. Example: pipeline.

Predicting the

future.

• A Facebook user reaching 7 friends within 10 days of signing up

(Chamath Palihapitiya)

• If someone comes back to Zynga a day after signing up for a game,

they’ll probably become an engaged, paying user

(Nabeel Hyatt)

• A Dropbox user who puts at least one file in one folder on one device

(ChenLi Wang)

• Twitter user following a certain number of people, and a certain

percentage of those people following the user back

(Josh Elman)

• A LinkedIn user getting to X connections in Y days (Elliot Schmukler)

Some examples

(From the 2012 Growth Hacking conference. http://growthhackersconference.com/)

Which means it’s time to talk

about correlation.

Number of Analysts

Chess StarTrek correlated Liked Maths

causal

Number of Analysts

Correlated vs Causal P2 E4

Correlated vs Causal

But it is not the cause!

Strong Correlation

Correlated

Two variables that are

related (but may be

dependent on

something else.)

Ice cream &

drowning.

Causal

An independent variable

that directly impacts a

dependent one.

Summertime &

drowning.

A leading, causal metric

is a superpower. h" p ://www.flickr.com/photos/bloke_with_camera/401812833/sizes/o/in/photostream/

Growth hacking, demystified.

correlation

causality

Optimize the

causal factor

Pick a metric

to change

Why is Nigerian spam so badly

written?

Aunshul Rege of Rutgers University, USA in 2009

Experienced scammers expect a “strike rate” of 1 or 2 replies per 1,000 messages

emailed; they expect to land 2 or 3 “Mugu” (fools) each week.

One scammer boasted “When you get a reply it’s 70% sure you’ll get the money” “By

sending an email that repels all but the most gullible,” says [Microsoft Researcher

Corman] Herley, “the scammer gets the most promising marks to self-select, and tilts

the true to false positive ratio in his favor.”

1000 emails

1-2 responses

1 fool and their money, parted.

Bad language (0.1% conversion)

Gullible (70% conversion)

1000 emails

100 responses

1 fool and their money, parted.

Good language (10% conversion)

Not-gullible (.07% conversion)

This would be horribly inefficient since

humans are involved.

Turns out the word “Nigeria” is the best

way to identify promising prospects.

Nigerian spammers

really understand their target market.

They see past vanity metrics.

The Lean Analytics framework.

Eric’s three engines of growth

Virality

Make people

invite friends.

How many they

tell, how fast they

tell them.

Spend money to

get customers.

Customers are

worth more than

they cost.

Stickiness

Keep people

coming back. Approach

Get customers

faster than you

lose them.

Math that

matters

@agatestudio

Lean Analytics Stages

Empathy • I’ve found a real, poorly-met need & reachable market faces

Stickiness • I’ve figured out how to solve the problem, in a way they will adore and pay for!

Virality • I’ve built the right product/features/functionality that keeps users around.

Revenue • The users and features fuel growth organically and artificially.

Scale • I’ve found a sustainable, scalable business with right margin in a healthy ecosystem.

1. Ecommerce

2. Two sided marketplace

3. SaaS

4. Mobile app

5. Media/Publishing

6. User generate content

Six business model types

Model + Stage = One Metric That Matters.

One Metric

That Matters.

The business you’re in

E-Com 2-Sided SaaS Mobile Media UCG

Empathy

Stickiness

Virality

Revenue

’re a

Really? Just one?

Yes, one!

Because… In a startup`s “focus" is

hard to achieve.

Having only one metric

resolves this problem.

www.theeastsiderla.com

Prevents distraction

Metrics are like squeeze toys.

http://www.flickr.com/photos/connortarter/4791605202/

Revenue stage:

CompareAndSave.com

(2-sided marketplace)

• Focus on one metric of CTR

• Reduced CPC

• Increased RPC (Effected of reverse economies of scale &

tiered cpa volumes)

• Marketplace: Consumers + Banks

Technically a “comparison engine”

Empathy

Stickiness

Virality

Revenue

commerce SaaS Media

Mobile

User-gen

content

2-sided

market

Loyalty,

conversion

CAC, shares,

reactivation

Transaction,

Affiliates,

white-label

Engagement,

Inherent

virality, CAC

Upselling,

CAC, CLV

API, magic #,

mktplace

Content,

Invites,

sharing

donations

Analytics,

user data

Inventory,

listings

SEM, sharing

Transactions,

commission

verticals

(Money from transactions)

Downloads,

churn, virality

WoM, app

ratings, CAC

ARPDAU

Spinoffs,

publishers

(Money from active users)

Traffic, visits,

returns

Content

virality, SEM

CPE, affiliate

%, eyeballs

Syndication,

licenses

(Money from ad clicks)

Workshop Task: 1. Select business type

(E-Com, 2-Sided, SaaS, Mobile, Media, UCG) 2. Determine Stage

(Empathy, Stickiness, Virality, Revenue, Scale) 3. Pick one metric 4. Set line in the sand (benchmark)

Useful sheet bit.ly/BigLeanTable

Other measurement models bit.ly/kpishake

What other metrics

do you want to know about?

Drawing some lines in the sand.

A company loses a quarter of its

customers every year.

Is this good or bad?

Baseline:

10% visitor engagement/day

30% of users/month use web or mobile app

10% of users/day use web or mobile app

1% of users/day use it concurrently

Fred Wilson’s social ratios

Baseline:

2-5% monthly churn

• The best SaaS get 1.5% - 3% a month. They have multiple Ph.D’s

on the job.

• Get below a 5% monthly churn rate before you know you’ve got a

business that’s ready to grow (Mark MacLeod) and around 2%

before you really step on the gas (David Skok)

• Last-ditch appeals and reactivation can have a big impact.

Facebook’s “don’t leave” reduces attrition by 7%.

Who is worth more?

Lifetime:

Roberto Medri, Etsy

Visits

The Lean Analytics cycle

Did we move the

needle?

Make changes

in production

Hypothesis

Design a test

Make changes

in production

Measure the

results

Success!

Pivot or give

Pick a KPI

Find a potential

improvement

Draw a line

With data: find

a commonality

Without data:

make a good

Draw a new line

Repeat test Did we

move the

needle?

Do AirBnB hosts

get more business

if their property is

professionally

photographed?

Gut instinct (hypothesis)

Professional photography helps AirBnB’s business

Candidate solution (MVP)

20 field photographers posing as employees

Measure the results

Compare photographed listings to a control group

Make a decision

Launch photography as a new feature for all hosts

5,000 shoots per month

by February 2012

Draw a new line

Pivot or

give up Find a potential

improvement Try again

Success!

Did we move the

needle?

Measure

the results

Make changes

in production

Design a test

Hypothesis

With data:

find a

commonality

Without data:

make a good

Draw a line Pick a KPI

“G e e, t ho s e

ho u se s t ha t d o

w e ll l o o k r e a l l y

n i c e.”

Ma y be i t ’s t he

c a m er a .

“C o m pu ter : Wha t

d o a l l t he

h ig hl y r e n te d

ho u se s ha v e i n

c o m m o n ?”

C a m er a m o d e l .

With data:

find a commonality

Without data: make a

good guess

Some non-tech

examples.

I lied. Everyone is a tech company.

http://www.flickr.com/photos/puuikibeach/4789015423

Cost of attention: way up.

http://www.flickr.com/photos/elcapitanbsc/3936927326

Cost of experiments:down.

Let’s pick on restaurants

for a while.

A line in the sand

Labor costs

Gross revenue

Just right

Understaffed?

Too costly?

A leading indicator

http://www.flickr.com/photos/avlxyz/4889656453 http://www.flickr.com/photos/mysticcountry/3567440970

50 reservations

at 5PM

250 covers

that night

(Varies by

restaurant.

McDonalds

≠ Fat Duck.)

http://www.flickr.com/photos/southbeachcars/6892880699

Restaurant MVP

Is tip amount a leading indicator of long-

term revenue?

Why does every table get the same

Is purple ink better? http://tippingresearch.com/uploads/managing_tips.pdf

Growth hacking

(is a word you should hate but will hear a lot about.)

Growth hacking, demystified.

correlation

causality

Optimize the

causal factor

Pick a metric

to change

Guerrilla

marketing

driven

learning

Subversiveness

GROWTH

HACKING

• A Facebook user reaching 7 friends within 10 days of signing up

(Chamath Palihapitiya)

• If someone comes back to Zynga a day after signing up for a game, they’ll

probably become an engaged, paying user (Nabeel Hyatt)

• A Dropbox user who puts at least one file in one folder on one device (ChenLi

• Twitter user following a certain number of people, and a certain percentage of

those people following the user back (Josh Elman)

• A LinkedIn user getting to X connections in Y days (Elliot Schmukler)

(These are also great segments to analyze.)

Leading indicators: Growth Hacks

Lean Analytics: How to find & test innovative Growth Hacks using Analytics

Data & Analytics

Branding Hacks - Laura Busche at the Lean Startup Conference 2014

Lean Analytics for Intrapreneurs by Allistair Croll

Analytics - Lean & Green

13 Fat Loss Hacks - formulatedfitness.s3. · PDF file13 Fat Loss Hacks A "Cheat Sheet" for sculpting a lean, sexy beach body

Lean Analytics and Engagement with CleverTap

Lean Analytics Roundtable - Breaking Barriers 2016

Lean Analytics by Marvin

5 Content Analytics Hacks to 10X Views and Shares

Lean Analytics - Basics, Video & 7 Take Aways

Lean Analytics Cycle

©2014 LEAN ANALYTICS ASSOCIATIONlean-analytics.org/wp-content/uploads/2016/09/LAA... · ©2014 LEAN ANALYTICS ASSOCIATION ! 2 Lean Product Development for competitive innovation

6 advanced google analytics hacks you need to know

Top SEO Hacks for Google Analytics- Nikki Rae - #BrightonSEO

Lean Startup Analytics/Metrics

7 Practical Tips on Lean Analytics

udemy - lean analytics cert

Lean UX Secrets: 5 UX Hacks to Build Better Experiences, Faster

Google Analytics Hacks

Lean analytics

Lean Analytics-sneak peek final