Transcript

Lean Analytics

DCUApril, 2014

@acroll

Some housekeeping.

Don’t sell what you can make. Make what you can sell.

Kevin Costner is a lousy entrepreneur.

The core of Lean is iteration.

Most startups don’t know what they’ll be when they grow up.

Hotmailwas a database company

Flickrwas going to be an MMO

Twitterwas a podcasting company

Autodeskmade desktop automation

Paypalfirst built for Palmpilots

Freshbookswas invoicing for a web design firm

Wikipediawas to be written by experts only

Mitelwas a lawnmower company

Unfortunately,we’re all liars.

Everyone’s idea is the best right?

People love this part!

(but that’s not always a good thing)

This is where things fall apart.

No data, no learning.

Analytics can help.

Analytics is the measurement of movement towards your business

goals.

In a startup, the purpose of analytics is to iterate to product/market fit

before the money runs out.

I have two kids.At least one of them is a girl.

What are the chancesthe other is a boy?

BB BG

GB GG

2 of 3 (66%) are boys.

GB GG BG

Some fundamentals.

A good metric is:

Understandable

If you’re busy explaining the data, you won’t be busy acting on it.

Comparative

Comparison is context.

A ratio or rate

The only way to measure change and roll up the tension between two metrics (MPH)

Behaviorchanging

If you’re busy explaining the data, you won’t be busy acting on it.

Thesimplestrule

badmetric.

If a metric won’t change how you behave, it’s a

h"p://www.flickr.com/photos/circasassy/7858155676/

Metrics help you know yourself.

Acquisition

Hybrid

Loyalty

70%of retailers

20%of retailers

10%of retailers

You are just like

Customers that buy >1x in 90d

Once

2-2.5per year

>2.5per year

Your customers will buy from you

Then you are in this mode

1-15%

15-30%

>30%

Low acquisition cost, high checkout

Increasing return rates, market share

Loyalty, selection, inventory size

Focus on

(Thanks to Kevin Hillstrom for this.)

Qualitative

Unstructured, anecdotal, revealing, hard to aggregate, often too positive & reassuring.

Warm and fuzzy.

Quantitative

Numbers and stats. Hard facts, less insight, easier to analyze; often sour and disappointing.

Cold and hard.

Exploratory

Speculative. Tries to find unexpected or interesting insights. Source of unfair advantages.

Cool.

Reporting

Predictable. Keeps you abreast of the normal, day-to-day operations. Can be managed by exception.

Necessary.

Rumsfeld on Analytics

(Or rather, Avinash Kaushik channeling Rumsfeld)

Things we

know

don’tknow

we know Are facts which may be wrong and should be checked against data.

we don’tknow

Are questions we can answer by reporting, which we should baseline & automate.

we knowAre intuition which we should quantify and teach to improve effectiveness, efficiency.

we don’tknow

Are exploration which is where unfair advantage and interesting epiphanies live.

MayAprMarFeb

Slicing and dicing data

Jan

0

5,000

Activ

e use

rs

Cohort:Comparison of similar groups along a timeline.(this is the April cohort)

A/B test:Changing one thing (i.e. color) and measuring the result (i.e. revenue.)

MultivariateanalysisChanging several things at once to see which correlates with a result.

☀☁☀☁

Segment:Cross-sectional

comparison of all people divided by

some attribute (age, gender, etc.)

Which of these two companiesis doing better?

  January February March April May

Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50Is this company growing or stagnating?

Cohort 1 2 3 4 5

January

February

March

April

May

$5 $3 $2 $1 $0.5

$6 $4 $2 $1

$7 $6 $5

  $8 $7

      $9

How about this one?

Cohort 1 2 3 4 5

January

February

March

April

May

Averages

$5 $3 $2 $1 $0.5

$6 $4 $2 $1  

$7 $6 $5    

$8 $7      

$9        

$7 $5 $3 $1 $0.5

Look at the same data in cohorts

Lagging

Historical. Shows you how you’re doing; reports the news. Example: sales.

Explaining the past.

Leading

Forward-looking. Number today that predicts tomorrow; reports the news. Example: pipeline.

Predicting the future.

A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya)

If someone comes back to Zynga a day after signing up for a game, they’ll probably become an engaged, paying user (Nabeel Hyatt)

A Dropbox user who puts at least one file in one folder on one device (ChenLi Wang)

Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman)

A LinkedIn user getting to X connections in Y days (Elliot Schmukler)

Some examples

(From the 2012 Growth Hacking conference. http://growthhackersconference.com/)

Which means it’s time to talk about correlation.

1

10

100

1000

10000

Ice cream consumption DrowningsJan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec

Correlated

Two variables that are related (but may be dependent on something else.)

Ice cream & drowning.

Causal

An independent variable that directly impacts a dependent one.

Summertime & drowning.

A leading, causal metricis a superpower.

h"p://www.flickr.com/photos/bloke_with_camera/401812833/sizes/o/in/photostream/

Growth hacking, demystified.

Find correlation

Test causality

Optimize the causal factor

Pick a metric to change

Is social action a leading indicator of donation?

http

://bl

og.ju

stgi

ving.

com

/nine

-reas

ons-

why

-soc

ial-a

nd-m

obile

-are

-the-

futu

re-o

f-fun

drais

ing/

Is mobile use?ht

tp://

blog

.just

givin

g.co

m/n

ine-re

ason

s-w

hy-s

ocial

-and

-mob

ile-a

re-th

e-fu

ture

-of-f

undr

aising

/

Why is Nigerian spam so badly written?

Aunshul Rege of Rutgers University, USA in 2009

Experienced scammers expect a “strike rate” of 1 or 2 replies per 1,000 messages emailed; they expect to land 2 or 3 “Mugu” (fools) each week.One scammer boasted “When you get a reply it’s 70% sure you’ll get the money”“By sending an email that repels all but the most gullible,” says [Microsoft Researcher Corman] Herley, “the scammer gets the most promising marks to self-select, and tilts the true to false positive ratio in his favor.”

1000 emails

1-2 responses

1 fool and their money, parted.

Bad language (0.1% conversion)

Gullible (70% conversion)

1000 emails

100 responses

1 fool and their money, parted.

Good language (10% conversion)

Not-gullible (.07% conversion)

This would be horribly inefficient since

humans are involved.

Turns out the word “Nigeria” is the best way to identify promising prospects.

Nigerian spammersreally understand their target market.

They see past vanity metrics.

The Lean Analytics framework.

Eric’s three engines of growth

Virality

Make people invite friends.

How many they tell, how fast they

tell them.

Price

Spend money to get customers.

Customers are worth more than

they cost.

Stickiness

Keep people coming back.

Approach

Get customers faster than you

lose them.

Math that matters

Dave’s Pirate MetricsAARRR

AcquisitionHow do your users become aware of you?

SEO, SEM, widgets, email, PR, campaigns, blogs ...

ActivationDo drive-by visitors subscribe, use, etc?

Features, design, tone, compensation, affirmation ...

RetentionDoes a one-time user become engaged?

Notifications, alerts, reminders, emails, updates...

RevenueDo you make money from user activity?

Transactions, clicks, subscriptions, DLC, analytics...

ReferralDo users promote your product?

Email, widgets, campaigns, likes, RTs, affiliates...

Stage

EMPATHY I’ve found a real, poorly-met need that a reachable market faces.

STICKINESS I’ve figured out how to solve the problem in a way they will keep using and pay for.

VIRALITY I’ve found ways to get them to tell their friends, either intrinsically or through incentives.

REVENUE The users and features fuel growth organically and artificially.

SCALE I’ve found a sustainable, scalable business with the right margins in a healthy ecosystem.

GateTh

e fiv

e st

ages

Empathy stage:Localmind hacks Twitter

Needed to find out if a core assumption—strangers answering questions—was valid.Ran Twitter experiment instead of writing codeAsked senders of geolocated Tweets from Times Square random questions; counted response rateConclusion: high enough to proceed

Stickiness stage:qidiq streamlines invites

Survey owner adds recipient to groupSurvey owner asks question

Recipient reads survey questionRecipient responds to questionRecipient sees survey results

(Later, if needed…)Recipient visits site; no password!Recipient does password recovery

One-time link sent to emailRecipient creates password

Recipient can edit profile, etc.

Survey owner adds recipient to group

Survey owner asks question

Recipient gets invite

Recipient reads survey question

Recipient responds to question

Recipient installs mobile app

Recipient creates account, profile

Recipient sees survey results

Recipient can edit profile, etc.

10-2

5% R

ESPO

NSE R

ATE

70-9

0% R

ESPO

NSE R

ATE

Six business model archetypes(Yours is probably a blend of these.)

E-commerce SaaS (freemium?) Mobile app (gaming) Two sided marketplace Media User generated content

(Which means eye charts like these.)

Customer Acquisition Cost

paid direct search wom inherent virality

VISITOR

Freemium/trial offer

Enrollment

User

Disengaged User

Cancel

Freemium churn

Engaged User

Free user disengagement

Reactivate

Cancel

Trial abandonment rate

Invite Others

Paying Customer

Reactivationrate

Paid conversion

FORMER USERS

User Lifetime Value

Reactivate

FORMER CUSTOMERS

Customer Lifetime Value

Viral coefficientViral rate

Resolution

Support data

Account Cancelled Billing Info Exp.

Paid Churn Rate

Tiering

Capacity Limit

Upselling rate Upselling

Disengaged DissatisfiedTrial Over

Model + Stage = One Metric That Matters.

One MetricThat Matters.

The business you’re in

E-Com SaaS Mobile 2-Sided Media UCGEmpathy

Stickiness

Virality

Revenue

ScaleThe

stag

e yo

u’re

at

Really? Just one?

Yes, one.

In a startup, focus is hard to achieve.

Having only one metricaddresses this problem.

Moz cuts down on metricsSaaS-based SEO toolkit in the scale stage. Focused on net adds.

Was a marketing campaign successful? Were customer complaints lowered? Was a product upgrade valuable?

Net adds up:

Can we acquire more valuable customers? What product features can increase engagement? Can we improve customer support?

Net adds flat:

Are the new customers not the right segment? Did a marketing campaign fail? Did a product upgrade fail somehow? Is customer support falling apart?

Net adds down:

Metrics are like squeeze toys.

http://www.flickr.com/photos/connortarter/4791605202/

Empathy

Stickiness

Virality

Revenue

Scale

E-commerce SaaS MediaMobile

appUser-gencontent

2-sidedmarket

Interviews; qualitative results; quantitative scoring; surveys

Loyalty, conversion

CAC, shares, reactivation

Transaction, CLV

Affiliates, white-label

Engagement, churn

Inherent virality, CAC

Upselling, CAC, CLV

API, magic #, mktplace

Content, spam

Invites, sharing

Ads, donations

Analytics, user data

Inventory, listings

SEM, sharing

Transactions, commission

Other verticals

(Money from transactions)

Downloads, churn, virality

WoM, app ratings, CAC

CLV, ARPDAU

Spinoffs, publishers

(Money from active users)

Traffic, visits, returns

Content virality, SEM

CPE, affiliate %, eyeballs

Syndication, licenses

(Money from ad clicks)

Better: bit.ly/BigLeanTable

What other metricsdo you want to know about?

Drawing some lines in the sand.

A company loses a quarter of its customers every year.

Is this good or bad?

Not knowing what normal ismakes you do stupid things.

Baseline:5-7% growth a week

“A good growth rate during YC is 5-7% a week,” he says. “If you can hit 10% a week you're doing exceptionally well. If you can only manage 1%, it's a sign you haven't yet figured out what you're doing.” At revenue stage, measure growth in revenue. Before that, measure growth in active users.

Paul Graham, Y Combinator

• Are there enough people who really care enough to sustain a 5% growth rate?

• Don’t strive for a 5% growth at the expense of really understanding your customers and building a meaningful solution

• Once you’re a pre-revenue startup at or near product/market fit, you should have 5% growth of active users each week

• Once you’re generating revenues, they should grow at 5% a week

Baseline:10% visitor engagement/day

Fred Wilson’s social ratios

30% of users/month use web or mobile app

10% of users/day use web or mobile app

1% of users/day use it concurrently

Baseline:2-5% monthly churn• The best SaaS get 1.5% - 3% a month. They have multiple Ph.D’s

on the job.• Get below a 5% monthly churn rate before you know you’ve got a

business that’s ready to grow (Mark MacLeod) and around 2% before you really step on the gas (David Skok)

• Last-ditch appeals and reactivation can have a big impact. Facebook’s “don’t leave” reduces attrition by 7%.

Who is worth more?

Today

A Lifetime:$200

Roberto Medri, Etsy

B Lifetime:$200

Visits

Baseline:Calculating customer lifetime

25%monthly churn

100/25=4The average

customer lasts 4 months

5%monthly churn

100/5=20The average

customer lasts 20 months

2%monthly churn

100/2=50The average

customer lasts 50 months

Baseline:CAC under 1/3 of CLV• CLV is wrong. CAC Is probably wrong, too.• Time kills all plans: It’ll take a long time to find

out whether your churn and revenue projections are right

• Cashflow: You’re basically “loaning” the customer money between acquisition and CLV.

• It keeps you honest: Limiting yourself to a CAC of only a third of your CLV will forces you to verify costs sooner.

Lifetime of 20 mo.$30/mo. per

customer$600 CLV

$200 CACNow segment those users!

1/3 spend

The Lean Analytics cycle

Draw a new linePivot orgive up

Try again

Success!

Did we move the needle?

Measure the results

Make changes in production

Design a test

Hypothesis

With data:find a

commonality

Without data: make a good

guess

Find a potential improvement

Draw a linePick a KPI

Do AirBnB hosts get more business if their property is professionally photographed?

Gut instinct (hypothesis)Professional photography helps AirBnB’s business

Candidate solution (MVP)20 field photographers posing as employees

Measure the resultsCompare photographed listings to a control group

Make a decision Launch photography as a new feature for all hosts

5,000 shoots per month by February 2012

Hang on a second.

Gut instinct (hypothesis)Professional photography helps AirBnB’s business

SRSLY?

Draw a new linePivot orgive up

Try again

Success!

Did we move the needle?

Measure the results

Make changes in production

Design a test

Hypothesis

With data:find a

commonality

Without data: make a good

guess

Find a potential improvement

Draw a linePick a KPI

“Gee, those houses that do well look really

nice.”

Maybe it’s the camera.

“Computer: What do all the

highly rented houses have in

common?”

Camera model.

With data:find a commonality

Without data: make a good guess

Landing page design A/B testing

Cohort analysis General analytics

URL shortening

Funnel analytics

Influencer Marketing

Publisher analytics

SaaS analytics

Gaming analytics

User interaction Customer satisfaction KPI dashboardsUser segmentation

User analytics Spying on users

Some non-tech examples.

I lied. Everyone is a tech company.

http://www.flickr.com/photos/puuikibeach/4789015423 http://www.flickr.com/photos/elcapitanbsc/3936927326

Cost of experiments: down. Cost of attention: way up.

Let’s pick on restaurants for a while.

A line in the sand

Labor costs

Gross revenue

30%

24%

20%

Too costly?

Just right

Understaffed?

=

A leading indicator

http://www.flickr.com/photos/avlxyz/4889656453http://www.flickr.com/photos/mysticcountry/3567440970

50 reservationsat 5PM

250 coversthat night

(Varies by restaurant. McDonalds ≠ Fat Duck.)

http://www.flickr.com/photos/southbeachcars/6892880699

Restaurant MVP

Is tip amount a leading indicator of long-term revenue?

Why does every table get the same menu?

Is purple ink better?http://tippingresearch.com/uploads/managing_tips.pdf

Growth hacking

(is a word you should hate but will hear a lot about.)

Growth hacking, demystified.

Find correlation

Test causality

Optimize the causal factor

Pick a metric to change

Guerrillamarketing

Data-drivenlearning

Subversiveness

GROWTHHACKING

A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya)

If someone comes back to Zynga a day after signing up for a game, they’ll probably become an engaged, paying user (Nabeel Hyatt)

A Dropbox user who puts at least one file in one folder on one device (ChenLi Wang)

Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman)

A LinkedIn user getting to X connections in Y days (Elliot Schmukler)

(from the 2012 Growth Hacking conference) (These are also great segments to analyze.)

The leading indicator

• Growth hacking is simply what marketing should have been doing, but it fell in love with Don Draper and opinions along the way

• Optimize a factor you think is correlated with growth

The growth hack

AirBnB and Craigslist

What if you’re ina big organization?

The Zero Overhead principle

A central theme to this new wave of innovation is the application of core product tenets from the consumer space to the enterprise.In particular, a universal lesson that I keep sharing with all entrepreneurs building for the enterprise is the Zero Overhead Principle: no feature may add training costs to the user.

DJ Patil

The B2B stereotype

http

://w

ww.

tech

dige

st.tv

/200

7/02

/im_a

_pc_

im_a

_ma.

htm

l

Domainexpert

Disruptionexpert

•Domain expert knows industry and the problem domain. Has a Rolodex; proxy for customers.

•Disruption expert knows tech that will produce a change Sees beyond the current model.

The Lean Analytics lifecycle of an Intrapreneur

Empathy Consulting to test ideas and bootstrap the business

Lock-in, IP control, overfitting

Stickiness Standardization and integration; shift from custom to generic

Ability to integrate; support

Virality Word of mouth, references, case studies

Bad vibes; exclusivity

Revenue Growing direct sales, professional services, support

Pipeline, revenue recognition, comp

Scale Channels, analysts, ecosystems, APIs, vertically targeted products

Crossing the chasm; Gorillas

Stage Do this Fear this

Enterprise-focused startups:Metrics that matter

• Ease of customer engagement and feedback

• Alpha/beta pipeline• Stickiness and usability• Integration costs• User engagement with the app

• Disentanglement after a sale• Support costs• User groups and feedback• Pitch success for channel tools• Barriers to exit

What if you’re ina big organization?

If a startup is an organization designed to search for a sustainable, repeatable business model, then an established company is an organization designed

to perpetuate one.

The Lean Analytics lifecycle of an Intrapreneur

Empathy Find problems; don’t test demand. Skip the business case, do analytics

Entitled, aggrieved customers

Stickiness Know your real minimum based on expectations, regulations

Hidden “must haves”, feature creep

Virality Build inherent virality in from the start; attention is the new currency

Luddites who don’t understand sharing

Revenue Consider the ecosystem, channels, and established agreements

Channel conflict, resistance, contracts

Scale Hand the baton to others gracefully Hating what happens to your baby

Beforehand Get buy-in Political fallout

Some things that work.

Frame it like a studyProduct creation is almost accidental.Unlike a VC or startup, when the initiative fails the organization still learns.

http://www.flickr.com/photos/creative_tools/8544475139

Transformative isolation:Skunkworks

Use outliers and missed searches to hunt for good ideas & adjacencies

(Multi-billion-dollar hygiene product company)

1/8 men have an incontinence issue. 1/3 women do.When search results show a significant number of men searching, this suggests the adjacent (male) market is underserved.

Use data to create a taste for data

Sitting on Billions of rows of transactional dataDavid Boyle ran 1M online surveysOnce the value was obvious to management, got license to dig.

Focus on the desired behavior, not just the information.

http://www.psychologytoday.com/blog/yes/200808/changing-minds-and-changing-towels

26% increase in towel re-use with an appeal to social norms; 33% increase when tied to

the specific room.

Energy Conservation “Nudges” and Environmentalist Ideology: Evidence from a Randomized Residential Electricity

Field Experiment - Costa & Kahn 2011

The effectiveness of energy conservation “nudges” depends on an individual’s political ideology ... Conservatives who learn that their

consumption is less than their neighbors’ “boomerang” whereas liberals reduce their consumption.

Understand hidden constraints

That pencil story is a myth. Graphite is conductive and explosive. The Minimum Viable Product is Viable for a reason.

Know what has tobe built in-house

SAP integrationEmployment law

Run it as a consulting business first.

(Just don’t get addicted to it. Your goal is to learn and overcome integration challenges and find the 20% of features that 80% of the market

will pay for.)

When in doubt, collect dataFrom tackling the FTA rate to visualizing the criminal justice supply chain.

Everything’s an excuse to experiment

Find other ways to collect data; everything is an experiment.

Don’t just collect data, chase it.

Some tools and traps

Traction graphs

Your business model

The stage you’re at

Your one metric

... change often if you’re doing it right.

So how do you track that over time?

Traction graphs

Jan Feb Mar Apr May Jun

Signupsper day

Conversionrate

Churnrate

Viralcoefficient

This axis changes for each metric

Traction graphs

Jan Feb Mar Apr May Jun

Signupsper day

Conversionrate

Churnrate

Viralcoefficient

0%

Use vanity to get to meaningful metrics

Your goal is to produce outcomes

If the outcomes require action, and vanity motivates actors, use it

But show how the vanity metric is a leading indicator of the real one

x

Web traffic

Revenue

Activation

CartSize

Conversion rate

The three threesThreeassumptions

What big bets are you making?•“People will answer questions”•“Organizers are frustrated with how to run conferences”•“We'll make money from parents”•“Amazon is reliable enough for our users.”

Three actionsto take

What are you doing to make these assumptions happen (or identify they’re wrong and change course?)•Product enhancements•Marketing strategies

Three experimentsto run

•Feature tests•Continuous deployment•A/B testing•Customer survey

The three threes

Threeassumptions

Three actionsto take

Three experimentsto run

Monthly

Weekly

Daily

Board, investors, founders

Executive team

Employees

Strategy

Tactics

Execution

The three threesThreeassumptions

Three actionsto take

Three experimentsto run

Get more people

Increase answer %

Test betterquestions

Change the UI

Test timings

Questions from peers

Many people will answer questions

The problem-solution canvasCURRENT STATUS

• List key metrics you’re tracking, where they’re at, and compare with last few weeks• How are things trending?

LAST WEEK’S LESSONS LEARNED AND ACCOMPLISHMENTS)

• What did you learn last week?• What was accomplished?• On track: YES / NO?

The Goal is to Learn

The problem-solution canvasHYPOTHESIZED SOLUTIONS

• List possible solutions that you’ll start working on next week. Rank them.• Why do you believe each solution will help you solve or complete solve the problem?

METRICS / PROOF + GOALSProblem #1 (put name here)

• Metrics you’ll use to measure whether or not the solutions are doing what you hoped (solving the problem)• List proof (qualitative) you’ll use as well• Define goals for the metric

HYPOTHESIZED SOLUTIONS

• List possible solutions that you’ll start working on next week. Rank them.• Why do you believe each solution will help you solve or complete solve the problem?

METRICS / PROOF + GOALS

• Metrics you’ll use to measure whether or not the solutions are doing what you hoped (solving the problem)• List proof (qualitative) you’ll use as well• Define goals for the metric

Problem #2 (put name here)

“The most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them.”

Lloyd S. Nelson

Pic by Twodolla on Flickr. http://www.flickr.com/photos/twodolla/3168857844

ARCHIMEDES HAD TAKEN

BATHS BEFORE.

Once, a leader convinced othersin the absence of data.

Now, a leader knowswhat questions to ask.

Follow-on:The other business modelsystem diagrams

The mobile app !customer lifecycle!

Ratings Reviews

Search

Leaderboards

Purchases

Downloads

Installs

Play

Disengagement

Reactivation

Uninstallation

Disengagement

Account"creation

Virality

Downloads,"Gross revenue

ARPU

App sales

Activation

Churn, CLV

In-app"purchases

App

stor

e!

Incentivized

Legitimate

Fraudulent

Ratings!