Lean Analytics for Startups and Enterprises

Using Lean Analytics for Startups and

Enterprises

Ben Yoskovitz | @byosko

Introduction

@byosko

I am a

product guy entrepreneur author angel investor

Find me online

Blog: http://instigatorblog.com

Slideshare: http://slideshare.net/LeanAnalytics

Book: http://leananalyticsbook.com

Email: byosko@gmail.com

@byosko

CORPORATE PARTNERS VENTURE-BACKABLE FOUNDERS PRE-SEED FUNDING

BETTER STARTUPS

+ +=

Highline BETA is a startup co-creation company that launches new ventures with leading corporations.

http://highlinebeta.com @byosko

Metrics:The Fundamentals

Metrics: The fundamentals

● How data fits in

● What makes a good metric

● Types of metrics

● Analytical superpowers

@byosko

How to get things built properly (in theory)

Everyone has great ideas, right?

People love this part (but that’s not always a good thing!)

This is where things start to fall apart.

No data, no learning.

Build Measure Learn seems so easy!

INTELLECTUALLY HONESTY

Follow the Lean model and it becomes

increasingly hard to lie, especially to yourself.

FOCUS

Don’t chase shiny objects. You might

succeed without focus, but it’ll be by accident.

BETTER DECISION MAKING

Everyone has data. The key is figuring

out what pieces will improve your learning

and decision making.

USE YOUR GUT PROPERLY

Instincts are experiments.

Data is proof.

So what makes a good metric?

Question: What are the metrics you’re tracking?

● Take 2 minutes to write down the key metrics you’re tracking (or your business is tracking) right now.

● These could be at a business level or project level.

● At the end of this section we can re-evaluate if the metrics you’re tracking are still the right ones.

@byosko

WHAT IS ANALYTICS?

Analytics is the measurement of movementtowards business goals.

A good metric is:

Understandable

If you’re busy explaining the data, you won’t be busy acting on it.

Comparative

Active Users vs. Active Users/month

Ratio / Rate

% Monthly Active Users

Behavior Changing

You’ll know how you’ll change your business based on what the metric tells you.

@byosko

If a metric won’t change how you behave, it’s a

bad metric.

THE GOLDEN RULE OF METRICS

http://www.flickr.com/photos/circasassy/7858155676/

Acquisition1-15% Low cost of acquisition, high checkout

Customers that buy >1x in 90d

Then you are in this mode

Your customers will buy from you

You are just like Focus on

15-30%

>30%

Hybrid

Loyalty

Once

2-2.5

>2.5

per year

per year

70%

20%

10%

of retailers

of retailers

of retailers

Increasing return rate, market share

Loyalty, selection, inventory size

(Thanks to Kevin Hillstrom for this.)

Metrics help you know yourself:

Types of Metrics

Vanity vs. Actionable metrics

Vanity ActionableMakes you feel good but doesn’t change how you’ll act.

Helps you pick a direction and change your behavior.

“Up and to the right.” These are good.

@byosko

Beware of vanity metrics:

Users

Follows / friends / likes

Logins

This tells you nothing about what they did, why they stuck around, or why they left.

Count actions instead. Count how many followers will do your bidding.

What are they actually doing when they login? Logins don’t tell you about actions and value.

DownloadsSure, people need to download your app in order to use it, but so what?

@byosko

The best (worst!) vanity metric of all time…

# of Features

@byosko

https://www.flickr.com/photos/pinoyed/5009440499

Qualitative vs. Quantitative metrics

Qualitative QuantitativeUnstructured, anecdotal, revealing, hard to aggregate.

Numbers and stats; hard facts, but less insights.

Warm and fuzzy. Cold and hard.

@byosko

Discover qualitatively.

Prove quantitatively.

Do Airbnb hosts get more business if their property is professionally photographed?

Gut instinct (hypothesis)Professional photography helps Airbnb’s business

Concierge MVPSent 20 photographers out into the field

Measure the resultsCompared photographed listings to control group

Make a decisionLaunched photography as a new feature to all hosts

CASE STUDY

Do professional photos make a difference?

Exploratory vs. Reporting metrics

Exploratory ReportingSpeculative. Tries to find unexpected or interesting insights. Source of unfair advantages.

Predictable. Keeps you abreast of normal, day-to-day operations. Can be managed by exception.

Cool. Necessary.

@byosko

! Started as Circle of Friends ! Leveraged Facebook early ! Grew to 10M users fast

ENGAGEMENT SUCKED!

CASE STUDY

Finding insights in the data

ENGAGEMENT SOLVED.

CASE STUDY

Moms are crazy! (in a good way)! Messages to one another were ~50% longer

! 115% more likely to attach a picture to a post

! 110% more likely to engage in a threaded conversation

! Invited friends were 50% more likely to become engaged users

! 60% more likely to accept invitations to the app

Lagging vs. Leading metrics

Lagging LeadingHistorical metric that shows you how you’re doing: reports the news.

Number today that shows a metric tomorrow: makes the news.

Start here. Try and get here.

@byosko

Examples of leading metrics

A Facebook user reaching 7 friends within 10 days of signing up. (Chamath Palihapitiya)

A Dropbox user who puts at least 1 file in 1 folder on 1 device. (ChenLi Wang)

A Twitter user who follows a certain number of people, and a certain percentage of those people follow the user back. (Josh Elman)

A LinkedIn user getting to X connections in Y days. (Elliot Schmukler)

@byosko

1. People who install the Chrome extension 2. People who connect more than 1 social account 3. People who share 15 pieces of content in 7 days

CASE STUDY

Buffer discovered 3 leading metrics

Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec

Correlation vs. causation

Correlated vs. Causal metrics

Correlated CausalTwo variables that are related (but may be dependent on something else.)

An independent variable that directly impacts a dependent one.

Ice cream and drowning.

Summertime and drowning / Summertime and eating ice cream

@byosko

A leading, causal metricis a superpower.

Causality is a superpower because it lets you change the future.

Correlation lets you predict the future

Causality lets you change the future

“I will have 420 engaged users and 75 paying customers next month.”

“If I can make more first time visitors stay for 17 minutes I will increase sales in 90 days.”

Pick a metric to change

Find correlation

Test for causality

Optimize the causal factor

@byosko

Cohort analysis

https://blog.kissmetrics.com/cohort-and-multi-touch-attribution/

@byosko

Ricky (product manager) has some ideas for improving the “Proposal Send Screen” (based on qualitative feedback & his gut), but before prioritizing this work, he digs into the data.

http://proposify.biz

Putting basic data to use

50% of people send proposals through Proposify (50% don’t) (quantitative)

— Is this good or bad?

Putting basic data to usehttp://proposify.biz

Ricky isn’t sure. So he’s going to need to look at additional data (exploratory):

• Churn • Proposal won rate • Any correlations here?

Putting basic data to usehttp://proposify.biz

@byosko

• Also needs to do more direct customer development to learn more (qualitative)

• All of this work might lead to additional, meaningful product dev (actionable)

Putting basic data to usehttp://proposify.biz

Look back at the metrics you’re tracking

● Remember the metrics you wrote down earlier? How do they stack up now? Are they good metrics?

● What might you change about the metrics you’re tracking as a business and/or on a project/feature level?

@byosko

Quick summary on the basics of analytics

● Analytics is about measuring movement towards business goals

● Analytics is about simplifying not complicating

● Analytics is about helping you focus on what really matters

● Remember the Golden Rule: A good metric has to change your behaviour

@byosko

Measuring Success:An introduction to

Lean Analytics

Lean Analytics Framework

● The five stages of business & product development

● Mapping business models

● The One Metric That Matters (KPIs)

● The Lean Analytics Cycle

@byosko

Two keys: the Business you’re in & the Stage you’re at

What business are you in?

What stage are you at?

! E-Commerce ! SaaS ! Free Mobile App ! 2-Sided Marketplace ! Media ! User-Generated Content

! Empathy ! Stickiness ! Virality ! Revenue ! Scale

@byosko

Big companies need one more thing.An understanding of what type

of innovation they’re doing.

Core Adjacent TransformativeDo the same thing

better.Nearby product, market,

or method.Start something

entirely new.

Regionaloptimizations.

Innovation, go-to-market strategies.

Reinvent the business model.

• Get there faster • Smaller batches • Solution, then testing • Increased accountability

• Customer development • Test similar cases • Parallel deployment • Analytics & cycle time

• Fail fast • Skunkworks/R&D • Focus on the search • Ignore the current model &

margins

Many models for enterprise innovation

Know the problem (customers tell you it)

Know the solution (customers/regulations/

norms dictate it.)

Know the problem (market analysis)

Don’t know the solution (non-obvious innovation

confers competitive advantage.)

Don’t know the problem (just an emerging need/change)

Don’t know the solution.

Waterfall:Execution

matters

Agile/scrum:Iteration matters

Lean Startup: Discovery

matters

Another way to look at it

Core Adjacent Transformative

Currentstate

Business optimization

Product,market,method

innovation

Business model

innovation

You can convince executives of this

because some of it is familiar.

This terrifies them because it eats the current business.

A three-maxima model for enterprise innovation

Improvement Adjacency RemodellingDo the same,only better.

Explore what’snearby quickly

Try out new business models

Lean approaches apply, but the metrics vary widely.

Sustain / core

Innovate / adjacent

Disrupt / transformative

Sustaining Adjacent DisruptiveNext year’s car Electric car,

same dealerOn-demand, app-based

car service

So the metrics that matter to a big

company are dependent on the type of innovation being done.

Stages of business & product development

Eric’s three engines of growth

Stickiness Virality Price

Approach

Math that matters

Keep people coming back.

Get customers faster than you

lose them.

Make people invite friends.

How many they tell, how fast they

tell them.

Spend money to get customers.

Customers are worth more than

they cost.

@byosko

Dave McClure’s Pirate Metrics

Dave McClure’s Pirate Metrics

Acquisition

Activation

Retention

Referral

Revenue

How do your users become aware of you?

Do drive-by visitors subscribe, use, etc.?

Does a one time user become engaged?

Do users promote your product?

Do you make money from user activity?

The Lean Analytics Stages

Empathy You’ve found a real, poorly-met need that a reachable market faces.

You’ve figured out how to solve the problem in a way that users will adopt, keep using and pay for.

Your users and features fuel growth organically and artificially.

You’ve found a sustainable, scalable business with the right margins in a healthy ecosystem.

STAGE GATE

Stickiness

Virality

Revenue

Scale

The Lean Analytics Stages

Empathy You’ve found a real, poorly-met need that a reachable market faces.

You’ve figured out how to solve the problem in a way that users will adopt, keep using and pay for.

Your users and features fuel growth organically and artificially.

You’ve found a sustainable, scalable business with the right margins in a healthy ecosystem.

STAGE GATE

Stickiness

Virality

Revenue

Scale

Most products (and startups) fail at this point.

CASE STUDY

! Stage: Empathy/Stickiness

! Model: E-Commerce

! Originally tied to Instagram with an “Insta-Order” feature

Jumping the gun on product development

Optimize for 1st time purchases or repeat orders?

WITH INSTA-ORDER

Click checkout

Confirmation page

Confirm order

Success page

Sign in to PayPal

Back to PayPal

Authorized pre-approved payments

WITHOUT INSTA-ORDER

Click checkout

Sign in to PayPal

Confirmation page

Confirm order

Success page

● 2x transactions ● Lower bounce rate ● Sign-in goals increased

“THERE ARE NO SHORTCUTS TO ANY PLACE WORTH GOING.” - Beverly Sills

Mapping business models

Does recurring revenue work for everyone?

CASE STUDY

@byosko

The leader in predictive analytics for people. Clearfit helps thousands of companies build better teams. As featured in:

CASE STUDY

10x revenue increase off of 3x in sales volume

“People don’t do subscriptions for haircuts, hamburgers or hiring. You have to understand your customer, who they are, how and why they buy, and how they value your product or service.” - Ben Baldwin

The goal is to understand the customer’slifecycle / journey through every

touchpoint with your product.

Paid Direct WOM Search Inherent virality

Customer Acquisition Cost

VISITOR

User

FORMER USERS

Engaged user

Reactivate Trial over

Invite others

Paying customer

Disengaged

Account cancelled

Freemium / trial offer

Enrollment

Disengaged user

Cancel Cancel

Reactivate

FORMER CUSTOMERS

Billing info exp.

Resolution

Dissatisfied

Capacity Limit

UpsellingSignup conversion

rate

Free user disengagement

Freemium churnReactivation

rate

User lifetime value Customer lifetime value

Trial abandonment rate

DAU/WAU/MAUPaid

conversion

Viral coefficient Viral rate

Paid churn rate

Support data

Tiering

Upselling rate

SaaS Customer Lifecycle

Returning Paid Direct Search Viral

Customer Acquisition Cost

VISITOR

E-Commerce Customer Lifecycle

Navigation Search Reco Engine

1-time buyer

Cart

Additions

Conversion

Logistics, delays

Delivery

Enrollment

Call to Action

Sharing

Unsocial buyer

Sharing rate

Returning rate

Customer Lifetime Value

Open rate, engagement

Transaction size

Emphasis on maximizing cart value, minimizing acquisition

costs

Bounced

Not interested

Abandoned

Bounce rate

Unsatisfied

Ratings, delivery issues

Feature usage, product discovery

CASE STUDY

A

A/B testing what really matters

B

CASE STUDY

B

! 41% increase in revenue per customer! (People bought a lot more product.)

! Conversion also went up, but was secondary in importance.

All business models have issuesCAC vs. LTV -- margins are usually very small. A $10M e-commerce business is small.

Freemium requires tens of millions of free users. They can be expensive to support. Will enough convert?

The average # of apps downloaded by North Americans per month is now 0. Monetizing is incredibly hard. Popularity is fleeting.

Chicken & egg problem. Supply and demand. How do you build up both enough?

Real monetization requires hundreds of millions of engaged visitors. People’s attention is hard to capture and keep.

Content creation. Will it be good enough? Will enough people do it? Why?

E-Commerce

SaaS

Mobile Apps

2-sided Marketplace

Media

UCG

@byosko

You know what business you’re in. You know what stage you’re at.

NOW WHAT?

The One Metric That Matters

The business you’re in

E-Com SaaS Mobile 2-Sided Media UCG

The

stag

e yo

u’re

at

Empathy

Stickiness

Virality

Revenue

Scale

THE ONE METRIC THAT MATTERS

@byosko

What really matters when you’re backing up your car?

Moz cuts down on metrics to track

SaaS-based SEO toolkit in the Scale stage. Focused on net adds.

Net adds up:Was a marketing campaign successful? Were customer complaints lowered? Was a product upgrade valuable?

Net adds flat:Can we acquire more valuable customers? What product features can increase engagement? Can we improve customer support?

Net adds down:Are the new customers not the right segment? Did a marketing campaign fail? Did a product upgrade fail somehow? Is customer support falling apart?

Timehop only cares about virality

! Focused on % of daily active users that share content

! Aiming for 20-30% of daily active users to share content

“All that matters now is virality. Everything else--be it press, publicity stunts or something else--is like pushing a rock up a mountain: it will never scale. But being viral will.” -- Jonathan Wegener, founder

# of transactions (for merchants) # of nights booked sales

total time reading

https://medium.com/data-lab/mediums-metric-that-matters-total-time-reading-86c4970837d5#.tidx5bunjhttp://quibb.com/links/metrics-to-inform-your-model-lessons-from-square-stripe-and-quora

http://500.co/aircall-growth-uber/

monthly active users monthly recurring revenue (MRR)

Examples of OMTM

@byosko

www.flickr.com/photos/connortarter/4791605202/

METRICS ARE LIKE SQUEEZE TOYS

Better: http://bit.ly/BigLeanTable

The Layer Cake of Metrics

Project OMTM

Project OMTM

Project OMTM

Project OMTM

Project OMTM

Project OMTM

Department OMTM Department OMTM Department OMTM

OMTM: Business Help Indicator

What’s your OMTM?

● So what’s your OMTM? Do you know? Can you write it down? Is it available to everyone at your company?

● Can you see how your work matters to the overall health of the business and how you might measure that value creation?

@byosko

Drawing lines in the sand

Growth5% / week (revenue or active users)

Time on site17 minutes

Free to paid2% of free users

Mobile file size< 50MB

Engaged visitors30% monthly users 10% daily users

Paid load time< 5 seconds

Churn2% / month

CLV:CAC3:1

Some benchmarks

@byosko

CASE STUDY: Solare draws a line in the sand

@byosko

50 reservations by 5pm 250 covers that night

=

CASE STUDY: Solare discovers a leading indicator

@byosko

The Lean Analytics Cycle

Identify a key business problem,

pick the OMTM, draw a line inthe sand, and get started.

Draw a new line

ZxLERATOR | NYC | SUMMER 2016 89

LEAN ANALYTICS: THE FRAMEWORK Day 4 - Lean Analytics

Pivot or give up

Try again

Success!

Did we move the needle?

Measure the results

Make changes in production

Design a test

Hypothesis

With data: find a commonality

Without data: make a good guess

Find a potential improvement

Draw a linePick a OMTM

Lean Analytics Cycle

Quick summary on the Lean Analytics framework

● What you track depends on what type of innovation you’re doing: core, adjacent or disruptive

● What you track depends on your business model and stage (for a startup, project, product or even at a feature-level)

● Find the One Metric That Matters so you can focus as much as possible

● The more holistically you can assess your business, the better off you’ll be (map it all and find the hot spots!)

@byosko

The value of datain building

better products.

Data is a key input and filter in building better products.

COMPETITION, OTHER PRODUCTS, BEST PRACTICES

BUILDLEARN

IDEAS

CORPORATE GOALS (SOME GOOD,

SOME BAD)GUTS & INSTINCTS

PARTNERS OTHER DEPARTMENTS

INDUSTRY TRENDS, ETC. DATA

CUSTOMER INPUT DATA

COMPETITION, OTHER PRODUCTS, BEST PRACTICES

PARTNERS

INDUSTRY TRENDS, ETC.

GUTS & INSTINCTS

OTHER DEPARTMENTS

CORPORATE GOALS

DATA AS A

FILTER

BETTER DECISIONS

CUSTOMER INPUT

Product & Design (defining goals /

objectives)

User & customer feedback

Sales

Marketing

Customer Support

Etc.

! In-person interviews ! Surveys ! Customer support

inquiries ! Real-time online

Supported by data

Your gut

Company vision

Collecting Input & Customer Discovery

Your own ideas

Data is also a

communication tool.

http://www.instigatorblog.com/data-common-language/2016/09/22/

@byosko @byosko

Data is complex.

How we communicate itdoesn’t have to be.

Alistair Croll acroll@gmail.com @acroll

Ben Yoskovitz byosko@gmail.com @byosko