Using Lean Analytics for Startups and
Enterprises
Ben Yoskovitz | @byosko
Introduction
@byosko
I am a
product guy entrepreneur author angel investor
Find me online
Blog: http://instigatorblog.com
Slideshare: http://slideshare.net/LeanAnalytics
Book: http://leananalyticsbook.com
Email: [email protected]
@byosko
CORPORATE PARTNERS VENTURE-BACKABLE FOUNDERS PRE-SEED FUNDING
BETTER STARTUPS
+ +=
Highline BETA is a startup co-creation company that launches new ventures with leading corporations.
http://highlinebeta.com @byosko
Metrics:The Fundamentals
Metrics: The fundamentals
● How data fits in
● What makes a good metric
● Types of metrics
● Analytical superpowers
@byosko
How to get things built properly (in theory)
Everyone has great ideas, right?
People love this part (but that’s not always a good thing!)
This is where things start to fall apart.
No data, no learning.
Build Measure Learn seems so easy!
INTELLECTUALLY HONESTY
Follow the Lean model and it becomes
increasingly hard to lie, especially to yourself.
FOCUS
Don’t chase shiny objects. You might
succeed without focus, but it’ll be by accident.
BETTER DECISION MAKING
Everyone has data. The key is figuring
out what pieces will improve your learning
and decision making.
USE YOUR GUT PROPERLY
Instincts are experiments.
Data is proof.
So what makes a good metric?
Question: What are the metrics you’re tracking?
● Take 2 minutes to write down the key metrics you’re tracking (or your business is tracking) right now.
● These could be at a business level or project level.
● At the end of this section we can re-evaluate if the metrics you’re tracking are still the right ones.
@byosko
WHAT IS ANALYTICS?
Analytics is the measurement of movementtowards business goals.
A good metric is:
Understandable
If you’re busy explaining the data, you won’t be busy acting on it.
Comparative
Active Users vs. Active Users/month
Ratio / Rate
% Monthly Active Users
Behavior Changing
You’ll know how you’ll change your business based on what the metric tells you.
@byosko
If a metric won’t change how you behave, it’s a
bad metric.
THE GOLDEN RULE OF METRICS
http://www.flickr.com/photos/circasassy/7858155676/
Acquisition1-15% Low cost of acquisition, high checkout
Customers that buy >1x in 90d
Then you are in this mode
Your customers will buy from you
You are just like Focus on
15-30%
>30%
Hybrid
Loyalty
Once
2-2.5
>2.5
per year
per year
70%
20%
10%
of retailers
of retailers
of retailers
Increasing return rate, market share
Loyalty, selection, inventory size
(Thanks to Kevin Hillstrom for this.)
Metrics help you know yourself:
Types of Metrics
Vanity vs. Actionable metrics
Vanity ActionableMakes you feel good but doesn’t change how you’ll act.
Helps you pick a direction and change your behavior.
“Up and to the right.” These are good.
@byosko
Beware of vanity metrics:
Users
Follows / friends / likes
Logins
This tells you nothing about what they did, why they stuck around, or why they left.
Count actions instead. Count how many followers will do your bidding.
What are they actually doing when they login? Logins don’t tell you about actions and value.
DownloadsSure, people need to download your app in order to use it, but so what?
@byosko
The best (worst!) vanity metric of all time…
# of Features
@byosko
https://www.flickr.com/photos/pinoyed/5009440499
Qualitative vs. Quantitative metrics
Qualitative QuantitativeUnstructured, anecdotal, revealing, hard to aggregate.
Numbers and stats; hard facts, but less insights.
Warm and fuzzy. Cold and hard.
@byosko
Discover qualitatively.
Prove quantitatively.
Do Airbnb hosts get more business if their property is professionally photographed?
Gut instinct (hypothesis)Professional photography helps Airbnb’s business
Concierge MVPSent 20 photographers out into the field
Measure the resultsCompared photographed listings to control group
Make a decisionLaunched photography as a new feature to all hosts
CASE STUDY
Do professional photos make a difference?
Exploratory vs. Reporting metrics
Exploratory ReportingSpeculative. Tries to find unexpected or interesting insights. Source of unfair advantages.
Predictable. Keeps you abreast of normal, day-to-day operations. Can be managed by exception.
Cool. Necessary.
@byosko
! Started as Circle of Friends ! Leveraged Facebook early ! Grew to 10M users fast
ENGAGEMENT SUCKED!
CASE STUDY
Finding insights in the data
ENGAGEMENT SOLVED.
CASE STUDY
Moms are crazy! (in a good way)! Messages to one another were ~50% longer
! 115% more likely to attach a picture to a post
! 110% more likely to engage in a threaded conversation
! Invited friends were 50% more likely to become engaged users
! 60% more likely to accept invitations to the app
Lagging vs. Leading metrics
Lagging LeadingHistorical metric that shows you how you’re doing: reports the news.
Number today that shows a metric tomorrow: makes the news.
Start here. Try and get here.
@byosko
Examples of leading metrics
A Facebook user reaching 7 friends within 10 days of signing up. (Chamath Palihapitiya)
A Dropbox user who puts at least 1 file in 1 folder on 1 device. (ChenLi Wang)
A Twitter user who follows a certain number of people, and a certain percentage of those people follow the user back. (Josh Elman)
A LinkedIn user getting to X connections in Y days. (Elliot Schmukler)
@byosko
1. People who install the Chrome extension 2. People who connect more than 1 social account 3. People who share 15 pieces of content in 7 days
CASE STUDY
Buffer discovered 3 leading metrics
Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec
Correlation vs. causation
Correlated vs. Causal metrics
Correlated CausalTwo variables that are related (but may be dependent on something else.)
An independent variable that directly impacts a dependent one.
Ice cream and drowning.
Summertime and drowning / Summertime and eating ice cream
@byosko
A leading, causal metricis a superpower.
Causality is a superpower because it lets you change the future.
Correlation lets you predict the future
Causality lets you change the future
“I will have 420 engaged users and 75 paying customers next month.”
“If I can make more first time visitors stay for 17 minutes I will increase sales in 90 days.”
Pick a metric to change
Find correlation
Test for causality
Optimize the causal factor
@byosko
Cohort analysis
https://blog.kissmetrics.com/cohort-and-multi-touch-attribution/
@byosko
Ricky (product manager) has some ideas for improving the “Proposal Send Screen” (based on qualitative feedback & his gut), but before prioritizing this work, he digs into the data.
http://proposify.biz
Putting basic data to use
50% of people send proposals through Proposify (50% don’t) (quantitative)
— Is this good or bad?
Putting basic data to usehttp://proposify.biz
Ricky isn’t sure. So he’s going to need to look at additional data (exploratory):
• Churn • Proposal won rate • Any correlations here?
Putting basic data to usehttp://proposify.biz
@byosko
• Also needs to do more direct customer development to learn more (qualitative)
• All of this work might lead to additional, meaningful product dev (actionable)
Putting basic data to usehttp://proposify.biz
Look back at the metrics you’re tracking
● Remember the metrics you wrote down earlier? How do they stack up now? Are they good metrics?
● What might you change about the metrics you’re tracking as a business and/or on a project/feature level?
@byosko
Quick summary on the basics of analytics
● Analytics is about measuring movement towards business goals
● Analytics is about simplifying not complicating
● Analytics is about helping you focus on what really matters
● Remember the Golden Rule: A good metric has to change your behaviour
@byosko
Measuring Success:An introduction to
Lean Analytics
Lean Analytics Framework
● The five stages of business & product development
● Mapping business models
● The One Metric That Matters (KPIs)
● The Lean Analytics Cycle
@byosko
Two keys: the Business you’re in & the Stage you’re at
What business are you in?
What stage are you at?
! E-Commerce ! SaaS ! Free Mobile App ! 2-Sided Marketplace ! Media ! User-Generated Content
! Empathy ! Stickiness ! Virality ! Revenue ! Scale
@byosko
Big companies need one more thing.An understanding of what type
of innovation they’re doing.
Core Adjacent TransformativeDo the same thing
better.Nearby product, market,
or method.Start something
entirely new.
Regionaloptimizations.
Innovation, go-to-market strategies.
Reinvent the business model.
• Get there faster • Smaller batches • Solution, then testing • Increased accountability
• Customer development • Test similar cases • Parallel deployment • Analytics & cycle time
• Fail fast • Skunkworks/R&D • Focus on the search • Ignore the current model &
margins
Many models for enterprise innovation
Know the problem (customers tell you it)
Know the solution (customers/regulations/
norms dictate it.)
Know the problem (market analysis)
Don’t know the solution (non-obvious innovation
confers competitive advantage.)
Don’t know the problem (just an emerging need/change)
Don’t know the solution.
Waterfall:Execution
matters
Agile/scrum:Iteration matters
Lean Startup: Discovery
matters
Another way to look at it
Core Adjacent Transformative
Currentstate
Business optimization
Product,market,method
innovation
Business model
innovation
You can convince executives of this
because some of it is familiar.
This terrifies them because it eats the current business.
A three-maxima model for enterprise innovation
Improvement Adjacency RemodellingDo the same,only better.
Explore what’snearby quickly
Try out new business models
Lean approaches apply, but the metrics vary widely.
Sustain / core
Innovate / adjacent
Disrupt / transformative
Sustaining Adjacent DisruptiveNext year’s car Electric car,
same dealerOn-demand, app-based
car service
So the metrics that matter to a big
company are dependent on the type of innovation being done.
Stages of business & product development
Eric’s three engines of growth
Stickiness Virality Price
Approach
Math that matters
Keep people coming back.
Get customers faster than you
lose them.
Make people invite friends.
How many they tell, how fast they
tell them.
Spend money to get customers.
Customers are worth more than
they cost.
@byosko
Dave McClure’s Pirate Metrics
Dave McClure’s Pirate Metrics
Acquisition
Activation
Retention
Referral
Revenue
How do your users become aware of you?
Do drive-by visitors subscribe, use, etc.?
Does a one time user become engaged?
Do users promote your product?
Do you make money from user activity?
The Lean Analytics Stages
Empathy You’ve found a real, poorly-met need that a reachable market faces.
You’ve figured out how to solve the problem in a way that users will adopt, keep using and pay for.
Your users and features fuel growth organically and artificially.
You’ve found a sustainable, scalable business with the right margins in a healthy ecosystem.
STAGE GATE
Stickiness
Virality
Revenue
Scale
The Lean Analytics Stages
Empathy You’ve found a real, poorly-met need that a reachable market faces.
You’ve figured out how to solve the problem in a way that users will adopt, keep using and pay for.
Your users and features fuel growth organically and artificially.
You’ve found a sustainable, scalable business with the right margins in a healthy ecosystem.
STAGE GATE
Stickiness
Virality
Revenue
Scale
Most products (and startups) fail at this point.
CASE STUDY
! Stage: Empathy/Stickiness
! Model: E-Commerce
! Originally tied to Instagram with an “Insta-Order” feature
Jumping the gun on product development
Optimize for 1st time purchases or repeat orders?
WITH INSTA-ORDER
Click checkout
Confirmation page
Confirm order
Success page
Sign in to PayPal
Back to PayPal
Authorized pre-approved payments
WITHOUT INSTA-ORDER
Click checkout
Sign in to PayPal
Confirmation page
Confirm order
Success page
● 2x transactions ● Lower bounce rate ● Sign-in goals increased
“THERE ARE NO SHORTCUTS TO ANY PLACE WORTH GOING.” - Beverly Sills
Mapping business models
Does recurring revenue work for everyone?
CASE STUDY
@byosko
The leader in predictive analytics for people. Clearfit helps thousands of companies build better teams. As featured in:
CASE STUDY
10x revenue increase off of 3x in sales volume
“People don’t do subscriptions for haircuts, hamburgers or hiring. You have to understand your customer, who they are, how and why they buy, and how they value your product or service.” - Ben Baldwin
The goal is to understand the customer’slifecycle / journey through every
touchpoint with your product.
Paid Direct WOM Search Inherent virality
Customer Acquisition Cost
VISITOR
User
FORMER USERS
Engaged user
Reactivate Trial over
Invite others
Paying customer
Disengaged
Account cancelled
Freemium / trial offer
Enrollment
Disengaged user
Cancel Cancel
Reactivate
FORMER CUSTOMERS
Billing info exp.
Resolution
Dissatisfied
Capacity Limit
UpsellingSignup conversion
rate
Free user disengagement
Freemium churnReactivation
rate
User lifetime value Customer lifetime value
Trial abandonment rate
DAU/WAU/MAUPaid
conversion
Viral coefficient Viral rate
Paid churn rate
Support data
Tiering
Upselling rate
SaaS Customer Lifecycle
Returning Paid Direct Search Viral
Customer Acquisition Cost
VISITOR
E-Commerce Customer Lifecycle
Navigation Search Reco Engine
1-time buyer
Cart
Additions
Conversion
Logistics, delays
Delivery
Enrollment
Call to Action
Sharing
Unsocial buyer
Sharing rate
Returning rate
Customer Lifetime Value
Open rate, engagement
Transaction size
Emphasis on maximizing cart value, minimizing acquisition
costs
Bounced
Not interested
Abandoned
Bounce rate
Unsatisfied
Ratings, delivery issues
Feature usage, product discovery
CASE STUDY
A
A/B testing what really matters
B
CASE STUDY
B
! 41% increase in revenue per customer! (People bought a lot more product.)
! Conversion also went up, but was secondary in importance.
All business models have issuesCAC vs. LTV -- margins are usually very small. A $10M e-commerce business is small.
Freemium requires tens of millions of free users. They can be expensive to support. Will enough convert?
The average # of apps downloaded by North Americans per month is now 0. Monetizing is incredibly hard. Popularity is fleeting.
Chicken & egg problem. Supply and demand. How do you build up both enough?
Real monetization requires hundreds of millions of engaged visitors. People’s attention is hard to capture and keep.
Content creation. Will it be good enough? Will enough people do it? Why?
E-Commerce
SaaS
Mobile Apps
2-sided Marketplace
Media
UCG
@byosko
You know what business you’re in. You know what stage you’re at.
NOW WHAT?
The One Metric That Matters
The business you’re in
E-Com SaaS Mobile 2-Sided Media UCG
The
stag
e yo
u’re
at
Empathy
Stickiness
Virality
Revenue
Scale
THE ONE METRIC THAT MATTERS
@byosko
What really matters when you’re backing up your car?
Moz cuts down on metrics to track
SaaS-based SEO toolkit in the Scale stage. Focused on net adds.
Net adds up:Was a marketing campaign successful? Were customer complaints lowered? Was a product upgrade valuable?
Net adds flat:Can we acquire more valuable customers? What product features can increase engagement? Can we improve customer support?
Net adds down:Are the new customers not the right segment? Did a marketing campaign fail? Did a product upgrade fail somehow? Is customer support falling apart?
Timehop only cares about virality
! Focused on % of daily active users that share content
! Aiming for 20-30% of daily active users to share content
“All that matters now is virality. Everything else--be it press, publicity stunts or something else--is like pushing a rock up a mountain: it will never scale. But being viral will.” -- Jonathan Wegener, founder
# of transactions (for merchants) # of nights booked sales
total time reading
https://medium.com/data-lab/mediums-metric-that-matters-total-time-reading-86c4970837d5#.tidx5bunjhttp://quibb.com/links/metrics-to-inform-your-model-lessons-from-square-stripe-and-quora
http://500.co/aircall-growth-uber/
monthly active users monthly recurring revenue (MRR)
Examples of OMTM
@byosko
www.flickr.com/photos/connortarter/4791605202/
METRICS ARE LIKE SQUEEZE TOYS
Better: http://bit.ly/BigLeanTable
The Layer Cake of Metrics
Project OMTM
Project OMTM
Project OMTM
Project OMTM
Project OMTM
Project OMTM
Department OMTM Department OMTM Department OMTM
OMTM: Business Help Indicator
What’s your OMTM?
● So what’s your OMTM? Do you know? Can you write it down? Is it available to everyone at your company?
● Can you see how your work matters to the overall health of the business and how you might measure that value creation?
@byosko
Drawing lines in the sand
Growth5% / week (revenue or active users)
Time on site17 minutes
Free to paid2% of free users
Mobile file size< 50MB
Engaged visitors30% monthly users 10% daily users
Paid load time< 5 seconds
Churn2% / month
CLV:CAC3:1
Some benchmarks
@byosko
CASE STUDY: Solare draws a line in the sand
@byosko
50 reservations by 5pm 250 covers that night
=
CASE STUDY: Solare discovers a leading indicator
@byosko
The Lean Analytics Cycle
Identify a key business problem,
pick the OMTM, draw a line inthe sand, and get started.
Draw a new line
ZxLERATOR | NYC | SUMMER 2016 89
LEAN ANALYTICS: THE FRAMEWORK Day 4 - Lean Analytics
Pivot or give up
Try again
Success!
Did we move the needle?
Measure the results
Make changes in production
Design a test
Hypothesis
With data: find a commonality
Without data: make a good guess
Find a potential improvement
Draw a linePick a OMTM
Lean Analytics Cycle
Quick summary on the Lean Analytics framework
● What you track depends on what type of innovation you’re doing: core, adjacent or disruptive
● What you track depends on your business model and stage (for a startup, project, product or even at a feature-level)
● Find the One Metric That Matters so you can focus as much as possible
● The more holistically you can assess your business, the better off you’ll be (map it all and find the hot spots!)
@byosko
The value of datain building
better products.
Data is a key input and filter in building better products.
COMPETITION, OTHER PRODUCTS, BEST PRACTICES
BUILDLEARN
IDEAS
CORPORATE GOALS (SOME GOOD,
SOME BAD)GUTS & INSTINCTS
PARTNERS OTHER DEPARTMENTS
INDUSTRY TRENDS, ETC. DATA
CUSTOMER INPUT DATA
COMPETITION, OTHER PRODUCTS, BEST PRACTICES
PARTNERS
INDUSTRY TRENDS, ETC.
GUTS & INSTINCTS
OTHER DEPARTMENTS
CORPORATE GOALS
DATA AS A
FILTER
BETTER DECISIONS
CUSTOMER INPUT
Product & Design (defining goals /
objectives)
User & customer feedback
Sales
Marketing
Customer Support
Etc.
! In-person interviews ! Surveys ! Customer support
inquiries ! Real-time online
Supported by data
Your gut
Company vision
Collecting Input & Customer Discovery
Your own ideas
Data is also a
communication tool.
http://www.instigatorblog.com/data-common-language/2016/09/22/
@byosko @byosko
Data is complex.
How we communicate itdoesn’t have to be.
Recommended
