Croll lean analytics workshop (3h) - lean ux nyc april 2014

  • Published on

  • View

  • Download

Embed Size (px)


How to use data to build a better business faster. Based on the book Lean Analytics, this presentation looks at startup metrics and offers a framework for deliberate growth and iterative improvement of a new business. It also includes examples from larger organizations trying to change from within.


  • Lean Analytics Lean UX NYC April, 2014 @acroll
  • Some housekeeping.
  • Dont sell what you can make. Make what you can sell. Kevin Costner is a lousy entrepreneur.
  • The core of Lean is iteration.
  • Most startups dont know what theyll be when they grow up. Hotmail was a database company Flickr was going to be an MMO Twitter was a podcasting company Autodesk made desktop automation Paypal rst built for Palmpilots Freshbooks was invoicing for a web design rm Wikipedia was to be written by experts only Mitel was a lawnmower company
  • Unfortunately, were all liars.
  • Everyones idea is the best right? People love this part! (but thats not always a good thing) This is where things fall apart. No data, no learning.
  • Analytics can help.
  • Analytics is the measurement of movement towards your business goals.
  • In a startup, the purpose of analytics is to iterate to product/market t before the money runs out.
  • I have two kids. At least one of them is a girl.
  • What are the chances the other is a boy?
  • 2 of 3 (66%) are boys. GB GG BG
  • Some fundamentals.
  • A good metric is: Understandable If youre busy explaining the data, you wont be busy acting on it. Comparative Comparison is context. A ratio or rate The only way to measure change and roll up the tension between two metrics (MPH) Behavior changing If youre busy explaining the data, you wont be busy acting on it.
  • The simplest rule bad metric. If a metric wont change how you behave, its a h"p://
  • Metrics help you know yourself. Acquisition Hybrid Loyalty 70% of retailers 20% of retailers 10% of retailers You are just like Customers that buy >1x in 90d Once 2-2.5 per year >2.5 per year Your customers will buy from you Then you are in this mode 1-15% 15-30% >30% Low acquisition cost, high checkout Increasing return rates, market share Loyalty, selection, inventory size Focus on (Thanks to Kevin Hillstrom for this.)
  • Qualitative Unstructured, anecdotal, revealing, hard to aggregate, often too positive & reassuring. Warm and fuzzy. Quantitative Numbers and stats. Hard facts, less insight, easier to analyze; often sour and disappointing. Cold and hard.
  • Exploratory Speculative. Tries to nd unexpected or interesting insights. Source of unfair advantages. Cool. Reporting Predictable. Keeps you abreast of the normal, day-to-day operations. Can be managed by exception. Necessary.
  • Rumsfeld on Analytics (Or rather, Avinash Kaushik channeling Rumsfeld) Things we know dont know we know Are facts which may be wrong and should be checked against data. we dont know Are questions we can answer by reporting, which we should baseline & automate. we know Are intuition which we should quantify and teach to improve effectiveness, efciency. we dont know Are exploration which is where unfair advantage and interesting epiphanies live.
  • MayAprMarFeb Slicing and dicing data Jan 0 5,000 Activeusers Cohort: Comparison of similar groups along a timeline. (this is the April cohort) A/B test: Changing one thing (i.e. color) and measuring the result (i.e. revenue.) Multivariate analysis Changing several things at once to see which correlates with a result. Segment: Cross-sectional comparison of all people divided by some attribute (age, gender, etc.)
  • Which of these two companies is doing better?
  • January February March April May Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50 Is this company growing or stagnating? Cohort 1 2 3 4 5 January February March April May $5 $3 $2 $1 $0.5 $6 $4 $2 $1 $7 $6 $5 $8 $7 $9 How about this one?
  • Cohort 1 2 3 4 5 January February March April May Averages $5 $3 $2 $1 $0.5 $6 $4 $2 $1 $7 $6 $5 $8 $7 $9 $7 $5 $3 $1 $0.5 Look at the same data in cohorts
  • Lagging Historical. Shows you how youre doing; reports the news. Example: sales. Explaining the past. Leading Forward-looking. Number today that predicts tomorrow; reports the news. Example: pipeline. Predicting the future.
  • A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya) If someone comes back to Zynga a day after signing up for a game, theyll probably become an engaged, paying user (Nabeel Hyatt) A Dropbox user who puts at least one le in one folder on one device (ChenLi Wang) Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman) A LinkedIn user getting to X connections in Y days (Elliot Schmukler) Some examples (From the 2012 Growth Hacking conference.
  • Which means its time to talk about correlation.
  • 1 10 100 1000 10000 Ice cream consumption Drownings Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec
  • Correlated Two variables that are related (but may be dependent on something else.) Ice cream & drowning. Causal An independent variable that directly impacts a dependent one. Summertime & drowning.
  • A leading, causal metric is a superpower. h"p://
  • Growth hacking, demystied. Find correlation Test causality Optimize the causal factor Pick a metric to change
  • Is social action a leading indicator of donation?
  • Is mobile use?
  • Why is Nigerian spam so badly written?
  • Aunshul Rege of Rutgers University, USA in 2009 Experienced scammers expect a strike rate of 1 or 2 replies per 1,000 messages emailed; they expect to land 2 or 3 Mugu (fools) each week. One scammer boasted When you get a reply its 70% sure youll get the money By sending an email that repels all but the most gullible, says [Microsoft Researcher Corman] Herley, the scammer gets the most promising marks to self-select, and tilts the true to false positive ratio in his favor. 1000 emails 1-2 responses 1 fool and their money, parted. Bad language (0.1% conversion) Gullible (70% conversion) 1000 emails 100 responses 1 fool and their money, parted. Good language (10% conversion) Not-gullible (.07% conversion) This would be horribly inecient since humans are involved.
  • Turns out the word Nigeria is the best way to identify promising prospects.
  • Nigerian spammers really understand their target market. They see past vanity metrics.
  • The Lean Analytics framework.
  • Erics three engines of growth Virality Make people invite friends. How many they tell, how fast they tell them. Price Spend money to get customers. Customers are worth more than they cost. Stickiness Keep people coming back. Approach Get customers faster than you lose them. Math that matters
  • Daves Pirate Metrics AARRR Acquisition How do your users become aware of you? SEO, SEM, widgets, email, PR, campaigns, blogs ... Activation Do drive-by visitors subscribe, use, etc? Features, design, tone, compensation, afrmation ... Retention Does a one-time user become engaged? Notications, alerts, reminders, emails, updates... Revenue Do you make money from user activity? Transactions, clicks, subscriptions, DLC, analytics... Referral Do users promote your product? Email, widgets, campaigns, likes, RTs, afliates...
  • Stage EMPATHY Ive found a real, poorly-met need that a reachable market faces. STICKINESS Ive gured out how to solve the problem in a way they will keep using and pay for. VIRALITY Ive found ways to get them to tell their friends, either intrinsically or through incentives. REVENUE The users and features fuel growth organically and articially. SCALE Ive found a sustainable, scalable business with the right margins in a healthy ecosystem. Gate Thevestages
  • Empathy stage: Localmind hacks Twitter Needed to nd out if a core assumptionstrangers answering questionswas valid. Ran Twitter experiment instead of writing code Asked senders of geolocated Tweets from Times Square random questions; counted response rate Conclusion: high enough to proceed
  • Stickiness stage: qidiq streamlines invites Survey owner adds recipient to group Survey owner asks question Recipient reads survey question Recipient responds to question Recipient sees survey results (Later, if needed) Recipient visits site; no password! Recipient does password recovery One-time link sent to email Recipient creates password Recipient can edit prole, etc. Survey owner adds recipient to group Survey owner asks question Recipient gets invite Recipient reads survey question Recipient responds to question Recipient installs mobile app Recipient creates account, prole Recipient sees survey results Recipient can edit prole, etc. 10-25%RESPONSERATE 70-90%RESPONSERATE
  • Six business model archetypes (Yours is probably a blend of these.)
  • E-commerce SaaS (freemium?) Mobile app (gaming) Two sided marketplace Media User generated content
  • (Which means eye charts like these.) Customer Acquisition Cost paid direct search wom inherent virality VISITOR Freemium/trial oer Enrollment User Disengaged User Cancel Freemium churn Engaged User Free user disengagement Reactivate Cancel Trial abandonment rate Invite Others Paying Customer Reactivation rate Paid conversion FORMER USERS User Lifetime Value Reactivate FORMER CUSTOMERS Customer Lifetime Value Viral coecient Viral rate Resolution Support data Account Cancelled Billing Info Exp. Paid Churn Rate Tiering Capacity Limit Upselling rate Upselling Disengaged DissatisedTrial Over
  • Model + Stage = One Metric That Matters. One Metric That Matters. The business youre in E-Com SaaS Mobile 2-Sided Media UCG Empathy Stickiness Virality Revenue Scale Thestageyoureat
  • Really? Just one?
  • Yes, one.
  • In a startup, focus is hard to achieve.
  • Having only one metric addresses this problem.
  • Moz cuts down on metrics SaaS-based SEO toolkit in the scale stage. Focused on net adds. Was a marketing campaign successful? Were customer complaints lowered? Was a product upgrade valuable? Net adds up: Can we acquire more valuable customers? What product features can increase engagement? Can we improve customer support? Net adds at: Are the new customers not the right segment? Did a marketing campaign fail? Did a product upgrade fail somehow? Is customer support falling apart? Net adds down:
  • Metrics are like squeeze toys.
  • Empathy Stickiness Virality Revenue Scale E- commerce SaaS Media Mobile app User-gen content 2-sided market Interviews; qualitative results; quantitative scoring; surveys Loyalty, conversion CAC, shares, reactivation Transaction, CLV Afliates, white-label Engagement, churn Inherent virality, CAC Upselling, CAC, CLV API, magic #, mktplace Content, spam Invites, sharing Ads, donations Analytics, user data Inventory, listings SEM, sharing Transactions, commission Other verticals (Money from transactions) Downloads, churn, virality WoM, app ratings, CAC CLV, ARPDAU Spinoffs, publishers (Money from active users) Trafc, visits, returns Content virality, SEM CPE, afliate %, eyeballs Syndication, licenses (Money from ad clicks)
  • Better:
  • What other metrics do you want to know about?
  • Drawing some lines in the sand.
  • A company loses a quarter of its customers every year. Is this good or bad?
  • Not knowing what normal is makes you do stupid things.
  • ...