Lean Analytics - Communication Layer into the World of Business

Lean AnalyticsThe Communication Layer into the

World of Business

Budapest, 2016 @byosko

Agenda09:00-11:00 - Tools & frameworks for “talking business”

• Lean Startup (it’s not just for startups!)• Business Mapping (Lean Canvas)• Product Management & Data (roadmaps & feature prioritization)

11:00-11:30 - Coffee break11:30-12:30 - Lean Analytics: The basics

• Making data simpler for everyone else• What makes a good metric• Types of metrics

12:30-13:30 - Lunch13:30-15:00 - Lean Analytics II

• Lean Analytics framework• One Metric that Matters• Lean Analytics Cycle

15:00-15:30 - Coffee break15:30-17:30 - Interactive session & Q&A

TOOLS & FRAMEWORKS FOR “TALKING BUSINESS”1

WHO FEELS LIKE THIS?

DATA IS A COMMUNICATION TOOL

http://www.instigatorblog.com/data-common-language/2016/09/22/

@byosko

WE HAVE TO TALK BUSINESS FIRST

TALKING BUSINESS: LEAN STARTUP

LEAN STARTUP ISN’T JUST FOR STARTUPS

ENTERPRISES ARE GOING LEAN…

“By 2021, more than 50% of established corporations will be leveraging lean startup techniques.” (Gartner)

Core Adjacent TransformativeDo the same thing

better.Nearby product, market,

or method.Start something

entirely new.

Regional optimizations.

Innovation, go-to-market strategies.

Reinvent the business model.

• Get there faster • Smaller batches • Solution, then testing • Increased accountability

• Customer development • Test similar cases • Parallel deployment • Analytics & cycle time

• Fail fast • Skunkworks/R&D • Focus on the search • Ignore the current model &

margins

Many models for enterprise innovation

Core Adjacent TransformativeKnow the problem

(customers tell you it)

Know the solution (customers/regulations/

norms dictate it.)

Know the problem (market analysis)

Don’t know the solution (non-obvious innovation confers

competitive advantage.)

Don’t know the problem (just an emerging need/change) Don’t know the solution.

Waterfall:Execution matters

Agile/scrum:Iteration matters

Lean Startup: Discovery matters

Another way to look at it

Currentstate

Business optimization

Product,market,method

innovation

Business model

innovation

You can convince executives of this

because some of it is familiar.

This terrifies them because it eats the current business.

A three-maxima model for enterprise innovation

Improvement Adjacency RemodellingDo the same,only better.

Explore what’s nearby quickly

Try out newbusiness models

Lean approaches apply, but the metrics vary widely.

Sustain / core

Innovate / adjacent

Disrupt / transformative

Sustaining Adjacent DisruptiveNext year’s car Electric car,

same dealerOn-demand, app-based

car service

LET’S GO BACK TO HOW LEAN WORKS…

Don’t sell what you can make. Make what you can sell.

Kevin Costner is a lousy entrepreneur.

The core of Lean is iteration.



Key principles of Lean Startup

• “Get out of the building”• Customer discovery is key• Identify and solve the riskiest thing first• Reduce waste• Most important: learning


“THERE ARE NO ANSWERS INSIDE THE BUILDING.”

- Steve Blank, author of Four Steps to the Epiphany


Build Measure Learn

• What do I build?• How do I know if it’s the

right thing?• How do I start?


START WITH A PROBLEM:

What problem are you trying to solve?


Fall in love with the…

PROBLEM

…not the solution.

LEAN STARTUP IS ABOUT FINDING THE TRUTH,

AND DOING SO AS QUICKLY AS POSSIBLE.

SO HOW DO WE FIGURE OUT WHAT PROBLEMS

TO SOLVE?


http://talkingtohumans.com https://leanstack.com/running-lean-book/ http://keytakeaways.io/books/four-steps-epiphany/

A few good references on customer development, problem/solution interviews and more

http://talkingtohumans.com

https://leanstack.com/running-lean-book/

http://keytakeaways.io/books/four-steps-epiphany/

TALKING BUSINESS: ASSUMPTIONS

Understand your business assumptions

• Make a list of your riskiest assumptions first

• Assumptions have to be testable

• “We believe that…”

Do you know the definition of “Assume”?

BUSINESS ASSUMPTIONS Day 2 - Lean Startup

ZxLERATOR | NYC | SUMMER 2016 31

Understand your business assumptions• My target customer will be? (Tip: how would you describe your primary

target customer)• The problem my customer wants to solve is? (Tip: what does your

customer struggle with or what need do they want to fulfill?)• My customer’s need can be solved with? (Tip: give a very concise

description / elevator pitch of your product)• Why can’t my customer solve this today? (Tip: what are the obstacles

that have prevented my customer from solving this already?)• The measurable outcome my customer wants to achieve is? (Tip:

what measurable change in your customer’s life makes them love your product?)

• My primary customer acquisition tactic will be? (Tip: what is your current guest for the top 1 or 2 marketing channels?)


BUSINESS ASSUMPTIONS Day 2 - Lean Startup


Understand your business assumptions• My earliest adopter will be? (Tip: remember that you can’t get to the

mainstream customer without getting early adopters first) • I will make money (revenue) by? (Tip: don’t list all the ideas for making

money, but pick your primary one)• My primary competition will be? (Tip: think about both direct and

indirect competition?)• I will beat my competitors primarily because of? (Tip: what truly

differentiates you from the competition?)• My biggest risk to financial viability is? (Tip: what could prevent you

from getting to breakeven? Is there something baked into your revenue or cost model that you can de-risk?)

• My biggest technical or engineering risk is? (Tip: is there a major technical challenge that might hinder building your product?)


WHAT ASSUMPTIONS DO YOU HAVE, IF PROVEN

WRONG, WOULD CAUSE YOUR BUSINESS TO FAIL?

MAPPING YOUR BUSINESS

TALKING BUSINESS: MAPPING YOUR BUSINESS

• It’s difficult to create value for a business without understanding how the business functions

• There are several systems for designing/mapping a business

• The goal is to understand all the actors involved, their motivations, interests and biases

• Common questions you’ll want to answer: Where does the money come from? Who is the customer? Who are the partners?

The importance of mapping your business

Does recurring revenue work for everyone?

CASE STUDY


The leader in predictive analytics for people. Clearfit helps thousands of companies build better teams. As featured in:

CASE STUDY

10x revenue increase off of 3x in sales volume

“People don’t do subscriptions for haircuts, hamburgers or hiring. You have to understand your customer, who they are, how and why they buy, and how they value your product or service.” - Ben Baldwin

Business Model Canvas: http://www.businessmodelgeneration.com/canvas/bmc

Lean Canvas: https://leanstack.com/lean-canvas/

Two good approaches


http://www.businessmodelgeneration.com/canvas/bmc

https://leanstack.com/lean-canvas/

https://leanstack.com/lean-canvas/


http://leanstack.com/lean-canvas/

• A 1-page “Business Plan”• Helps identify key aspects of your business and biggest risks• Should be doable in 20-minutes• A visualization of the business assumptions list from Talking

to Humans• Ideally you update it over time at major milestones / key

discoveries

Introducing Lean Canvas


Lean Canvas

Day 3 - Business Models & Value Propositions

DESIGNING A BUSINESS MODEL

1 234

5

67

8

9


http://leanstack.com/lean-canvas/

Lean Canvas Best Practices

1

2

3

Try and test one thing at a time

Have a hypothesis around how to test each section and decide on an MVP to experiment with

Update the Lean Canvas whenever you’ve learned something significant


1. Testing the Problem

• Ongoing customer discovery; problem interviews -- largely collecting qualitative feedback

• Remember: No one pays for their 5th problem to be solved

• For existing alternatives define one clear, direct competitor.• Consider other ways users/customers can address their problems• What products or services exist as alternatives to what you’re offering?

• Be careful about problems that are too high level (“universal truths”)


2. Identifying Customer Segments

• Define 3-4 specific user personas for your early adopter groups

• Who is your intended audience?

• What type of person do you anticipate benefiting most from your product?

• Don’t forget there may be a difference between the users of your product and the buyer

• For early adopters: what makes them different?

• Understand your best users: http://www.instigatorblog.com/your-best-users/2014/06/20/

http://www.instigatorblog.com/your-best-users/2014/06/20/


3. Testing the Unique Value Proposition

• What do you do, why are you different, and why are you worth investing in?

• What’s the high level concept?

• Conduct solution interviews (this is where you’ll start to test your UVP)

• Create landing pages with different messaging


More on Value Propositions

Steve Blank’s XYZTemplate: “We help X do Y doing Z”.

Sample: We help non-technical marketers discover return on investment in social media by turning engagement metrics into revenue metrics

Dave McClure’s Elevator RideTemplate: • Short, simple, memorable: what, how, why.• 3 keywords or phrases• KISS (no expert jargon)

Sample: Mint.com is the free, easy way to manage your money online.

http://torgronsund.com/2011/11/29/7-proven-templates-for-creating-value-propositions-that-work/

http://torgronsund.com/2011/11/29/7-proven-templates-for-creating-value-propositions-that-work/


4. Testing Solutions

• Conduct solution interviews — get qualitative feedback from users/customers

• Measure interest in solutions (before building anything)

• Start mapping out your MVPs

• Map key features to key problems


5. Testing Channels

• How will users/customers come in contact with your brand? Where will they first learn about your business?

• Think about the various touch points before people buy, during purchase, and after

• Don’t worry too much about cost (initially), focus more on channel success in terms of conversion and quality of users/customers


6. Testing Revenue

• How will you make money? Who will pay?

• Early on, revenue is a proxy of value creation, but now you have to test the economic engine of the business more seriously

• Remember: users of your product may not be the customers, or you may have multiple customers (e.g. top-down enterprise software)

• Metrics of importance: ARPU, LTV, CAC, MRR, sales, etc.


6. Testing Pricing

http://download.red-gate.com/ebooks/DJRTD_eBook.pdf

“Prices are a shortcut to our most sensitive emotional responses.” – Tom Whitwell

https://medium.com/dreamit-perspectives/founders-guide-to-product-pricing-b187093e2483#.vdpwcl752

● Don’t overcomplicate too early

● Sell based on aspirational value (not negative value)

● Don’t ask customers what they’d pay, they don’t know

● Test it (over and over again)



https://medium.com/fluxx-studio-notes/the-first-rule-of-pricing-is-you-do-not-talk-about-pricing-1875caa39b89#.skikdqeya




7. Measuring Costs

• What are the costs?

• Where do they come from?

• How are they impacted by growth/scale?

• For Web-related businesses, costs are usually low and not worth focusing on initially; but for other types of businesses (manufacturing, hardware) costs are much higher. For Web-related businesses, costs are mostly in acquisition (where LTV > CAC becomes important)


8. Metrics

• We’ll discuss this in more detail later today!


9. Defining an Unfair Advantage

• What makes you stand out from competitors?

• What do you know or have access to that no one else does?

• This is very hard to come up with; there are very few legitimate unfair advantages

• Insider information • Single-minded obsession with the “One Thing” • Personal authority • Existing customers / distribution

http://blog.asmartbear.com/unfair-advantages.html




LEAN CANVAS

Example Lean Canvas: Freckle


http://www.slideshare.net/de-pe/lean-canvas-process-and-examples


http://www.slideshare.net/de-pe/lean-canvas-process-and-examples


1

2

3

Fill out your Lean Canvas in 20 minutes

Think through the business assumption questions from Talking to Humans

We’ll share and discuss after

Time to give it a try!

PRODUCT MANAGEMENT & DATA

Product management is the “glue” between everyone and everything

TALKING BUSINESS: PRODUCT MANAGEMENT

PRODUCT MANAGEMENT IS THE PROCESS BY WHICH WE TURN

VISION INTO REALITY

PRODUCT MANAGEMENT = STRATEGY - PRIORITIZATION EXECUTION


The role of product managers

• Product managers are the “glue” between all stakeholders (internal & external)

• Work towards achieving the company’s vision through strategy & execution

• Empower others to get things done

• Prioritize feature development based on all the inputs

• Ensure deliverability as expected (on time)

• Measure results (success or failure)


“The job of a product manager is to: Help your team (and company) ship the right product to your users.”

- Josh Elman

https://medium.com/@joshelman/a-product-managers-job-63c09a43d0ec#.5mhlul6l1

https://medium.com/@joshelman/a-product-managers-job-63c09a43d0ec#.5mhlul6l1

DATA IS A KEY INPUT AND FILTER IN PRODUCT

MANAGEMENT

COMPETITION, OTHER PRODUCTS,

BEST PRACTICES

BUILDLEARN

IDEAS

CORPORATE GOALS (SOME GOOD,

SOME BAD)GUTS & INSTINCTS

PARTNERS OTHER DEPARTMENTS

INDUSTRY TRENDS, ETC.

DATA

CUSTOMER INPUT DATA

COMPETITION, OTHER PRODUCTS,

BEST PRACTICES

PARTNERS

INDUSTRY TRENDS, ETC.

GUTS & INSTINCTS

OTHER DEPARTMENTS

CORPORATE GOALS

DATA AS A

FILTER

BETTER DECISIONS

CUSTOMER INPUT

@byosko

Company vision

Internal and external inputs

Perpetual problem / solution validation

Project Scope Creation

Sprints

Quarterly roadmap Quarterly roadmap Quarterly roadmapQuarterly roadmapBacklog

How it all comes together


@byosko

Company vision

Internal and external inputs

Perpetual problem / solution validation

Project Scope Creation

Sprints

Quarterly roadmap Quarterly roadmap Quarterly roadmapQuarterly roadmapBacklog

How it all comes together


DATA HAS AN IMPACT ON EVERY STEP IN THE PRODUCT DEVELOPMENT PROCESS


Product roadmaps (problem discovery)

Product roadmaps need to answer three critical questions:

1. Where are we going? (Vision)

2. Why are we going there? (Business Goals / Value Creation)

3. And how are we going to do it? (Resources & Planning)


Best practices for great product roadmaps

DON’T FORGET TO KISS!

• Focus on goals/outcomes not features

• Categorize and organize by themes

• Make sure problems are clearly defined and understood

• Timelines are loose at best (this isn’t the core focus)

• Use as a conversation starter, not as something writ in stone


Identifying the core problems / goals (objectives)

• Collect and analyze feedback / inputs

• Identify the core problems (goals) for the business (problem discovery)

• Identify the One Metric That Matters for each problem/goal and draw a line in the sand

• Hypothesize potential solutions

• Be outcome driven

Collecting input

Product & Design (defining goals /

objectives)

User & customer feedback

Sales

Marketing

Customer Support

Etc.

● In-person interviews ● Surveys ● Customer support

inquiries ● Real-time online

Supported by data

Your gut

Company visionYour own ideas


Collecting input

• Do you have a customer panel? (e.g. private FB group, mailing list, etc.)

• Can you schedule regular in-person sessions?

• Be wary of salespeople suggesting, “If we only built feature X, I could close this deal.”

• Attend customer and sales meetings; spend time on the help desk.

• Work with the product team to segment your user/customer base.


Intercom’s product roadmap inputs

“Building a great product is an art as much as a science. It requires making hard decisions and trade-offs, in circumstances ranging from being overwhelmed with data to having no data.”

https://blog.intercom.io/where-do-product-roadmaps-come-from/

https://blog.intercom.io/where-do-product-roadmaps-come-from/

Roman Pichler’s GO Product Roadmap

● Blends high-level roadmap and release planning

● Name can function as a theme

● Goal is a high-level purpose aligned with the stage you’re at (e.g. Stickiness, Virality, etc. or use AARRR)

● Roadmap leads to epics, stories, specs & MVPs

http://www.romanpichler.com/blog/working-go-product-roadmap/

The release date or timeframe

The reason for creating the release

The metrics to determine if the goal has been met

The high-level features necessary to meet the goal

The name of the new release

Date or timeframe Date or timeframe Date or timeframe Date or timeframe

Name / version Name / version Name / version Name / version

FeaturesFeaturesFeaturesFeatures

Goal Goal Goal Goal

MetricsMetricsMetricsMetrics

http://www.romanpichler.com/blog/working-go-product-roadmap/


Feature prioritization

1. Start with the key goals / outcomes & metrics

2. Rank order the solutions (features) based on value creation a. Identify a “line in the sand” for each solution (target)

3. Rank order the solutions based on effort

Ultimately, what will create

the MOST value for the LEAST effort?

Idea / Solution Score Target Engagement Value

Effort

Send more push notifications at regular intervals regarding key actions we want users to take inside the app

9 +10% MAU 3 1

Improve first user experience by providing users with a key action they need to take right away (no “white screen of death”)

5 +15% MAU 5 3

Add more robust reporting features for date range searches, saved reports

1.5 +5% MAU 2 4

Leaderboard for users hitting sales targets 1.5 +3% MAU 1 2

Goal: Increase Engagement (target: +15% MAU)

Try using a “dot voting” system per idea/solution

Only a rough estimate is needed here; goal is to have comparative values

@byosko@byosko

Use a simple weighted formula to score/rank items (e.g. Engagement Value * 3 / Effort)


Sample ranking system

TALKING BUSINESS: PRODUCT MANAGEMENT https://blog.intercom.io/rice-simple-prioritization-for-product-managers/

● Reach: How many people will be impacted by the new feature over a given period of time?

● Impact: How much will this new feature impact an individual user/customer? 3 for “massive impact”, 2 for “high”, 1 for “medium”, 0.5 for “low” and finally 0.25 for “minimal”.

● Confidence: How confident are we in our estimates? 100% is “high confidence”, 80% is “medium”, 50% is “low”.

● Effort: How much time will it take to deliver the new feature? Measure in “person months”.

Reach x Impact x Confidence

Effort

RICE Score

Intercom’s RICE model for feature prioritization

https://blog.intercom.io/rice-simple-prioritization-for-product-managers/


Data’s role in product management

• As an input for discovering interesting trends that are worth pursuing in product development

• As a filter for validating the relative importance of features

• As a check throughout the entire product development process from establishing a product roadmap, to feature prioritization, to scope creation and sprint management

• This is all about effective communication

MAKING DATA SIMPLER FOR EVERYONE ELSE2

NOW THAT WE KNOW HOW TO “TALK BUSINESS” IT’S

TIME TO “TALK DATA”

@byosko @byosko

YOU AND YOUR COWORKERS SPEAKING DATA

https://www.flickr.com/photos/jdhancock/8031897271

LEAN ANALYTICS: THE BASICS


1

2

3

Think about a project or product you’re involved with today

Can you write down the key metrics that matter?

We’ll compare after going through the next section

Quick exercise


Analytics is the measurement of movement towards business goals.


A GOOD METRIC IS:

Understandable

If you’re busy explaining the data, you won’t be busy acting on it.

Comparative

Active Users vs. Active Users/month

A ratio or rate

% Monthly Active Users

Behavior changing

You know how you’ll change your business based on what the metric tells you.

LEAN ANALYTICS: THE BASICS Day 3 - Lean Analytics


Herbert Simon

If a metric won’t change how you behave, it’s a…

BAD METRIC

Metrics help you know yourself


Acquisition

Hybrid

70% of retailers

You are just like

Customers that buy >1x in 90d

Once

2-2.5 per year

Your customers will buy from you

Then you are in this mode

1-15%

15-30%

Low acquisition cost, high checkout

Focus on

20% of retailers

Increasing return rates, market share

Loyalty>30% Loyalty, selection, inventory size

>2.5 per year

10% of retailers

(Thanks to Kevin Hillstrom for this.)

TYPES OF METRICS


VANITY vs. ACTIONABLE

Vanity Actionable

Makes you feel good but doesn’t change how you’ll act.

Helps you pick a direction and change your behavior.

“Up and to the right” These are good.


QUALITATIVE vs. QUANTITATIVE

Qualitative Quantitative

Unstructured, anecdotal, revealing, hard to aggregate.

Numbers and stats; hard facts, but less insights.

Warm and fuzzy. Cold and hard.

DISCOVER QUALITATIVELY.

PROVE QUANTITATIVELY.


Do Airbnb hosts get more business if their property is professionally photographed?


Case study: Does professional photography make a difference?

Gut instinct (hypothesis)Professional photography helps Airbnb’s business

Built a Concierge MVPSent 20 photographers out into the field

Measured resultsCompared photographed listings to control group

Made a decisionLaunched photography as a new feature to all hosts



EXPLORATORY vs. REPORTING

Exploratory Reporting

Speculative. Tries to find unexpected insights. Source of unfair advantage.

Predictable. Keeps you abreast of normal, day-to-day operations. Can be managed by exception.

Cool. Necessary.


Case study: Finding insights in the data

• Started as Circle of Friends• Leveraged Facebook early• Grew to 10M users fast

ENGAGEMENT SUCKED!


Case study: Moms are crazy (but in a good way!)

ENGAGEMENT SOLVED!

• Messages to one another were ~50% longer• 115% more likely to attach a picture to a post• 110% more likely to engage in a threaded conversation• Invited friends were 50% more likely to become engaged users• 60% more likely to accept invitations to the app


LAGGING vs. LEADING

Lagging Leading

Historical metric that shows you how you’re doing: reports the news.

Number today that shows a metric tomorrow: makes the news.

Start here. Try and get here.


Examples of leading metrics

A Facebook user reaching 7 friends within 10 days of signing up. (Chamath Palihapitiya)

A Dropbox user who puts at least 1 file in 1 folder on 1 device. (ChenLi Wang)

A Twitter user who follows a certain number of people, and a certain percentage of those people follow the user back. (Josh Elman)

A LinkedIn user getting to X connections in Y days. (Elliot Schmukler)


Case study: Buffer’s leading metrics revealed

Buffer discovered 3 leading metrics for long-term engagement:

1 People who install the Chrome extension

2 People who connect more than 1 social account

3 People who share 15 pieces of content in 7 days

Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec

Correlation vs. causation


CORRELATED vs. CAUSAL

Correlated Causal

Two variables that are related (but may be dependent on something else.)

An independent variable that directly impacts a dependent one.

Ice cream and drowning.

Ice cream and summertime.Drowning and summertime.


Cohort analysis

Case study: Putting basic data to use

Ricky (product manager) had some ideas for improving the “Proposal Send Screen” (based on qualitative feedback & gut), but before prioritizing this work, he digs into the data.

• 50% of people send proposals through Proposify (50% don’t) (quantitative) — Is this good or bad?

Ricky isn’t sure. So he’s going to need to look at additional data (exploratory):

• Churn• Size of customer• Proposal won rate• Any correlations here?

• Can also do some direct customer developer to learn more (qualitative)

• Might lead to additional, meaningful product dev (actionable)


1

2

Look at the metrics you wrote down before, how many of them stand up?

How can you make your metrics simpler for people to understand?

Quick exercise

LEAN ANALYTICS: THE FRAMEWORK3

LEAN ANALYTICS: THE FRAMEWORK

Lean Analytics Framework

What business are you in?

What stage are you at?

• E-Commerce• SaaS (freemium)• Mobile app (gaming)• Two-sided marketplace• Media• User-generated content

• Empathy• Stickiness• Virality• Revenue• Scale

WHAT STAGE ARE YOU AT?



Dave McClure’s Pirate Metrics


Dave McClure’s Pirate Metrics


Eric Ries’s Three Engines of Growth

Stickiness Virality Price

Approach

Math that matters

Keep people coming back.

Get customers faster than you lose them.

Make people invite friends.

How many they tell, how fast they tell them.

Spend money to get customers.

Customers are worth more than they cost.


Lean Analytics Stages

Empathy

Stickiness

Virality

Revenue

Scale

Stage

You’ve found a real, poorly-met need that a reachable market faces.

You’ve figured out how to solve the problem in a way that users will adopt, keep using, and pay for.

Your users and features fuel growth organically and artificially.

You’ve found a sustainable, scalable business with the right margins in a healthy ecosystem.

Gate


Lean Analytics Stages

Empathy

Stickiness

Virality

Revenue

Scale

Stage

You’ve found a real, poorly-met need that a reachable market faces.

You’ve figured out how to solve the problem in a way that users will adopt, keep using, and pay for.

Your users and features fuel growth organically and artificially.

You’ve found a sustainable, scalable business with the right margins in a healthy ecosystem.

Gate

Most projects/products/startups fail here


Case Study: Jumping the gun on product development

• Stage: Empathy/Stickiness• Model: E-Commerce• Originally tied to Instagram with

an “Insta-Order” feature


Case Study: Optimize for 1st purchase or repeat orders?

LEAN ANALYTICS: THE FRAMEWORK Day 4 - Lean Analytics

With Insta-Order feature Without Insta-Order

• 2x transactions• Lower bounce rate• Sign-in goals increased


“THERE ARE NO SHORTCUTS TO ANY PLACE WORTH GOING.” - Beverly Sills


Case Study: Localmind hacks Twitter (use a proxy to test your hypotheses)

BIGGEST RISK:Would people be willing to answer questions about a place in real-time?


Case Study: Localmind hacks Twitter (use a proxy to test your hypotheses)

Herbert Simon Tested results • The response rate to tweeted questions was very high • Good enough proxy to de-risk the solution and convince the team to continue

Ran an experiment on Twitter• Tracked geolocated tweets in Times Square• Sent @ messages to people who had just tweeted, asking questions about the area

Would people answer questions? • Before writing a line of code, they wanted to answer this question • This was their biggest risk; if questions went unanswered, the experience would suck

WHAT BUSINESS ARE YOU IN?


Business models

• E-Commerce • SaaS (freemium) • Mobile app (gaming) • Two-sided marketplace • Media • User-generated content

THE GOAL IS TO UNDERSTAND THE CUSTOMER’S LIFECYCLE / JOURNEY

THROUGH EVERY TOUCH POINT WITH YOUR PRODUCT.

Paid Direct WOM Search Inherent virality

Customer Acquisition Cost

VISITOR

User

FORMER USERS

Engaged user

Reactivate Trial over

Invite others

Paying customer

Disengaged

Account cancelled

Freemium / trial offer

Enrolment

Disengaged user

Cancel Cancel

Reactivate

FORMER CUSTOMERS

Billing info exp.

Resolution

Dissatisfied

Capacity Limit

UpsellingSignup conversion

rate

Free user disengagement

Freemium churn Reactivation rate

User lifetime value Customer lifetime value

Trial abandonment rate

DAU/WAU/MAU Paid conversion

Viral coefficient Viral rate

Paid churn rate

Support data

Tiering

Upselling rate

SaaS Customer Lifecycle

Returning Paid Direct Search Viral

Customer Acquisition Cost

VISITOR

E-Commerce Customer Lifecycle

Navigation Search Reco Engine

1-time buyer

Cart

Additions

Conversion

Logistics, delays

Delivery

Enrolment

Call to Action

Sharing

Unsocial buyer

Sharing rate

Returning rate

Customer Lifetime Value

Open rate, engagement

Transaction size

Emphasis on maximizing cart value, minimizing acquisition

costs

Bounced

Not interested

Abandoned

Bounce rate

Unsatisfied

Ratings, delivery issues

Feature usage, product discovery

Emphasis on repurchase rate, LTV


Case Study: WineExpress A/B tests what really matters

LEAN ANALYTICS: THE FRAMEWORK Day 4 - Lean AnalyticsThink of this over time…A B

Case Study: WineExpress A/B tests what really matters

B

• 41% increase in revenue per customer! (People bought a lot more product.)

• Conversion also went up, but was secondary in importance


All business models have issues

E-Commerce

SaaS

Mobile Apps

2-Sided Marketplace

Media

UCG

CAC vs. LTV — margins are usually small. A $10M e-commerce business is often considered quite small.

Freemium requires tens of millions of free users. They can be expensive to support. Will enough convert?

The average # of apps downloaded by North Americans per month is now 0. Monetizing is incredibly hard. Popularity is fleeting.

Chicken & egg problem. Supply and demand. How do you build up enough of both?

Real monetization requires hundreds of millions of engaged visitors. People’s attention is hard to capture and keep.

Content creation. Will it be good enough? Will enough people do it? Why?

YOU KNOW WHAT BUSINESS YOU’RE IN AND WHAT STAGE YOU’RE AT.

NOW WHAT?


The One Metric That Matters

The business you’re in

E-Commerce SaaS Mobile 2-Sided Marketplace UCG Media

The

stag

e yo

u’re

at

Empathy

Stickiness

Virality

Revenue

Scale

ONE METRIC THAT MATTERS


The One Metric That Matters

• The metric should indicate that your user experienced the core value of the product

• It should reflect user’s engagement and activity level

• It should be the “one thing” that indicates the business is heading in the right direction

• The metric ideally should be easy to understand & communicate across teams

• The OMTM will change over time as the business evolves


Case study: Moz cuts down on metrics to track

SaaS-based SEO toolkit in the Scale stage.Focused on net adds.

Net adds up:Was a marketing campaign successful? Were customer complaints lowered? Was a product upgrade valuable?

Net adds flat:Can we acquire more valuable customers? What product features can increase engagement? Can we improve customer support?

Net adds down:Are the new customers not the right segment? Did a marketing campaign fail? Did a product upgrade fail somewhere? Is customer support falling apart?

Examples of the One Metric That Matters


# of transactions (for merchants) # of nights booked sales

total time reading

https://medium.com/data-lab/mediums-metric-that-matters-total-time-reading-86c4970837d5#.tidx5bunjhttp://quibb.com/links/metrics-to-inform-your-model-lessons-from-square-stripe-and-quora

http://500.co/aircall-growth-uber/

monthly active users monthly recurring revenue (MRR)

https://medium.com/data-lab/mediums-metric-that-matters-total-time-reading-86c4970837d5#.tidx5bunj

http://quibb.com/links/metrics-to-inform-your-model-lessons-from-square-stripe-and-quora

http://500.co/aircall-growth-uber/


Metrics are like squeeze toys


http://bit.ly/BigLeanTable


The Layer Cake of Metrics

Project OMTM

Project OMTM

Project OMTM

Project OMTM

Project OMTM

Project OMTM

Department OMTM Department OMTM Department OMTM

OMTM: Business Help Indicator

DRAWING LINES IN THE SAND


Some interesting benchmarks

Growth5% / week (revenue or active users)

Time on site17 minutes

Free to paid2% of free users

Mobile file size< 50MB

Engaged visitors30% monthly users10% daily users

Paid load time< 5 seconds

Churn2% / month

CLV:CAC3:1


Case study: Solare draws a line in the sand


Case study: Solare discovers a leading indicator

50 reservations by 5pm 250 covers that night

=

THE LEAN ANALYTICS CYCLE

Draw a new line


Lean Analytics Cycle

LEAN ANALYTICS: THE FRAMEWORK Day 4 - Lean Analytics

Pivot or give up

Try again

Success!

Did we move the needle?

Measure the results

Make changes in production

Design a test

Hypothesis

With data: find a

commonality

Without data: make a good

guess

Find a potential improvement

Draw a linePick a OMTM

Q&A / DISCUSSION


Map your business model

1

2

Map your business as a systems diagram

Think about all the touch points of a user / customer and how they interact with your business (idea, product, people, etc.)

3 What are the riskiest areas / biggest areas of uncertainty?

4 Where are you going to focus? Can you identify the OMTM at each stage of your business?

Q&A / DISCUSSION

CONCLUSIONS

THE END!

1Data people need to be able to

“talk business”

So where are we at now?

• Understand the types of innovation and how a company is approaching new product development & business models

• Lean Startup applies beyond startups, but it’s not easy

• Understand how the sausage is made (product management)

• Product isn’t built in a vacuum; there’s a rigorous, experiment-oriented process that helps

• Get involved in product roadmaps & feature prioritization

• Data is both an input and a filter into the entire process of building a business

THE END!

1Data people need to be able to

“talk business”

So where are we at now?

• Understand the types of innovation and how a company is approaching new product development & business models

• Lean Startup applies beyond startups, but it’s not easy

• Understand how the sausage is made (product management)

• Product isn’t built in a vacuum; there’s a rigorous, experiment-oriented process that helps

• Get involved in product roadmaps & feature prioritization

• Data is both an input and a filter into the entire process of building a business

2Everyone else needs to be able to

“speak data”

• Data is the common language that everyone needs to be reasonably fluent in

• We need to simplify the data (not what we track) for others

• Data needs to be tied to real/core business goals • The One Metric That Matters is a good construct

for forcing simplification and focus • It’s important to understand the business you’re in

and the stage you’re at in order to identify the right metrics to focus on

Once, a leader convinced others in the absence of data.

Now, a leader knows what questions to ask.

Alistair Croll [email protected] @acroll

Ben Yoskovitz [email protected] @byosko

mailto:[email protected]

mailto:[email protected]



DESIGNING A BUSINESS MODEL

Data & Analytics

Lean Analytics - Communication Layer into the World of Business