Growth Experiment Process test before you invest

February 22, 2016 2

The only true wisdom is in knowing you know nothing.

Socrates

February 22, 2016 3

We need to reengineer companies to focus on figuring out who the customer is, what's the market and what kind of product you should build.

Eric Ries, Lean Startup

February 22, 2016 4

I would say, as an entrepreneur

everything you do - every action

you take in product

development, in marketing,

every conversation you have,

everything you do - is an

experiment.

If you can conceptualize your

work not as building

features, not as launching

campaigns, but as running

experiments, you can get

radically more done with less

effort.

Eric Ries, Lean Startup

February 22, 2016 5

You are here Active Users*

Industries Sectors

Segments

You make assumptions about your path to subscribers and active users.

Personas

Subscribers

Successful experiments get

you here*

The Process

February 22, 2016 6

Brainstorm Prioritize Test Implement Analyze Systemize

[sample]Activation rate= step 1 conv. + step 2 conv. + step 3 conv.

Probability;Impact; Resources(effort/cost)

Write experiment design

Execute on experiment

Understandimpact, accuracy, and why?

Capture lessons learned.

Generate ideas for each step in conversion process. Create backlog of experiments.

Make assumptions that will be tested.

What is minimal test required to understand hypothesis?

Put plan and schedule in place. Execute.

Results of experiment? How close to your hypothesis? Why that result?

Productize as much as possible, build playbooks to make repeatable.

Tool: Backlog document

Tool: Experiment document + Backlog doc

Tool: Experiment document

Tool: project plan. Timing, tasks, resources, tools.

Tool: Experiment document(learnings and action items)

Playbooks

Source: Brian Balfour, http://www.coelevate.com/

Set Goal

Objective: [sample]Improve activation rate.

Timeframe: 30 -90 days

Goal 1: (hit 90% of time)Goal 2: (hit 50% of time)Goal 3: (hit 10% of time)

ExecuteObjective

Timeline

February 22, 2016 7

0 9030 60

Set Goals

Brainstorm

Prioritize

Test

Implement

Analyze

Systemize

Weekly Dashboard updates

Backlog Doc

Experiment Doc

Project plan

Experiment Doc v2

Playbook

Fees

February 22, 2016 8

Option 1 Option 2 Option 3

# of Experiments 1-2 3-4 5-8

Time period 90 days 90 days 90 days

Tasks • Facilitate brainstorming/prioritization

• Produce experiment documents

• Manage experiments• Document findings• Capture lessons learned

• Facilitate brainstorming/prioritization

• Produce experiment documents• Manage experiments• Document findings• Capture lessons learned

• Facilitate brainstorming/prioritization

• Produce experiment documents• Manage experiments• Document findings• Capture lessons learned

Deliverables • Backlog document• Experiment document • Project plan• Dashboard metrics• Lessons learned playbook

• Backlog document• Experiment document • Project plan• Dashboard metrics• Lessons learned playbook

• Backlog document• Experiment document • Project plan• Dashboard metrics• Lessons learned playbook

Assumptions Client will make available the necessary staff and tools require to run experiments.

Client will make available the necessary staff and tools require to run experiments.

Client will make available the necessary staff and tools require to run experiments.

Monthly Fees $1,500/mon $2,500/mon $7,500/mon

February 22, 2016 9

Sample Documents

Sample Docs

February 22, 2016 10

No. Experiment Name Status Category Metric Prediction Marketing Engineering Design1 FB Ads targeting beta sign ups Idea Acquisition CPA (facebook) -18% 1/2 day 0 0

2 Activation email step - reroute Defining Activation Activation rate +5% 1 hour 1 day 1 day

3 Retention - email social validation Defined Retention First 24 retention +2% 1 Day 1 hour 1/2 day

4 Revenue - email recapture/pro upgrade Defined Revenue CR/pay +5% 0 1 Day 0

Hypothesis ResourcesExperiment Backlog

Experiment Backlog

Sample Docs

February 22, 2016 11

Experiment Document

Types of Experiments

February 22, 2016 12

ConversionAcquisition Activation Usage

Case Story

February 22, 2016 13

Objective: Improve conversion rates for trial subscribers

Approach: Improve conversion rates to paid status for trial subscribers using A/B testing on custom landing pages. Testing different messaging and offers and tracking conversations.

Trial Subscribers

Split

Custom Landing page 1

Message 1

Custom Landing page 2

Message 2

Measure Results

Landing page/message 2

converted at 8%. Landing

page/message 1 at 14%.

Convert to Paid

Toolbox: Collaboration Site

February 22, 2016 14

Collaboration site

Backlog of experiment ideas

Experiment documents

Project timeline/ milestones

Toolbox: Tracking Dashboard

February 22, 2016 15