Upload
growco
View
2.596
Download
0
Embed Size (px)
Citation preview
Lessons Learned From A/B Testing Wins and Losses
WHO I AM: GROWTH HACKER
ANDY
Growth Team/groTH tem/ n.
A group within the company that develops the product experience in a way that creates more activated users
! Pre Product/Market fit: build the growth engine
! Post Product/Market fit: optimize the engine (make it bigger)
! Any time: generate hacks, non-obvious ways to reach your goals
Definition
Building the Testing Engine
1
GET THE PIECES IN PLACE BEFORE ADDING GAS
Do things that don’t scale
Test yourself to product market fit
What type of product are our users interested in?
How can we reach this users the best way?
Pour Gas on it
WE HAD TO EXPERIMENT WITH A LOT OF THINGS
How do you build what users actually want?
Child ID
Dog TrackerFamily Locator
Lost Phone
Ads Landing Pages
THINK OF IT AS DIVIDE & CONQUER
Optimizing the Test Engine
2
RISKLOW HIGH
Big Swings
Optimizations
REW
ARD
HIGH
Start Ups need step function take bold bets
Scalable growth through testing
Lather, Rinse, Repeat. Continuous Improvement
Evaluate Results
Test Design
Develop Hypothesis
Implement Leanings
Developing hypotheses
How to Develop Hypotheses1. Leverage the team, institutional knowledge,
understand important “Themes”
2. Qualitative studies, surveys
3. Quantitative analysis
Test Design
Test Design
How to Design Tests Effectively1. Determine right testing format
2. Optimize test to avoid overinvesting
3. Execute quickly, full instrumentation for analysis
Evaluate Results
How to Evaluate Results1. Analyze core metrics impact of each variant
2. Ensure all numbers are statistically significant
3. Understand long term impacts and side effects
Evaluate Results
Implement Learnings
How to Implement Learnings1. Understand what caused the change
2. Determine how this learnings applies to product aspects
3. Promote visibility of learnings/takeawaysImplement Leanings
Qualitative Tests Painted Door Tests Feature Tests
INVESTMENT COSTLOW HIGH
Painted Door
Top of Funnel Test
Full FeatureMany Some Few
Types of tests avoid overinvesting
Qualitative Tests
PurposeDetermine interest in product via conversations with your users and potential users
KeyTest as many ideas as possible but keep in mind that people want to please you
ExampleEmergency ContactsETA Sharing Carpooling
Qualitative Tests
Feature Tests
PurposeBuilding a functional feature and see how users interact
KeyDon’t build the full feature, start at the top of the funnel and work your way down
ExampleEmergency ContactsRecommenderFUE
Add EC
Send EC
Accept EC
Create ACT
(Partial) Feature Tests
Aggressive vs. Passive Tests
Passive Tests End Point Aggressive Tests
LEARNINGSLOW HIGH
HOW TO TRUST YOUR RESULTS?
Test Design - Explicit Control Variant Allocation
Design variants to have an additional control group
➔ Ensures control group meets test user criteria➔ Reduce risk of variant assignment issues
Test Design - Explicit Control Variant Allocation
All Users
Test Design - Explicit Control Variant Allocation
Eligible for experimentIneligible for experiment
Test Design - Explicit Control Variant Allocation
Eligible but excluded from experiment
Ineligible for experiment Included in experiment
Test Design - Explicit Control Variant Allocation
Treatment 1
Eligible but excluded from experiment
Ineligible for experiment
Control Variant 1
Variant 2
Test Design - Explicit Control Variant Allocation
Always consider keeping long-term holdout groups
➔ Understand potential impacts that may not be seen early in the user lifecycle
➔ Enables measurement of how impacts rollup into product growth
“A GOOD PLAN, VIOLENTLY EXECUTED TODAY, IS BETTER THAN A PERFECT PLAN EXECUTED NEXT WEEK.”
GENERAL GEORGE S PATTON