View
4.756
Download
3
Category
Tags:
Preview:
DESCRIPTION
Tim Wilson's presentation at eMetrics San Francisco in April 2013 covering an approach for analysts to establish and manage a well-defined process for hypothesis discovery, prioritization, and testing to help pivot companies to be data-driven -- to orient towards a culture of validated learning.
Citation preview
@tgwilson /emetrics eMetrics, April 2013
Web Optimization Process Management
Tim Wilson VP of Measurement & Analytics, Clearhead
@tgwilson /emetrics
@tgwilson /emetrics
The analyst as an agent of change
@tgwilson /emetrics
Our job: reorient the business towards validated learning.
@tgwilson /emetrics
Everyone wants to be data-driven these days.
@tgwilson /emetrics
People Process
Technology
Being Data Driven
@tgwilson /emetrics
People Process
Technology
Being Data Driven The Business’ perspective
@tgwilson /emetrics
People Process
Technology
Being Data Driven The Analyst’s perspective
@tgwilson /emetrics
People Process
Technology
Being Data Driven My perspective
@tgwilson /emetrics
Let’s talk about a process revolution.
@tgwilson /emetrics
Every revolution needs a mobilizing catch-phrase.
@tgwilson /emetrics
A.D.A.P.T. to
Act & Learn
@tgwilson /emetrics
Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
@tgwilson /emetrics
Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
@tgwilson /emetrics
Alignment means clearly established goals and KPIs.
@tgwilson /emetrics
@tgwilson /emetrics
@tgwilson /emetrics
Magic questions. 1. What are we trying to achieve?
2. How do we know if we’ve done that?
@tgwilson /emetrics
Two Magic Questions
A Process: Goals and KPIs
Establish Targets
Establish Appropriate
Measures
− What are we trying to achieve? − How will we know if we’ve done that?
− Open-ended questions − “What?” and “How?” (not “Why?”) − The power of restatement
− Historical data − Back-of-the-napkin creativity − Bracketing
@tgwilson /emetrics
Online Revenue Engagement ReachDrive online revenue through the presence of xxxx Increase overall site engagement Increase the reach of [site].com
Conversion Rate Lift +14.3% New Profiles Created 2,707 SEO Visits to xxxx 116
Visits that interacted with xxxx 7.08% Total Profiles created since launch 3,391
Visits that did not interact with xxxx 6.19%
% Interacting w/ xxxx 13.8%+3.19% +3.14% +1.6% Visits that interacted 36,605 Social Shares of xxxx 251
Visits that did not interact 229,125
Avg. Order Value Lift +2.7%
Visits that interacted with xxxx $23.37
Visits that did not interact with xxxx $22.76
Time on Site Lift +77.6% Social Referral Visits 46Visits that interacted with xxxx 24:49
$23.04 $22.96 +0.3% Visits that did not interact with xxxx 13:59
Total Revenue Lift $8,976
Conversion rate lift only $7,389
AOV lift only $1,389
AOV lift on incremental orders $199
Annualized Lift (ignoring seasonality) $192,727 Refer to the glossary for a detailed definition for each metric.
Overall AOVfor the site
Overall conversion rate for the site
Projected ratewithout xxxx
Sitewide % Lift
Projected AOVwithout xxxx
Sitewide % Lift
0.0%5.0%
10.0%15.0%
3/10 3/26
$0
$20
$40
3/10 3/26
050
100150200250
3/10 3/26
0.0%
5.0%
10.0%
15.0%
20.0%
3/10 3/26
0
5
10
15
20
3/10 3/26
0
10
20
30
40
50
3/10 3/26
0
2
4
6
8
3/10 3/26
00:0005:0010:0015:0020:0025:0030:00
3/10 3/26
@tgwilson /emetrics
Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
@tgwilson /emetrics
An analysis is nothing more than testing a hypothesis.
@tgwilson /emetrics
Where do hypotheses come from?
@tgwilson /emetrics
Analysts
Our Stakeholders
Everyone Else at the Company
@tgwilson /emetrics
Everyone in the business has hypotheses!
@tgwilson /emetrics
The trick is getting them articulated clearly.
@tgwilson /emetrics
What we say to marketers
“What is your hypothesis?”
What they hear
“I’m dragging you back to the 8th grade
science class you hated.”
@tgwilson /emetrics
I believe that…
If I am right, then… [some idea]
[some action]
@tgwilson /emetrics
Each idea can be added to your hypothesis library. Google Spreadsheet Excel uservoice.com Sharepoint Lotus Notes Jira
@tgwilson /emetrics
Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
@tgwilson /emetrics
To assess the effort required, you have to determine the approach.
@tgwilson /emetrics
Flickr / justinbaeder
@tgwilson /emetrics
The analyst must be aware of the full tool belt.
@tgwilson /emetrics
If I am right, then… [some action]
I believe that… [some idea]
To test this, I will… [tool/technology/approach]
@tgwilson /emetrics
Web Analytics
Voice of the Customer
A/B (and MV) Testing
Usability Testing
Email Surveys
Social Media Analytics
Focus Groups
Secondary Research
What’s in your company’s toolbelt?
@tgwilson /emetrics
Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
@tgwilson /emetrics
It’s not a simple formula.
@tgwilson /emetrics
But a formula can guide the discussion.
@tgwilson /emetrics
Likelihood belief is right
Likelihood change would occur
Likely scale of impact
Alignment w/ business goals
Effort to test (approach)
M L H
M L H
M L H
M L H
M L H
@tgwilson /emetrics
Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
@tgwilson /emetrics
@tgwilson /emetrics
Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
@tgwilson /emetrics
Driving action requires effective communication of the results.
@tgwilson /emetrics
The data does not speak for itself.
@tgwilson /emetrics
@tgwilson /emetrics
Who needs to do what?
@tgwilson /emetrics
Lead with the business problem statement.
@tgwilson /emetrics
Follow with the answer and the action.
@tgwilson /emetrics
Do not show your work.
@tgwilson /emetrics
@tgwilson /emetrics
And then track to action.
@tgwilson /emetrics
As a result, we… [took the action]
We tested… [the idea]
Resulting in… [quantified result]
@tgwilson /emetrics
Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
@tgwilson /emetrics
Learning means looking beyond the near-term action.
@tgwilson /emetrics
Hypothesis tested Action taken
Hypothesis tested No action taken
Hypothesis tested Action taken
Hypothesis tested Action taken
Hypothesis tested No action taken
Hypothesis tested Action taken
Hypothesis tested Action taken
Hypothesis tested Action taken
Deeper learning
Deeper learning
Deeper learning
@tgwilson /emetrics
Reorienting the business towards validated learning
@tgwilson /emetrics
Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
@tgwilson /emetrics
clearhead.me/emetrics
This presentation Guide: getting to great KPIs Hypothesis library template Approach options self-audit template Recommended reading
@tgwilson /emetrics
nthanksN
Recommended