Embed Size (px)
Citation preview
Your%Presenter
Michael%Aagaard
4 Senior%Conversion%Op:mizer%at%Unbounce%
4 Full4:me%CRO%for%the%lat%seven%years%
4 Interna:onal%speaker%
4 Founder%of%ContentVerve.com
Join%the%conversa:on
4%Include%the%hashtag%#thinkppc)in%your%TwiJer%tweets.%4%Or%use%the%webinar%ques:on%box%to%send%us%ques:ons.
“When can I stop my A/B Test?”
“Why run an A/B Test?”
To get a representative picture of how a variant would perform if implemented permanently “in the wild”.
To get a representative picture of how a variant would perform if implemented permanently “in the wild”.
Do Not Use 95% Significance Level as a Stopping Rule
1
After 1 day
+151.4% / significance 97%
655 visitors / 18 conversions
After 2 days
+98.4% / significance 95%
988 visitors / 23 conversions
After 1 week
+0.2% / significance 50%
3031 visitors / 58 conversions
After 2 weeks
+18.1% / significance 77%
5277 visitors / 78 conversions
After 4 weeks
+36.2% / significance 96%
Total sample: 9396 visitors / 122 conversions
What we have now: 9,396
What we need: 34,304 visitors in total
What we’re missing:24,908 visitors 73% of the required sample size.
http://www.evanmiller.org/ab-testing/sample-size.html
http://abtestguide.com/calc/
“Formulas don’t know whether they are being used properly, and they don’t warn you when your results are incorrect.”
- Deborah J. Rumsey, PhD
“Real accuracy depends on the quality of data as well as on the sample size.”
- Deborah J. Rumsey, PhD
Worth Getting Familiar With:
- Power Level - P-Value - Margin of Error - Confidence Interval - Null hypothesis / alternative hypothesis
Make Sure You Collect Enough Data to Yield Meaningful Insight
2
Full weeks Full business cycles
Weekly Business Cycle
Full weeks Full business cycles
Week One Week Two Week Three Week Four
Test duration: 2-4 weeks (2 or more business cycles)
Sample size: Pre-calculated sample size (users)
400+ conversions (goals)Significance: 95% or higher
Power: 80% P-Value: 0%
What I look for:
Be Careful With Averages - They Hide the Truth
3
Devices
Browsers
Channels
Country
Days of the week
Hours ofthe day
User types
Motivation
Campaigns
Age
Gender
Gender
BrowsersDevices
Browsers
Channels
Countries
Days of the week
Hours ofthe day
User types
Motivation
Campaigns
Age
Days of the month
Gender
Devices
Experiment
Integrate Test Data with Your Web Analytics Setup
And Get the Full Picture
Device?Channel?
Campaign?User Type?
Formulate a Clear Test Hypothesis
4
Why do we think we need to make a change?
What is it that we want to change?
What impact do we expect to see?
How will we measure this impact?
1. Because we saw [data/feedback]
2. We expect that [change] will cause [impact]
3. We’ll measure this using [data metric]
Because [exit surveys indicated that saving money is most important to our users.]
We expect that [tweaking our value prop to reflect this] will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our key metric.]
Because [exit surveys indicated that saving money is most important to our users.]
We expect that [tweaking our value prop to reflect this] will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our key metric.]
Because [exit surveys indicated that saving money is most important to our users.]
We expect that [tweaking our value prop to reflect this] will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our key metric.]
Because [exit surveys indicated that saving money is most important to our users.]
We expect that [tweaking our value prop to reflect this] will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our key metric.]
We expect to see reliable results in [four weeks.]
Learn Statistics!!!
5
http://amzn.to/1RFNBwA
Live%Q&A%Time
Have%more%questions?
Contact%Us%Directly
• Webinar%Feedback:%[email protected]
____________________________________________
Thank'you'for'attending' our'webinar!'#thinkppc
