Upload
bridgeline-digital
View
520
Download
0
Tags:
Embed Size (px)
DESCRIPTION
AB Testing requires a considerable amount of time, work, and planning. But as with most ventures that involve those components, the valuable results and insight gained from AB Testing more than payoff the initial investment. Watch a recording of a webinar with JR Hopwood from our iAPPS Success Group (iSG) to learn about the process of AB Testing game and how to convince others in your organization to AB Test as well.
Citation preview
A/B Testing 101with JR Hopwood
A B
Overview
Introduction
Need convincing?
Deciding on a test
Evaluate a test
Wrap up A BVs.
Introduction
About the presenter
JR Hopwood With over a decade of web analytics and website testing experience, JR works with iAPPS
website partners to assist them in getting the most out of their web applications.
The iAPPS Success Group (iSG) is a team of support and digital services specialists who help Bridgeline Digital customers succeed online.
Bridgeline Digital iAPPS Success Group
Need Convincing?
Need Convincing?
Do you like peanut butter?
Do you like jelly?
Do you like PB&J?
Need Convincing?
You just tested! Peanut Butter has always worked for you… But you wondered if Jelly could add some spice… And then, WHOA, what about adding them together???
A/B Testing works the same way We take something that may be ‘working’ today and see if we can improve on it Sometimes we’re right, sometimes we’re wrong
A/B Testing provides the numbers You no longer have to rely solely on opinions Testing analytics will show you precisely what works and what doesn’t
Need Convincing?
AB Testing allows you to keep what is working today and just test a subset of traffic
Current Page Test Page
75% 25%
All Visitors
Ways to create subsets:
PercentageTraffic Source
Site ActionEtc…
Need Convincing?
Now what if that translated to more conversions?
Current Page Test Page
75% 25%
All Visitors
5% conversion 10% conversion (+100%)
If each conversion was $1 and you had 1,000 visits you
made $12.50 more than you would have!
Need Convincing?
Conversely, what if that translated to less conversions?
Current Page Test Page
75% 25%
All Visitors
5% conversion 2.5% conversion (-50%)
If each conversion was $1 and you had 1,000 visits you only lost $6.25 over what you
would have made… you
tried.
Need Convincing?
Stop the bleeding! Luckily with AB Testing you can turn it off…
… just try and wait until there is some statistical significance Sometimes conversion numbers are skewed due to visitor behavior changes, so be patient Example: No one buys on the weekends
Statistical Significance
=Confidence
Need Convincing?
Improve ROI or minimize the losses Testing can only help conversion in the long run
Even a failed test can prove to you that you SHOULD NEVER do that variation again
Since you aren’t sending all of your traffic to the new variation the gains and losses have a lessor effect on your current bottom line
…but could provide a substantial change to your future one!
This gives you the opportunity to try new things without going ALL IN Just remember, you need enough traffic to gain Statistical Significance
Need Convincing?
Here’s an example scenario I see all the time… Page A converts at 25% and has for several months… Marketing team doesn’t want to touch it, because… CEO doesn’t want to lose revenue…
What is a marketer to do? Remind CEO that we aren’t changing the page for everyone
Do the analysis… show that even a 3% bump in conversion on that page translates into $n increase in revenue
Calculate the resource time of those involved (including agencies) Compare this against the potential gains… and you’ll get the green light
Even if your site is non-revenue driving still show conversion increases Show how these translate into real people If your site just pushes promotional brochure-ware, but for each brochure downloaded
your company makes $n in return… get the picture?
CEO’s like the bottom line Once you prove that testing can have an immediate impact on bottom line the tide will
turn and your CEO will start asking YOU when the next test is
Decide what to test
Decide what to test
Define a Goal
The How’s Gut / Common Sense Web Analytics Settle an argument… I’m not kidding
The What’s Design Functionality Content
Decide what to test
Don’t just test to test… Make sure you are staying true to your KPIs Whether you sell widgets or provide information you have something to test
We want to test because we want to… Increase revenue Increase conversion Decrease Bounce Rate Decrease Exits
Once your goal is defined the fun begins…
Decide What to Test | Gut / Common Sense
Why test a page you just feel doesn’t have the best usability? GUT…
Why NOT test a page that isn’t included in your funnel and that no one? COMMON SENSE…
It’s OK to use these as a basis for a test
Decide what to test | Gut / Common Sense
Some questions to consider:
Where is the largest fallout / drop off / abandonment?
What page presents the largest impact to conversion?
Does a page not match your brand?
Does a landing page not match the offer?
Decide What to Test | Web Analytics
Numbers don’t lie … often
Before making decisions based on your analytics please be sure it is implemented properly!
Metrics to consider Bounce Rates Exit Rates Funnel Abandonment
Decide What to Test | Web Analytics
Some questions to consider:
Are there pages that have a higher bounce rate than most?
Are there pages that have a higher exit rate than most?
Does a page in a funnel have an unusually high abandonment?
A little gut… These are great questions, but sometimes there are explanations…
Does an exit link to off site? Do you push visitors to call or email? Does the page have an iFrame, Flash or HTML 5 element?
Decide What to Test | Settle an argument
Marketing person A says one thing and Marketing person B says another Test it and prove which one does better!
Decide What to Test | Design
Obviously design changes are the low hanging fruit of website testing• Huge gains have been achieved by testing:
• Buttons• Imagery• Layout
Don’t go overboard…• The more changes you make to a page the less specific the test can be
• Example: If you wanted to test a form location on a landing page to see if the left or right side converted better (could be a great test)
• However, if at the same time you change the button color or copy…• What actually tested better? The layout change or the other elements??
Multivariate also has a time and a place• Multivariate testing allows for multiple changes to layout and sections on a web page at one
time• This can get tricky if you don’t have enough traffic OR if your tests are WILDLY different
• My recommendation would be to only move to multivariate after a few AB Tests so you get the hang of it and what to look for in the evaluation
Decide what to test | Functionality
There are so many ways to code a widget or functionality on a website• Think about how many Date Pickers you’ve seen out there…• Sometimes though there is a need to test these
• In our Date Picker example, is showing 2-months better than just showing 1?• What about a login, should you ask for user name and password at once or just user
name, then password on the next page?
Get layout right first…• Before you turn to functionality as the problem are you doing a good enough job explaining or
calling out on the page?
Decide What to Test | Content
It is amazing how the smallest content changes can make a difference• What would you click on more?
• “Begin Reservation”• “Continue”
• What content would you engage with more?• “You really need this product”• “We really think you should have this product”
Don’t couple with design…• As mentioned previously, if you change the layout or button colors you won’t be able to tell for
sure whether it was the copy or the other changes!• If you must do this at the same time, do a multivariate test so you can see how the variations
perform independently AND together
Evaluate a Test
Evaluate a test
Evaluate a Test
Testing platforms come with evaluation tools out of the box, use them!• The metric you’re looking for is Statistical Significance, Confidence or Probability to Beat
Original• What this shows is if the variation were to run 24/7/365 what is the chance it will outperform
the control, or original• Why this is necessary is that in the test the page actually ISN’T being shown to every
visitor• Statistical analysis crunches the numbers and determines the probability based on
various factors
Evaluate a Test
The higher the Significance the more confident you can be to move forward with that variation
• Obviously the more visitors that conduct the test and perform better on the competing variation the higher that significance will be
• Example: If you throw me 10 pitches and I hit 7, you can be somewhat sure I have about a 70% chance of hitting the next pitch. But if you throw me 100 pitches and I hit 70 you can be more confident in that 70%, since it has been confirmed over a longer period and by several pitches
DO NOT stop a test if Statistical Significance has not been reached• This only skews the data set and may not play out once you go to 100%
Things to Consider for your First Test
Don’t go overboard, but be bold• Start with elements that are easy to spot• Don’t let your first test simply be copy changes• The reason for this is so in your first test you’ll be able to see bigger variations in conversion
data and thus be able to learn about test confidence
Sometimes the end is better…• If you lose 10% of your visitors on the final step of your booking flow just think how quickly
your ROI could change if you lost only 5%... Or less• Sometimes small changes on those final pages are the difference between clicking “BILL ME”
and exiting• These visitors made it all the way to the final step… they didn’t leave because they are no
longer interested!
Have FUN!• Get other departments involved to secure buy in• “Hey design team, if you could redesign this page what would it look like?”• “Hey sales team, what would you change to make the site better?”
• Warning, this team is competitive!!!!• With A/B Testing the numbers don’t lie… a variation wins and variations lose
First Test Checklist
Have I defined my website goals?
Have I evaluated where a problem may exist?
Have I come up with a solid competing variation(s)?
Have I come up with a hypothesis on which variation will win?
Have I devised a segment strategy for who sees the test variations?
Is one variation outperforming another?
Do we have statistical significance?
Go with that variation
Repeat
Now it’s your turn to become the mad scientist!
Q & A
Thank You!
this webinar prepared bythe iSG Digital Services Team
at Bridgeline Digital
For more information contact iSG at:
Or call Seth Rich : (781) 404-5423