Scaling your Testing Program

Vivek VilatiaSenior Product Manager, VWO

Anshul GurbaxaniAssociate Product Manager, VWO

Prem KujurCustomer Success Manager, VWO

Ask Questions on Twitter

#AskVWO

What is A/B testing

In the most simplest form, A/B

Testing is comparing two versions

of a web page to see which version

performs better for a defined

conversion goal

Type of Tests

A/B TestChange any element or section

of a webpage and test both

versions.

Split URL TestTest performances of separate

URLs if the landing pages are

significantly different.

Multivariate TestRun tests on multiple variation

combinations together. Needs

sufficient traffic.

Graduating to

Advanced tests

Simple A/B tests Advanced A/B tests

➔ Great starting point

➔ Smaller teams

➔ Easy to setup

➔ Quick wins

➔ Test changes like copy,

arrangement of elements,

CTA colors etc

➔ Test advanced use cases with

greater impact

➔ Conversion optimization program

➔ Self-sufficient team

➔ The right tool-set

➔ Management buy-in

➔ Tons of perseverance

Audience Changes MetricsPages

Key steps in setting up your tests

Test pages | Defining a set of similar pages for your test

Asterix * in the URL matches pattern would replace the variable part of the URL

Represented by * in the URL matches pattern

Audience | Complex targeting rules

Visitors logged in on

your website

Are either from India or have their

preferred currency as INR

Changes | Code editor for complex changes

Changes | Code editor for complex changes

Changes | Conditional Test Activation

Changes | Conditional Test Activation

SignUp Form Type Form

Measuring theright metrics

Measuring the right metrics

Micro Conversions Macro Conversions

Measuring the right metrics

Never make your tests live without proper sanity checks

Quick previews

Cross Browser testing

Live previews

Check cookies using a plugin

Making sense of your A/B testing reports

Making sense of your A/B testing reports

Run your test for at least a week.

Along with a primary goal, have related secondary goals.

Ensure that your primary goal represents the direct and

exclusive impact of the changes being tested.

Making sense of your A/B testing reports - Best Practices

Slice and Dice your test results to learn the trends and

behaviour of various user segments.

Testing radical changes at times might just do the trick

Identify micro-conversions and important correlative metrics

increase in a given variation, implement it.

Inconclusive Tests

Ensure that you don’t use segments with very less data.

Make sure there is no error in the test setup.

Ensure that you are not running multiple tests

simultaneously in the same part of the journey you are A/B

testing

Inconclusive Tests - Pitfalls

Thank You!

Q&A