The Ins and Outs of Paid Search A/B Testing | 1

The Ins and Outs of Paid

Search A/B Testing

Written By: Dawn Busa, SEM Specialist II

Edited By: Tim Lavelle, Director SEO/Social Media

The Ins and Outs of Paid Search A/B Testing | 2

Introduction

Search Engine Marketing is a highly competitive field. To remain effective in the space, advertisers need

to be smart about how they approach reaching customers via paid search. One strategy that helps

ensure the success of paid search campaigns is regular A/B testing.

What is A/B Testing?

A/B testing compares a single variable to figure out which version of that variable provides the best ad

performance.

These side-by-side comparison tests enable paid search marketers to fine-tune their message in order to

achieve goals in a more efficient way, whether they’re optimizing for impressions and brand awareness,

or tangible conversions like purchases, form submissions, phone inquiries or leads.

A/B testing is typically conducted across various components of the paid search campaign, including

such elements as landing page design, calls to action, page content, and ad copy.

Why Conduct A/B Testing?

From a purely statistical standpoint, improving ad performance via A/B testing can improve click-

through rates, conversion rates, ad ranks and Google Quality Score, reducing click prices, optimizing

advertising spend and leading to ROI improvements.

Process: Brainstorm and Research

Before conducting A/B testing, it is important to first consider the following factors:

Defining goals and success metrics

Reviewing competitor behavior

Analyzing ad performance

Selecting an interface

Defining Goals & Success Metrics

To ensure a successful experiment, advertisers must first determine what they’re seeking to accomplish

– is the goal of this campaign solely focused on brand awareness? If so, a high click-volume and click-

through rate are strong key performance indicators.

Is the goal to generate leads? Look at conversions and cost-per-lead to provide a better understanding

of how the ad is performing.

Every goal has a different indicator, so goals must first be defined before performance can be reviewed,

and advertisers may apply the results of their A/B tests towards achieving those goals.

The Ins and Outs of Paid Search A/B Testing | 3

Reviewing Competitor Behavior

To ensure competitiveness, advertisers must review the landscape of their particular industry. Who are

the main competitors, and how are they utilizing Paid Search to conduct their own business? By

researching their behavior, advertisers can generate ideas on how to position their own ads in such a

way that they’re differentiated from the competition, improving their ability to capture attention.

A simple Google search for keyword phrases that you’re considering bidding on provides great insight on

the competitive landscape. For example, if you were considering running a campaign for a Mortgage

Broker, the first step in your competitive research would be to search “mortgage rates” on Google,

which returns the following results:

In this space, it’s obvious that listing APR rates in either the headline or ad copy is the conventional

approach, and most ads also include the terms “Mortgage” and “Rates” in their ad copy as well. With

this basic information, you can begin testing these elements in your own ad copy, seeing how they’ll

affect campaign performance.

In addition to simple searches, leveraging third party data aggregation tools, such as SpyFu,

KeywordSpy, or others, will provide a high-level overview of what competitors are doing, and how

they’re getting it done. SpyFu, in particular, offers an AdHistory feature that enables you to see the

progression of a competitors ads, including which elements were tweaked, and how well the ads were

performing during the time period that they were running.

Such an analysis allows marketers to leverage competing campaigns, and competitors’ tactical tests, as

A/B tests of their own, helping to determine which calls-to-action, offers, product features and

keywords appear to be the most successful within their particular niche.

The Ins and Outs of Paid Search A/B Testing | 4

Analyzing Ad Performance

Before launching any A/B tests, marketers must first determine whether or not they’ve collected enough

data to work with. When launching the tests, use the top performing ad as the control ad when

introducing new ads to test, then before making any changes to the campaign, ensure that there is a

sound statistical difference behind the collected data.

Selecting an Interface

Determine which interface will be used to compare the ad performance. When conducting A/B testing

on Google Adwords, ensure the proper ad rotation settings are utilized. The screenshot below shows

four options to choose from: Optimize for clicks, Optimize for conversions, Rotate Evenly and then

Optimize, and Rotate Indefinitely. This configuration is set at the campaign-level.

Optimize for Clicks

Under this ad rotation setting, Google will automatically show preference to the ad that is expected to

receive the most clicks, based on previous click-through rates. This setting will be ideal for advertisers

who are solely focused on getting click volume and using CTR as their success metric.

Optimize for Conversions

This option permits Google to favor the ad expected to bring most of the conversions to the campaign.

This setting is ideal for advertisers who are testing different landing pages with the same ad copy.

Should there be insufficient conversion data for the ad group, Google will automatically default testing

to optimize for clicks.

The Ins and Outs of Paid Search A/B Testing | 5

Rotate Evenly

This setting will allow both ads to run evenly across a time span of 90 days. If the settings remain

unchanged, Google will automatically optimize the ad serving depending on the bid setting. If the

campaigns are set to Enhanced CPC, Google will optimize for conversions, otherwise, Google will

automatically optimize for clicks.

Rotate Indefinitely

Although Google warns that this setting is “Not recommended for most advertisers”, this enables

advertisers to properly test both ad variations without having to worry about Google automatically

optimizing their ads. Advertisers should use this setting for A/B testing, as they will be able to properly

determine the best performing ad with a high level of confidence.

When advertising on Bing Ads, their ad rotation settings are found on the Ad Group level, rather than

the campaign settings. Currently, advertisers have two choices: Optimize for clicks or Rotate Ads Evenly.

Similar to Google’s settings, the former setting prioritizes the ads expected to bring the most click

volume, while the latter is best used for Optimizing conversions, or for Testing Ads Evenly.

The A/B Testing Process:

To optimize the results of A/B testing, advertisers should ensure that they follow these three simple

steps.

1. Test Only One Element at a Time

When testing ads against each other, it’s necessary that only one variable is changed at a time,

otherwise it becomes extremely difficult to gauge how individual changes impact performance.

Where multiple changes are made simultaneously, it’s not at all easy to pinpoint which component

change contributed to performance variations.

Advertisers can keep busy testing the following variables, followed by some examples:

Headlines

o Dynamic Keyword Insertions vs. Static ads

Body

o Calls-to-action

Buy Now vs. Order Today

o Special offers and discounts

10% Off vs. Free Shipping

o Punctuation/Capitalization

Ad Extensions

o Sitelinks, Location, App, Call Extensions, Consumer Ratings, Reviews, Seller Ratings, and

Social Annotations.

The Ins and Outs of Paid Search A/B Testing | 6

Display URLs

o Note that these do not have to match Destination URLs. Marketers should use them as

an opportunity to add more keywords into their copy.

2. Regularly Analyze Performance

Once enough data has been collected, analyze it to determine whether or not changes should be made

to the general campaign.

Note that when A/B testing, simply optimizing for the highest performing click-through rate, or the

highest conversion rate doesn’t always generate the best results. Instead, advertisers should optimize

ads in order to find the sweet spot that balances out the two metrics to return the best ROI.

Basing decisions solely on optimizing click-through rates may guarantee more traffic, but it doesn’t

necessarily mean that those additional clicks will lead to actual conversions.

Inversely, placing the entire emphasis on conversion rates could reduce inbound traffic by such a

substantial amount that campaign ROI decreases, leading to performance declines.

Attention to detail will be required to ensure a continuing proper balance between click-through rates,

Quality Score, cost-per-click and, most importantly, cost-per-acquisition.

3. Never Stop Testing

Once marketers have determined which ad is performing the best, they should pause the

underperforming version and continue testing by running a new variation against the current optimal

ad.

To maximize campaign performance over an extended period of time, make sure to continue running

A/B tests on a regular basis.

Conclusion

A/B testing is crucial to achieving search engine marketing success. The U.S. Interactive Media Search

Engine Marketing team is well-versed in running rigorous A/B tests, with extensive experience

leveraging this vital strategy to optimize client spends and help reach goals in a more efficient manner.

The Ins and Outs of Paid Search A/B Testing | 7

References Used:

Ad rotation settings (- AdWords Help) https://support.google.com/adwords/answer/112876?hl=en Conversion Rate Optimization in Paid Search: Why Click Through Rate Matters (Search Engine Land) http://searchengineland.com/conversion-rate-optimization-in-paid-search-why-click-through-rate-matters-118991 Bing Adcenter Ad rotation settings http://advertise.bingads.microsoft.com/en-us/help-topic/how-to/51002/ad-rotation-explained