THE POWERTEST™ METHODOLOGYYOUR FASTEST WAY TO FIND A CONTROL

HAL2L.COM • SEATTLE • MUNICH • PRAGUE • SHANGHAI

DON’T GUESS...KNOWDirect marketers agree testing is important—but few test and measure everything. A/B split testing is sometimes done, but it’s seen as expensive and time-consuming. Many marketers prefer to go with their best guess or intuition, or they just do what they did before. Multivariate testing is even rarer in the world of marketing. And yet multivariate testing is recognized as a critical component of continuous learning.

The decision often comes down to time and money. If you don’t have a huge budget and months to learn how to best market your product, what do you do?

The goal of your in-market testing is to learn which combination of audience targeting, offer and creative will work best.

The PowerTest™ methodology from HackerAgency gives you the learning you need—and much faster than traditional multivariate testing because you can test more variables at once for nearly the same budget. It makes sense if you’re:

•! Starting a new business and need to get to market without making expensive mistakes

•! Launching a new product

•! Seeing fatigue in your current marketing methods

•! Searching for a new marketing channel to test

•! Wanting to jumpstart growth for your business

The PowerTest methodology lets you learn what works in real-life marketing situations—in a single media channel or using multiple marketing media. By using PowerTest methods to design testing matrices and capture data, it’s ultimately possible to build a customized, enterprise-level sales attribution model that accurately tells you what works, what to tweak and where to spend your next marketing dollar most effectively.

It’s also possible to test in digital media first—to learn about targeting and offers quickly and less expensively, and then move into more scalable channels, such as DRTV and direct mail.

CONTROL+FLEXIBILITY

Using a PowerTest approach, you get the right mix of control and flexibility. You aren’t guessing what will work—instead, you let the market dictate your strategy. And you do it so nimbly that you can read results and launch in days or weeks rather than months.

The objective of a PowerTest is to develop a champion/challenger strategy as fast as possible. With a properly run PowerTest, you can usually find a winning approach—or several winners—in the first test, and almost always by the second test event.

HOW TO USE A POWERTEST STRATEGY

LAUNCHING YOUR POWERTEST

PHASE 1

The objective of Phase 1 is to determine these key success drivers:

•! Discover which offer will work for which audience segment

•! Determine which creative platforms will work best

Looking at a simple 120,000-person test, the key assumptions are:

•! We will test 2 creative concepts

•! We will test 2 different message platforms

•! We will test 3 offers

•! We will test in 10 media—for example: 5 email lists, 2 online display outlets and 3 direct mail lists

Based on the above test assumptions, the test matrix would look like the grid in Figure 1.

Figure  1.

READING PERFORMANCE

In our sample matrix, each of the 120 test cells has 1,000 names. This means that cell-level results are not likely to be statistically valid. To generate a statistically valid sample at the cell level, this test would have to reach about 600,000 people, which is often too risky, too expensive or both.

The results are still highly indicative and directionally trustworthy. And we can very likely get statistically significant results when we look at a summarized view. You will see strong trends and HotZones in your matrix that tell you what to do next. Let’s assume that the test generated the the response rates of Figure 2.

Figure  2.

In this example, it is safe to conclude that:

•! Concept 2 is a clear winner over Concept 1.

•! Within Concept 2 and Platform B, Offer 2 worked with Media A through E.

•! Offer 3 worked for Media G through J within Concept 2 and Platform B, yielding a statistically significant result at the 95% confidence interval.

VALIDATING YOUR POWERTEST

Figure  3.

PHASE 2

We conduct a CheckTest™ in Phase 2 using the results of Phase 1, this time with cells that are large enough to generate a statistically valid result (Figure 3).

With similar response rates, the results from this CheckTest would be statistically projectable. And we’ve been able to find a winning control with only 240,000 names instead of the 1,200,000 that would have been required for a test of this magnitude with conventional testing.

Now that you have a control (or champion), you can keep program performance high with a standard test methodology and conduct a PowerTest strategy again only if you need to. We recommend testing every time you run a marketing campaign. If possible, test in 10% to 20% of the universe. That way you can learn each time you go out to your audience, while still maintaining results through using your control strategy with the remaining 80% to 90%.

TAKING THE NEXT STEPThe PowerTest is often the initial phase of a more holistic, strategic marketing program. It’s important that all data is collected—preferably in a sophisticated Analytics Environment (see the HackerAgency white paper for details). As more media channels are leveraged, an enterprise-level sales attribution model helps you continually optimize your marketing budget. A more detailed white paper on Multi-Channel Sales Attribution is also available.

While the theory behind the PowerTest methodology is simple, execution can be complex. If you’d like assistance in planning and executing a PowerTest, contact Kristin Flor at 206.805.1500 or kflor@HAL2L.com.

At HackerAgency, we are obsessed with results. We’re a global direct marketing agency with expertise across all media channels and a focus on the metrics that matter. Our success generating ROI for our clients makes us the leader in the world of CRM. From Leads to Loyalty, no agency on earth develops and deploys data-driven strategies and creative that engages and compels customers to take action the way HackerAgency does. We don’t just move the needle—we bury it.