sotech.com | 800-576-2728 1sotech.com | 800-576-2728

A Marketing Research Consultancy

ADVERTISING & COMMUNICATIONS ONLINE RESEARCH

Methods and Options for Testing Marketing Communications in the Online Environment

Pre-Flight Concept Testing Post-Flight Effectiveness Measurement

Socratic Technologies

has been conducting

online market research

since 1994, almost as

long as there has been

a World Wide Web.

sotech.com | 800-576-2728 2

ADVERTISING & COMMUNICATIONS ONLINE RESEARCH

Advertising and Communications TestingUsing the Web to test advertising com-munications and creative executions is a well-established and thoroughly tested methodology. With close to 85% of the U.S. population now having Internet access, the ability to find representative samples for most populations is no longer a barrier to the use of the medium for serious, professional research.

Since 1998 Socratic Technologies has been at the forefront of researching and refining traditional methods for testing communications and messaging ele-ments and then adapting them for use in an online environment.

Many of these methods—including techniques for measuring relevance, believability, memorability and the com-munication of positive attributes related to purchase intent—have previously been possible only through the use of a human interviewer. Socratic’s approach to adapt-ing these traditional techniques, however, has concentrated on creating self-admin-istered test procedures for remote data collection via the Internet.

In addition to testing advertising ef-fectiveness in offline media, Socratic’s approach also works very well for eval-uating in-market online advertising. Our approach provides advertisers with critical information about the performance of their online advertising on three import-ant dimensions, while avoiding intrusive, annoying and ineffective pop-ups. Our approach is also Web site friendly, in that we require no programming changes on

the site and no ad inventory to accommo-date the survey.

This white paper will present methods and technologies for both pre-flight advertising concept evaluations and post-flight effec-tiveness measurements.

BenefitsSuperior Value

Advertising research is usually much less expensive when conducted via the Internet than in person—with savings of as much as 60% of a typical prerecruited, in-person style interview. In addition, because ex-hibits are typically in digital format (versus printed ads) the cost of producing and distributing the graphic stimuli is far less than physical exhibits.

Speed

Compared to traditional research, time- to-decision is radically reduced when using online testing. Much of the exhibit presentation programming for Socratic’s Internet graphic research tools is com-plete and has been done in such a way that new graphic elements can be added and adapted without large amounts of time-consuming custom programming. Fielding can be completed in days versus weeks for traditional data collection processes, particularly if done with one of Socratic’s prerecruited panel options.

Data Quality

Using Socratic’s online communication research tools produces a high level of data quality based on several factors. First, the computer-mediated, self- administered mode reduces interviewer bias and errors related to incorrect exhibit rotations and complex skip-logic

The Socratic

ClutterBookSM

is a virtual

magazine that allows

clients to conduct

traditional print ad

recall tests without the

time and expense of

sending magazines to

respondents.

sotech.com | 800-576-2728 3

ADVERTISING & COMMUNICATIONS ONLINE RESEARCH

by executing the research precisely and consistently every time. Second, the online environment produces more candid responses to design options by providing a more anonymous, unhurried interviewing environment. Third, unlike physical exhibits, digital exhibits do not degrade or show signs of wear that can adversely affect the responses after being handled during multiple interviews. Finally, the depth, richness and quantity of open-ended responses captured online are usually superior to other traditional forms of data collection.

Pre-Flight Concept TestingTest Objectives and Online Test ProceduresTo be effective, an advertisement must perform in three areas: impact, message communication, and brand attribute communication.

Each of these factors must interact to create a total impression that supports the desired goal of increased sales. Each of these elements must be satisfactory in order for the ad to be a “success.”

Taking any one element by itself as the only decision criterion can lead to poor results. For example, if you consider only impact, you may find that a garish and colorful ad may jump off the page yet the opinion of the product advertised may suffer, or the message about what the product is may be lost. You may be able to easily notice an ad that features fluo-rescent colors in a moiré pattern, but any communication in the ad will be likely lost amid the confusion of design choices.

Concept ImpactAd impact testing is composed of two measures: recognition and recall. Rec-ognition measures the extent to which the brand “pops” from the background clutter and is noticed at all. Our recog-nition tests measure whether the brand in the ad is recognized by respondents and whether that recognition registers positively in the respondent’s mind.

The time-tested method of evaluating print advertising recognition and recall is a “clutter book,” where the test ad is presented amid competitive advertising and background clutter. Socratic has taken this rather cumbersome, traditional test procedure and produced an easy-to-use online version called the Socratic ClutterBookSM.

Socratic ClutterBookDescription

Similar to techniques used for copy recognition and memorability testing, the Socratic ClutterBook technique presents advertisements, themes and other forms of visual stimuli within the context of com-peting “clutter.”

Shown “pages” of content, the respondent is tasked with performing a timed skim-through of the material and then must re-port memorable ads, themes, messages, etc. Respondents can page through the clutter book at their leisure, or the length of exposure for each page can be controlled.

Characteristics

The Socratic ClutterBook helps normalize time bias in traditional techniques by tracking view time by area of content. Unaided and aided recall, copy recall,

sotech.com | 800-576-2728 4

ADVERTISING & COMMUNICATIONS ONLINE RESEARCH

key message meaning, persuasiveness and other similar measurements can be collected. “Skim-through” interest can be measured using “click tracking,” wherein the respondent is asked to click on anything of interest during the skim-through process. Reconstruc-tion techniques can be used to aid the respondent in “re-building” the recalled test stimulus from alternative component elements (e.g., logo, background color, product illustration, copy points, etc.).

Output

Total awareness of test stimuli is reported as the proportion of unaided + aided awareness. Recall is usually reported as the proportion of the sample that correctly recalled the key message components and

brand. The degree to which the test stim-ulus was judged as “interesting” during the skim process is stated as the percent-age of subjects who clicked the item.

Results from these tests focus on general recall and specific messaging recall. The technology can also be used to correct for differences in time and duration of examination between respondents. As various text and advertising elements are displayed, participants also have the opportunity to click on various elements for either expanded views or to explore something of particular interest.

In addition, this technique can be used to test for confusion and or correct recollection of specific messages being communi-cated. This is of particular interest when

Figure 1: This tool presents ad stimuli against competing visuals.

The Socratic

PerceptometerSM

allows

clients to move video

ad tests from costly

central locations to

respondents’ living

rooms, resulting in more

diverse samples, lower

study costs and faster

results.

Figure 2: This tool measures the moment-to-moment impact of video ads.

sotech.com | 800-576-2728 5

ADVERTISING & COMMUNICATIONS ONLINE RESEARCH

targeted messages are being produced for sophisticated or narrowly defined market niches.

Moment-to-Moment Impact

Measuring moment-to-moment impact of video ads, whether TV, online or on DVD, can be done by measuring viewers’ spon-taneous reaction to the ad and testing aided and unaided recall of key message points and/or imagery. The reaction to video traditionally has been done using audience meters in large central testing environments. Socratic has developed an online perception meter called the Perceptometer. This flash-based online

tool allows for perfect synchronization of the respondent’s reaction with the video image, and permits this type of research to be conducted using large national sam-ples rather than central location testing.

Socratic PerceptometerSimilar to a traditional “audience meter,” the Perceptometer uses a streaming video feed to present advertisements for con- tinuous (second-to-second) measurement. The Perceptometer’s use of streaming video adds control and security to the presentation of prototype ads and audio- video elements. The technique captures:

• Continuous approval or interest ratings

sotech.com | 800-576-2728 6

ADVERTISING & COMMUNICATIONS ONLINE RESEARCH

• Overall perception

• Overall likelihood-of-action measure-ments (e.g., purchase, recommend, etc.)

• Memorability ratings of key themes and/or elements

• Real-time analysis of respondent-level data (e.g., most effective, least effective elements)

• Comparability to established norms for ratings

Output

Perceptometer output consists of a storyboard output of moment-to-moment measurements as well as overall success ratings for the stimuli. Clients can quickly assess which communications elements

were the most/least highly rated. Follow- up profiling questions ascertain why these ratings were given and measure recall of communication elements. Subgroup analysis can also be performed. Side-by-side testing of competing or alternative versions can be compared and contrast-ed. Where appropriate, Perceptometer rating norms can be used to judge the rel-ative effectiveness of messages, elements and overall performance. The meter can also be used without the feedback control as a video clip demonstration tool.

A demo of this technique can be found at: http://www.sotech.com/site/details.asp? tn=Tools&tid=19

Figure 3: Perceptometer Reporting Output

The Socratic

CommuniScoreSM

normative database

allows clients to

understand how their

ads compare to others

using the same media

in similar markets

or among various

audiences.

Figure 4: CommuniScore concept performance compared to normative database

sotech.com | 800-576-2728 7

ADVERTISING & COMMUNICATIONS ONLINE RESEARCH

The Socratic CommuniScoreSM

If an ad is successful in breaking through the clutter and catching the target’s attention, only half of the battle is won. If the message is not clear, relevant and congruent with the desired brand perception, that hard-won attention will be quickly lost. Socratic has developed a system of communication metrics, called the Socratic CommuniScore, which gauges the effectiveness of a message in six key areas:

• Overall opinion

• Relevance

• Believability

• Clarity

• Fit with brand

• Likelihood to take action

Overall Opinion

This metric measures a respondent’s general reaction to a message. This is not a likability question: it measures the “gut” reaction to the overall message and captures any issues with a message that

are otherwise difficult to categorize. This is a classic “disaster check” question.

Relevance

This metric quantifies the salience of the message to the respondent. This is a critical measure because any communi-cation that is deemed not relevant will be quickly dismissed and forgotten.

Believability

This metric quantifies the believability of the claims made in the message. If a mes- sage is not believed, it will be ineffective at best, and damaging to the brand at worst.

Clarity

This metric captures the understand-ability of the message. It is critical that messages be clear and understandable, as confusion in the mind of the target greatly reduces the effectiveness of any message.

Fit with Brand

This metric quantifies the congruence of the message with the target’s perception of the brand. A message that is not

The ads are slightly below average on purchase likelihood and believability. While relevance is very high relative to normative measures, the ads suffer from a lack of uniqueness.

The ads are slightly below average on purchase likelihood and believability. While relevance is very high relative to normative measures, the ads suffer from a lack of uniqueness.

Top 2 Box Index

0 100 200

“Phabulous”(n=301)

(a)

Purchase Likelihood

Relevance

Uniqueness

Believability

88.6000

162.5000

53.2000

78.3000

82.9000

140.6000

63.8000

84.8000

91.4000

175.0000

63.8000

87.0000

“Self-Portrait”(n=304)

(b)

“Running On Empty”(n=303)

(c)

Concept Performance: Normative Comparison

Reached and Affected

0 100 200 0 100 200

Figure 5: Detailed ad performance insights and useful prescriptives

sotech.com | 800-576-2728 8

ADVERTISING & COMMUNICATIONS ONLINE RESEARCH

consistent in tone and content with the target’s existing perception of the brand will cause confusion, which may force a change in the brand perception. This may or may not be a desired result of communication.

Likelihood to Take Action

This metric captures the respondent’s likelihood to take action based on the message. This measure is important be-cause it captures the extent to which the message moves the target to action.

The Socratic CommuniScoreSM presents a composite score, as well as individual

component scores. The individual scores are important because the objectives of any particular communication will vary, and the relative importance of each of the individual metrics will vary along with them. The composite score weights each individual metric equally.

This communication test can be applied to a variety of marketing messages, such as:

• Finished ads

• Ad concepts

• Taglines

• Positioning statements

The Bruzzone ModelSM

Example of Results

AmusingClever

ImaginativeOriginal

Fast MovingLively

Not Worn OutNot Dull

WarmSensitiveGentle

AppealingWell Done

FamiliarSeen Alot

ConvincingEffective

BelievableTrue To Life

Worth RememberingNot Easy to Forget

Not ConfusingNot PointlessInformative

Not IrritatingNot PhonyNot Silly

Humor

Uniqueness

Energetic

Freshness

Warmth

Appeal

Familiarity

Persuasive

Credible

Importance

Clarity

Genuine

Entertainment

Appropriateness

Empathy

How It Made You Feel About the Product

Relevance

Remembered Advertiser Name

Likeability

Attitude Toward Ad

Increased Awareness

Increased Buying Interest

Interest in Message

Attitude TowardProduct

Overall Impact

ATTRIBUTES THEMES INDICES & INDICATORS PERFORMANCE IMPACTAdvertisement Testing with51 Diagnostics

Better Than Average

Average

Below Average

sotech.com | 800-576-2728 9

ADVERTISING & COMMUNICATIONS ONLINE RESEARCH

Post-Flight Recognition and Brand Attribute CommunicationThe Bruzzone Model

SM

Starting in the early 1990s, Bruzzone Research in Alameda, CA, was well known for its advertising recognition and effectiveness testing model that evaluated how an advertisement performed on 51 diagnostic measures. For more than two decades, Bruzzone amassed a norma-tive database of thousands of ad tests showing diagnostic scores such as: how effective an ad was in breaking through cognitive clutter; whether it aided in the recollection of the name of the advertis-er; and whether the ad was successful in in creasing buying interest. Socratic now uses The Bruzzone Model and its measurement database for post-flight effectiveness testing.

The key advantage of the recognition approach used in The Bruzzone Model to testing advertising is that it is based on people’s ability to recognize things they have seen before, something they can do easily and with great accuracy. To track the impact of past advertising we begin by showing a variety of advertising to re-spondents and ask which they remember seeing before. In pretesting advertising we show the advertising embedded in the Socratic CutterBook or Perceptometer, wait several days, and repeat the same process.

Recognition is the best way to determine attention-getting value. We use this proce-dure to accomplish six key test objectives:

Intrusiveness The greatest danger any advertising ever faces is being ignored. You need the best possible measure of intrusiveness to see if the person even noticed the advertis-ing—and whether the process of brand recognition had a chance to get started.

Accuracy Recognition capitalizes on one of the great strengths of the human mind. This is documented by psychologists’ photo-graph sorting experiments, where people are given a stack of several hundred photos to look at. Then several hundred additional photos are mixed in and re-spondents are asked to sort out the ones they saw the first time. People of normal intelligence can routinely identify the ones they saw before with 95–98% accuracy.

SensitivityRecognition scores are more likely to show statistically significant differences because they spread out over a wider range. These scores distribute and do not aggregate at the low end of a range, as is the case with recall.

ReliabilityRecognition provides high test-retest reliability. You get the same results every time. Unlike recall, recognition persists for a long time.

ReproducibilityRecognition tests don’t break down in the field. They don’t produce wildly different results with different interviewers and coders, and with different methods of showing the advertising.

Advertising messages

play a central role in

forming the market’s

perception of a brand.

It is therefore critical

to understand how

those messages are

perceived. The Bruzzone

ModelSM

normative

database allows clients

to understand how their

ads compare to others

using the same media in

similar markets or among

various audiences.

sotech.com | 800-576-2728 10

ADVERTISING & COMMUNICATIONS ONLINE RESEARCH

Completeness

Recognition identifies two to three times as many people who have actually been exposed to an advertisement as a compa-rable test based on simple recall.

Masked AdvertisingAn accurate segregation of the sample into groups who noticed the advertising and those who didn’t is critical for all the remaining steps in testing the impact of advertising. The first is measuring the ability of the ad or commercial to get the name across.

In conducting recognition tests we block out or bleep all mentions of the name of the product so we can see how many remember who the advertiser was. This is the most vital piece of information communicated by any advertisement. Ads and commercials vary widely in their ability to get across the name of the advertiser.

Both attention-getting value and the ability to get the name across are well measured in recognition-based testing. They are among the most difficult char-acteristics to get a satisfying measure of when other techniques are used.

This communication test can best be applied to finished ads (for example, TV, print, online, billboards, radio, kiosks) after people have had a chance to be exposed to the advertising.

Theory The Bruzzone Model has a high level of face-validity. It shows advertising the way most people understand it. But it also has a solid foundation in theory and basic re-search. The split between emotional and rational reactions is based on the work of Petty and Cacioppo (1986) on central and peripheral processing. The way the basic factors feed into attitudes toward the ad and attitudes toward the product are based on the causal modeling of Lutz (1985; 2004) and his colleagues. The adjectives were drawn originally from the work of Wells, Levitt and McConville (1971) and enlarged in collaboration with Alex Biel and David Aaker (1985; 2002). The clusters are based on years of factor analysis by Clark Levitt (1976).

sotech.com | 800-576-2728 11

CONTACT

Socratic Technologies, Incorporated, is a leader in the science of computer-based and interactive research methods. Founded in 1994 and headquartered in San Francisco, it is a research-based consultancy that builds proprietary, interactive tools that accelerate and improve research methods for the study of global markets. Socratic Technologies specializes in product development, brand articulation, and advertising research for the business-to-business and consumer products sectors.

Registered Trademarks, Salesmarks and Copyrights The following product and service descriptors are protected and all rights are reserved. Brand Power RatingTM, BPRTM, Brand Power IndexTM, CATM, Configurator AnalysisTM, Customer Risk Quadrant AnalysisTM, NCURATM, ReportSafeTM, reSearch EngineTM, SABRTM, Site-Within-SurveyTM, Socratic CollageBuilderSM, Socratic ClutterBookSM, Socratic BrowserSM, Socratic BlurMeterSM, Socratic CardSortSM, Socratic ColorModelerSM, Socratic CommuniScoreTM, Socratic Forum®, Socratic CopyMarkupSM, Socratic Te-ScopeSM, Socratic PerceptometerSM, Socratic Usability LabSM, The Bruzzone ModelTM, Socratic ProductExhibitorSM, Socratic Concept HighlighterSM, Socratic Site DiagnosticSM, Socratic VirtualMagazineSM, Socratic VisualDifferentiatorSM, Socratic Web BoardsSM, Socratic Web SurveySM 2.0, Socratic WebComm ToolsetSM, SSDSM, Socratic WebPanel ToolsetSM, SWSSM 2.0, Socratic Commitment AnalysisTM, Socratic WebConnectSM, Socratic Advocacy Driver AnalysisTM.

Socratic Technologies, Inc. © 1994–2014. Reproduction in whole or part without written permission is prohibited. Federal law provides severe civil and criminal penalties for unauthorized duplication or use of this material in physical or digital forms, including for internal use. ISSN 1084-2624.

sotech.com | 800-576-2728

San Francisco HeadquartersSocratic Technologies, Inc. 2505 Mariposa Street San Francisco, CA 94110-1424 T 415-430-2200 (800-5-SOCRATIC)

Chicago Regional OfficeSocratic Technologies, Inc. 211 West Wacker Drive, Suite 1500 Chicago, IL 60606-1217 T 312-727-0200 (800-5-SOCRATIC)

Contact Ussotech.com/contact