Judging Advertising Creativity Using the Creative Product Semantic Scale

241

ABSTRACT

INTRODUCTION

A L I S A W H I T EF U Y U A N S H E N

B R U C E L . S M I T H

Judging Advertising CreativityUsing the Creative ProductSemantic Scale

The Creative Product Semantic Scale has been used to judgetangible creative products such as crafts, and intangibles suchas the ideas produced by problem solving groups. In this studyit was used to judge the creativity of 15 print advertisements,using samples representing college students, advertising pro-fessionals, and the general public. The study explored the dif-ferences and similarities of responses. There were no significantdifferences in the judgments of the three groups regarding theoriginality and logic of the ads. With regard to how well craftedor well executed the ads were, however, the judgments ofadvertising professionals differed from those of students andthe public.

Much of the research about advertising creativity has focusedon it from the perspective of the advertising professional.Fletcher (1990) said that really creative people have the abil-ity to judge their own ideas. Their subjective experience be-comes the measure, and no formal instrument need be used.Commonly, advertising professionals do conduct research toobjectively test whether advertisements are interesting, likable,understandable, and believable (Flandin, Martin, & Simkin,1992). Both informal and formal measures of advertisingassume that a creative advertisement will “work” (p. 207),meaning it will drive sales or accomplish another explicit com-mercial objective.

Amabile (1982) asserted that an advertising product is cre-ative to the “extent that appropriate observers independentlyagree it is creative” (p. 1001). Typically, “appropriate” observ-ers are advertising creative personnel such as copywriters and

Volume 36 Number 4 Fourth Quarter 2002

242

Judging Advertising Creativity

art directors, who produce advertisements and vote to bestowindustry award recognition.

This study sought to measure experimentally how advertis-ing creativity is judged, and whether those judgments differbetween groups. The researchers also sought to ascertainwhether other factors such as age, gender, education, race,and region where one lives significantly affect judgments aboutthe creativity of advertising.

No effort was made to measure the effectiveness of ads fromthe point of view of increased sales, recall, changed attitudes,or brand awareness. Those have been the traditional goals ofadvertising research. Here the emphasis was on how differentpeople experience advertisements and translate that experi-ence into quantitative measures of creativity. Advertisementsserve as examples of intangible creative products.

Countless measures of advertising effectiveness have beendeveloped over the years. Schlinger (1969) used Q-techniqueto elicit judgments about marketing messages. By asking sub-jects to sort marketing phrases or themes and then correlatingand factor analyzing the results she sought to ascertain whichthemes resonated with particular groups of consumers. Wells,Leavitt, and McConville (1971) created a “reaction profile” ofwords that consumers use to describe TV commercials, whichthe researchers later grouped together to represent a morelimited number of dimensions or ideas. McEwen and Leavitt(1976) examined the content of TV commercials themselvesand analyzed the interaction and interplay among elements inthe commercials.

Kover et al (1995) studied creativity and advertising effec-tiveness and tried to ascertain how consumers react to differ-ent kinds of advertising executions and whether they respondto ads as professionals expect them to. They found thatads that were perceived as unexpected by viewers were notnecessarily perceived as creative or persuasive.

In a subsequent study Kover et al (1997) compared adver-tising research over time and concluded that advertisingcreatives respond to and create advertising according to pro-fessional standards while viewers respond to advertising thatengenders feelings of personal enhancement. They point outthe negative consequences of what they call disjunction be-tween responses of advertising professionals and viewers toadvertising content, including the possibility that viewers willnot spend the time necessary to correctly interpret advertising

LITERATURE REVIEW

243

Journal of Creative Behavior

messages and that advertising creatives fail to “connect in theirwork with the people who eventually will see their work” (p. 42).

Subjects in Altsech’s (1996) quasi-experimental study ofadvertising creativity rated advertisements on 65 items relatedto originality, appropriateness or relevance, liking, and excite-ment or boredom (p. 29). He concluded that creativity andoriginality are synonymous, but that appropriateness is a“qualifier” for creativity (p. 42). That is, assessment of origi-nality and creativity may be discounted if an advertisementdoes not fit the product or audience.

The current study applies “experimental aesthetics,” whichBerlyne (1974) called the study of aesthetic problems throughexperiments that use content analysis to measure characteris-tics of artistic and other artifacts. Berlyne used semantic dif-ferential scales (Osgood, Suci, & Tannenbaum, 1957) tomeasure aesthetic judgments using pairs of bipolar adjectivesthat represent dimensions such as familiar-novel, simple-com-plex, and expected-surprising.

In this study the Creative Product Semantic Differential Scale(CPSS) was used to assess creativity. Like the instruments usedby Altsech and Berlyne, the CPSS uses semantic differentialscales to measure creativity. The bipolar adjectives, in fact,closely resemble and evolved from those used in earlier stud-ies. It was developed to facilitate the systematic analysis of cre-ative products (Besemer & O’Quin, 1986; O’Quin & Besemer,1989). They used it to measure a variety of everyday productssuch as T-shirts, a ceramic pitcher, and furniture. Subsequentresearchers have used the scale to judge intangible creativeproducts. Russell (1991) used it to judge creative problem solv-ing by elementary school students. Smith (1993) measuredthe creativity of ideas produced by adult problem solvinggroups.

The CPSS is similar in purpose to the instrument used byAltsech (1996) to measure advertising creativity. In additionto measuring the dimensions of originality and appropriate-ness, however, the CPSS also attempts to measure how wellcrafted or well executed an advertisement is. Measuring ex-ecution seemed critical to comparisons of creativity when us-ing samples that differ significantly in their training andexperience with advertising.

The samples were drawn from three populations in SouthDakota and Georgia: 43 advertising agency professionalsfrom Sioux Falls and Atlanta, 189 undergraduates from

METHOD

Subjects

244


introductory mass communication survey courses at theUniversity of South Dakota and the State University of WestGeorgia, and 61 individuals from both states who were selectedto represent the broad demographics of the general public.

Twenty-one percent of advertising professionals identifiedthemselves as account executives, 60% work in creative ser-vices, 12% in agency management, and the rest in other func-tions. The mean age of professionals was 32.5 years, 52% werefemale and 48% male, 95% were Caucasian and 5% AfricanAmerican. Sixty-nine percent of the professionals hadbachelor’s degrees, 5% had completed some graduatecoursework, and 5% had completed a graduate degree. Therest had completed some college coursework.

Twenty-eight percent of the college students were commu-nication majors. The rest were majoring in a variety of disci-plines. None of the students had yet taken an advertisingcourse. Subjects were demographically diverse. The averageage was 20.5 years, 51% were female and 49% male, 88% wereCaucasian, 6% African American and 6% identified themselvesas Asian, Hispanic, or multi-racial.

Sixty-one percent of the sample representing the public wasfemale and 39 percent were male. Their ages ranged from 21to 75, with an average age of 45. Ninety percent were Cauca-sian, 5% were African American, and five percent were Asianor Native American.

Subjects were given two booklets. One included the subjectconsent form, instructions, and assessment forms. The otherwas labeled “Advertisements” and contained high quality colorphotocopies of 15 ads that were selected from the following 12consumer magazines that appeal to various demographics:Travel America (June 1997), Redbook (July 1997), CookingLight (July 1997), Today’s Homeowner (May 1997), FamilyPC (Jul/Aug 1997), Hunting (May 1997), Ladies HomeJournal (July 1997), Home (July 1997), Better Homes &Gardens (May 1997), Weight Watchers (Mar/Apr 1997),People (July 14, 1997), and Time (July 14, 1997).

Each issue was searched for full-page general-interest ad-vertisements. Ads were sought that were not age or genderspecific. Ads that promoted a variety of products, representeda variety of styles, and used a variety of colors, fonts, andappeals were identified. Approximately 35 ads fitting thatdescription were chosen, and the list was narrowed to includea range of ads from various categories (three food, three bev-erage, three health/personal care, two garden, two pet, and

Procedures

245


two automobile). One ad had to be discarded and replacedwith a substitute because it could not be adequately reproducedusing color-photocopying technology.

Subjects were asked to evaluate each advertisement usingthe 15-item Creative Product Semantic Scale, and to providedemographic information for analysis purposes.

The Creative Product Semantic Differential Scale (CPSS)was used to assess creativity. Subjects rated the advertisementsby circling a number ranging from 1 to 7 that best describedthe ads in terms of bi-polar adjectives. For example, an ad wasjudged as to whether or not it was “appropriate” or “inappro-priate.” A rating of “4,” halfway between 1 and 7, would indi-cate a neutral response. A rating of “1,” closest to the“inappropriate” side of the attribute, would indicate the stron-gest association with the negative aspect of that attribute. Arating of “6,” close to the “appropriate” side of the attribute,would indicate a strong association with the positive aspect ofthe attribute.

Each ad was printed on a separate sheet and clearly linkedwith a particular evaluation sheet that was used to judge it.Ads were printed in random order. The order of the CPSS scaleswas also random, using computer-generated random order-ing. Eleven of the 15 evaluation scales were presented to sub-jects in positive to negative order. The other four were presentedin negative to positive order to discourage evaluators fromgoing down the list of items and marking all items with onerating. Before statistical analysis was conducted, adjective pairswere ordered negative to positive, and corresponding numberstransformed in the order of 1 to 7.

The complete CPSS uses 55 items on a 7-point scale. In thisstudy, three of the eleven subscales were used, representing atotal of 15 items. The three subscales chosen — originality, logic,and well craftedness — were selected because they were judgedappropriate for the creative product and were variations onsubscales used previously by other advertising researchers.All three dimensions of creativity in the CPSS scale were rep-resented. Karen O’Quin, one of the originators of the instru-ment, recommended using an abridged version of the CPSS.The longer instrument, she said, was very fatiguing to evalua-tors and yielded little improvement in results over using ashorter version (personal communication, February 24, 1992).In addition, she said, not all subscales are applicable to all cre-ative products.

Mean scores were calculated for each item and repeated-

Measurement

246


measure multivariate analysis of variance tests were used todetermine judgment differences between the public, studentsand advertising professionals. In addition, the 15 items werereduced to three constructs, termed subscales by the origina-tors of the instrument (original, logical, and well crafted).Repeated-measure of analysis of variance tests were conductedon the mean scores of both the individual adjective pairs andthe subscales to determine differences between students andadvertising professionals.

The experimental treatment, CPSS instrument, and instruc-tions to the subjects were pretested on a group of 14 universitystudents who evaluated them for clarity of instructions. Slightmodifications were made to the questionnaire before proceed-ing with the study.

CPSS Dimensions, Subscales, and Items Used.

Dimension Subscale Items in Subscale

Novelty Original over used-freshpredictable-novelusual-unusualunique-ordinaryoriginal-conventional

Resolution Logical illogical-logicalmakes sense-senselessirrelevant-relevantappropriate-inappropriateadequate-inadequate

Elaboration and Well crafted skillful-bunglingSynthesis well made-botched

crude-well craftedmeticulous-sloppycareless-careful

The three samples made different judgments about which adsthey ranked highest and lowest. As Table 2 shows, profession-als gave the Absolut Vodka ad the highest rating, while stu-dents gave the Minute Maid Orange Juice ad the highest rating,and the general public ranked the General Foods InternationalCoffee ad the highest. They also disagreed about which ad to

ANALYSIS ANDRESULTS

Evaluations byprofessionals,

students, and thegeneral public

TABLE 1.

247


give the lowest ranking to. For the professionals, the lowestranking ad was for DuPont Comforel pillows. Students and thegeneral public least favored the Ro-Tel Diced Tomato ad. Ingeneral, however, the three groups came to similar judgmentsabout ads. For example, the Purina One dog food ad wasranked eighth by all three groups. Most rankings were only abit different. A chi-square test of the rankings confirmed thatthere were no significant differences (df = 28, p = .811).

Overall Rankings of Ads by Advertising Professionals, Students,and the General Public.

Advertisement Prof. Prof. Student Student Public PublicDescription Rank Mean Rank Mean Rank Mean

Hershey’sSyrup 6 4.897 5 5.0276 7 4.889

Kibbles ‘nBits 10 4.309 11 4.5433 9 4.669

Purina OneDog Food 8 4.691 8 4.8011 8 4.734

PlymouthBreeze 5 5.129 4 5.0922 3 5.121

SubaruOutback 12 4.13 10 4.5487 11 4.494

OrthoWeed-B-Gon 9 4.521 9 4.5615 14 4.216

Off CitronellaCandles 2 5.343 3 5.1203 5 4.957

NeutrogenaSunblock 13 4.044 12 4.4226 10 4.521

ComforelPillows 15 3.549 13 4.251 13 4.273

AtroventNasal Spray 7 4.722 2 5.2539 4 5.012

Ro-TelTomatoes 14 3.576 15 4.2007 15 4.153

MinuteMaid Juice 4 5.173 1 5.3849 2 5.141

InternationalCoffees 3 5.205 6 4.9409 1 5.184

Gardenburger 11 4.132 14 4.2377 12 4.307Absolut Vodka 1 5.513 7 4.8925 6 4.955

N = 189 N = 43 N = 61

TABLE 2.

248


A repeated-measure MANOVA using the total CPSS ratingas the dependent variable was then conducted to furtherexplore how different groups evaluated advertising creativity.Individual ad (Ad) was specified as the within-subject factor.The three groups (Group), students, professionals and thegeneral public, were specified as the between-subject factor.Results showed a significant Ad x Group interaction effect(F = 1.85, p < .001), along with a significant within-subject maineffect (F = 20.66, p < .001).

In light of these preliminary findings, we conducted a seri-ous of repeated-measure ANOVA of each CPSS subscale withAd as the within-subject factor and Group as the between-subject factor. Results of these subsequent analyses indicatedthat group differences in evaluating advertising creativity werenot significant differences for all three subscales. Judgmentsof the ads using the original subscale, which measured theattributes original-conventional, novel-predictable, unusual-usual, unique-ordinary, and fresh-over used, were not signifi-cantly different (F = 1.39, p > .05). The logical subscale, whichmeasures the attributes appropriate-inappropriate, logical-illogical, makes sense-senseless, relevant-irrelevant, andadequate-inadequate, was not significant between the threesamples (F = 1.00, p > .05). However, there were significantgroup differences on the well-crafted subscale, which measuredthe execution of the ads with such attributes as well crafted-crude, meticulous-sloppy, skillful-bungling, well made-botched,and careful-careless (F = 2.95, p <.05).

Post hoc pairwise comparisons showed professionals’ aver-age evaluations on the well-crafted dimension was significantlydifferent from students (4.49 vs. 4.74, p < .05 ) and the public(4.49 v. 4.73, p = .05). These results mean that professionals,students and the public agreed on the originality and logic ofads, but not about how well-crafted they were.

Stepwise regression analysis was used to identify the princi-pal components that contributed to the overall and subscaleratings. With regard to the Total CPSS ratings and the ratingsfor the well-crafted subscale, where analysis of variance wassignificantly different, the principal components were experi-ence in advertising, followed by type of sample (professionals,students, public), followed by region of the country the personis from, age, and gender.

For the originality and logic subscales, however, the predic-tors were somewhat different. The most significant predictorfor the originality subscale was the ad number — or which ad

249


was being evaluated. Other predictors in order of consequencewere gender, rural versus urban, experience in advertising, typeof sample. For the logic subscale, experience in advertisinghad the highest loading as a predictor, followed by ad number,and then demographic variables.

When it came to the originality and logic subsets, the CPSSratings were more affected by which ad was being evaluated,and by the personal characteristics of the reviewer. For the TotalCPSS score and the well-crafted subscale, the type of samplewas most predictive, followed by the characteristics of thereviewer. The ad number was not a significant predictor. In otherwords, who is doing the judging matters most when it comesto how well crafted the advertisements were; whereas, the spe-cific characteristics of the ads themselves were most impor-tant when it came to judging originality and logic.

A chi-square test indicated that advertising professionalswere consistent in their rankings of the three subscales (df =28, p > .05). They tended to give any given advertisement ahigh, medium, or low ranking for all three subscales (see Tables3, 4, 5). The student and public samples were less consistent(df = 28, p < .01). They were much more likely than profes-sionals to give an advertisement a high ranking for originality(Absolute Vodka, for example) and a mediocre ranking for logicor well craftedness. This variance on the individual subscalesmay explain some of the variance in the total rankings, whichaverage the results from the three subscales.

Top Rankings for the Original Subscale.

Ranking Professionals Public Students

1 Absolut Absolut PlymouthVodka Vodka Breeze

2 Plymouth Plymouth AbsolutBreeze Breeze Vodka

3 Ortho International OrthoWeed-B-Gon Coffees Weed-B-Gon

4 International Minute InternationalCoffees Maid Coffees

5 Minute Atrovent AtroventMaid Nasal Spray Nasal Spray

TABLE 3.

250


Top Rankings for the Logic Subscale.


1 Off Neutrogena OffCandles Sunblock Candles

2 Neutrogena Off MinuteSunblock Candles Maid

3 International Kibbles NeutrogenaCoffees ‘n Bits Sunblock

4 Minute Minute AtroventMaid Maid Nasal Spray

5 Absolut Atrovent Hershey’sVodka Nasal Spray Syrup

Top Rankings for the Well Crafted Subscale.


1 Absolut Minute MinuteVodka Maid Maid

2 Off International AtroventCandles Coffees Nasal Spray

3 International Plymouth OffCoffees Breeze Candles

4 Minute Off PlymouthMaid Candles Breeze

5 Plymouth Absolut Hershey’sBreeze Vodka Syrup

In general, the visual ads with striking, large illustrationsranked highest. Examples include the Absolut Vodka, MinuteMaid, Off, Plymouth Breeze, and International Coffees ads.There is minimal text in these ads. The pictures do the selling.This was especially true of ads judged the most well-crafted.

The ads that ranked lowest were those with smaller or lessinteresting illustrations and a lot of text, including ads for PurinaOne, Subaru, Comforel Pillows, Ro-Tel, and Gardenburger.

The Otho, Neutrogena, and Atrovent advertisements did wellin some rankings. They had a fair amount of text but also hadunusual images or typography, making them more visuallystriking than the other text-dense ads that did not fare as well.

TABLE 4.

TABLE 5.

251


This combination of attributes helped these ads in the rankingsfor logic and originality.

To see if various demographic variables lead to differentevaluations of advertising creativity, we ran repeated-measureANOVAs with Ad as the within-subject factor and each of thedemographic variables as the between-subject factor. Resultsindicated that except for Region, virtually all the other variableshad either significant or marginally significant main effects orinteraction effects on ad evaluations (see Table 6). This meansthat there were significant differences in the overall evaluationsof ads based on age, gender, income, education, whether theyconsidered themselves rural or urban, their experience and theiradvertising job titles (see Table 6). There were also significantdifferences based on media consumption, including magazinereadership, newspaper readership and TV viewing habits. Howadvertising creativity is judged varied significantly based onall sorts of factors.

Analysis of Variance of Evaluation Differences by DemographicVariables.

Main InteractionVariable effect P values with Ad P values

F statistics F statistics

Age 1.47** .046 1.04 .285Gender 8.61** .004 2.96** .000Education 0.25 .907 1.84** .000Income 1.65 .164 1.13* .057Race 1.91* .093 1.25* .088Region 1.22 .286 1.10 .239Rural/urban 2.82* .062 1.29 .152Experience(years) 3.46* .064 4.26** .000

Experience(job titles) 2.99** .012 1.59** .002

Magazinereadership 1.19 .297 1.23** .036

Newspaperreadership 4.30** .002 1.30* .073

TV viewing 1.47* .066 0.97 .633

*: marginally significant p (between .1 and .05); **: p < .05

DemographicVariables.

TABLE 6.

252


These findings are strong evidences that how we judgeadvertising is a function of who we are, as represented by vari-ous demographic and other descriptive variables. Evaluationsdiffer significantly based on all sorts of differences betweenindividuals.

Different groups of people judge the creativity of print adver-tisements differently. The inclusion of well-crafted as a third-dimension is the key for the CPSS scale to capture the subtledifferences among advertising professionals, students and thegeneral public in judging advertising creativity. Specifically,advertising professionals, college students, and a conveniencesample of the general public judged the originality and logic ofads similarly. Where their opinions differed was with regard tohow well crafted the ads were.

Professionals, whose experience and training should givethem greater insights about creative executions, judged theads differently on the Well-crafted subscale, and the results forthat subscale were sufficiently different to affect the overall Totalratings as well.

People of different ages, gender, professional experience,and other demographic groups judged the ads differently.People of different ages experience the same ad in differentways. Men and women saw the ads differently. Many factorsapparently affect the experience of looking at an ad and judg-ing its creativity.

When a group of people judges the originality or logic of anad, it appears that what is most important is the content of thead itself. People from all three samples seemed to agree onwhich ads were most original and made the most sense. As faras how well executed an ad is, however, training and experi-ence in advertising becomes the key factor. Advertising pro-fessionals judge the execution of ads differently from othersample groups.

Advertising is pervasive in Western society and may beexamined in terms of its content, its vehicle, product, andprocess, among other things. Industry lore presumes desirableand effective advertising to be creative. Industry awards arepredicated on this assumption. To date, advertising creativityhas been judged primarily by advertising professionalswho have training and experience in the creation of advertis-ing. One may also assume that advertising professionals at-tempt to predict that which will capture the attention of theconsumer.

DISCUSSION

253


ALTSECH, M. B. (1996). The assessment of creativity in advertising andthe effectiveness of creative advertisements. Unpublished doctoraldissertation, Pennsylvania State University.

AMABILE, T. (1982). Social psychology of creativity: A consensualassessment technique. Journal of Personality and Social Psychology,43, 997-1013.

BERLYNE, D. E. (Ed.). (1974). Studies in the New Experimental Aesthetics:Steps Toward an Objective Psychology of Aesthetic Appreciation.Washington, D.C.: Hemisphere Publishing.

BESEMER, S., & O’QUIN, K. (1986). Analyzing creative products: Refinementand test of a judging instrument. Journal of Creative Behavior, 20(2),115-125.

FLANDIN, M. P., MARTIN, E. & SIMKIN, L. P. (1992). Advertising effectivenessresearch: A survey of agencies, clients and conflicts. InternationalJournal of Advertising, 11, 203-214.

FLETCHER, W. (1990). The management of creativity. International Journalof Advertising, 11, 203-214.

KOVER, A., GOLDBERG, S., & JAMES, W. (1995). Creativity v. effectiveness?An integrating classification for advertising. Journal of AdvertisingResearch, 35(6), 29-38+.

KOVER, A., JAMES, W. & SONNER, B. (1997). To whom do advertisingcreatives write? An inferential answer. Journal of Advertising Research,37(1), 41-53.

MCEWEN, W. J. & LEAVITT, C. (1976). A way to describe TV commercials.Journal of Advertising Research, 16(6) 35-39.

O’QUIN, K., & BESEMER, S. P. (1989). The development, reliability, andvalidity of the revised creative product semantic scale. CreativityResearch Journal, 2, 267-278.

OSGOOD, C. E. , SUCI, G., & TANNENBAUM, P. H. (1957). TheMeasurement of Meaning. Urbana: University of Illinois Press.

RUSSELL, S. G. (1991). A comparison of approaches to problem solvingwith elementary school students. (Doctoral dissertation, BostonUniversity, 1991). Dissertation Abstracts International, 52, 2505a-2506a.

SCHLINGER, M. J. (1969). Cues on Q-technique. Journal of AdvertisingResearch, 9(3) 53-60.

SMITH, BRUCE L. (1993). Interpersonal behaviors that damage theproductivity of creative problem solving groups. Journal of CreativeBehavior, 27(3), 171-187.

WELLS, W. D., LEAVITT, C., & MCCONVILLE, M. (1971). A reaction profilefor TV commercials. Journal of Advertising Research, 11(6) 11-17.

Alisa White, University of Texas at Arlington; Fuyuan Shen, PennsylvaniaState University; Bruce L. Smith, Southwest Texas State University.

REFERENCES

Documents

Judging Advertising Creativity Using the Creative Product Semantic Scale