THE STRATEGIES

THE STRATEGIES

The Strategies is a collection of recent work from a new market and social research company.

Strategies conducts reseach for creative agencies that proves the potential of their ideas.

In 2014 validating insight and creative was too expensive, and took too long. Not anymore.

For rates see back page or visit www.thestrategies.com.au

Photography by Will Bradenwww.will-braden.com

HOW WOULD A SCIENTISTGO ABOUT PRE-TESTING?

Ask anyone in advertising regularly involved in research, and they will tell you the ways we go about pre-testing, and market research in general, are less than ideal. These flaws persist because of two consistent pressures

placed upon market research, those being the need for it to be fast, and the need for it to be cheap.

Those two pressures are very effective at ensuring market research and pre-testing are done in the same ways over and over again, but it need not

be that way. How can we improve our research methodologies without additional time or cost? Start by asking the experts.

The following is a record of four conversations held with post graduate Psychologists at the Univeristy of Sydney in November 2014

Simon JacksonPsychology PHD candidate.

Simon Jackson is a psychology PHD candi-date specialising in individual differences in decision making. In his words

“Why do you pick A when I pick B?”

When I describe the way concept testing is often conducted in a discussion group, he immediately identifies the group setting as likely to undermine the validity of the deci-sions made by respondents,

“When we are part of a group we invariably tend towards overconfidence.” He adds, “In

research like this, your aim should be to maximise control. You recruit participants based on their belonging to a certain target market, but the control you give up by plac-ing them in a group setting really makes the

benefits of that negligible.”

It may come as no surprise that Simon also highlights the self report style of a discussion group as another threat to the validity of any research,

“ People’s ratings are very seldom the best predictors of any sort of behaviour.” “The

main problem i see is, the measures that you have described to me(such as likeability),

are they good proxies for the behaviour that you’re trying to address?”

How then would Simon go about designing a study to determine the potential effectiveness of a piece of advertising?

“Start by defining the variable of interest more narrowly than effectiveness. For instance, that could be influence over purchase intent, brand affinity, or premium perceptions. Then make sure you measure this variable with control

questions.”

Simon explained by performing the following test:

What is your surname?

How confident are you in the answer you’ve just given?

Would you bet on it?

What is the longest river in Asia?

How confident are you in the answer you’ve just given?

Would you bet on it?

The idea is that there is a control criterion an individual imposes on their confidence that allows them to translate that experience into an action. The analogy is that asking someone to rate how much they like something does not necessarily indicate whether they are likely to purchase or not. They have to like something enough to purchase, and the control question (in this case- would you bet on it?) will tell you if you have reached that point.

DECISION MAKING

Dr Alex RussellPost doctoral Research FellowSouthern Cross University

From personal experience with market research Dr Russell knows how the time and cost pressures associated with market research influence outcomes.

“The client gets what they want, often at the expense of research rigour. So some

market researchers will develop a battery of tests and end up doing them over and over again, without necessarily providing

the best research possible.”

Testing with Animatics is an example of this.

“Testing cartoons against proper, fully made ads, that’s never going to help. That

will most likely bias ratings.”

Dr Russell’s expertise in statistical analysis puts him in a position to answer an im-portant question to an advertising com-munity still learning how to derive insight from large tranches of data.

“There’s a broader philosophical argument about whether statistical significance is the way to go. No result should be pre-sented as perfect or infallible. By defini-tion one in every 20 significant results is an error. But statistical significance is an

easy way to make a decision.”

“There’s another argument regarding whether a result is statistically significant versus whether it is meaningful. Big sample sizes are nice, but

they can give statistically significant results that don’t necessarily mean anything.”

He advocates moving our focus away from just liking and purchase intent.

“Learning why something is likely to be pur-chased is the really important thing, and un-

fortunately people often don’t have insight into why they like things.”

How would Dr Russell design a pre-test?

“One example would be to place two prod-ucts side by side and ask how they differ, then measure the products on these characteristics and work out which ones drive liking.” “The most important thing really, is to work out

what needs to be measured. What is the central problem that this advertising must address.”

So the research should evaluate the ad against the communications challenge.

“Once you know which measures differentiate the ads, you can ask more focused questions. Test the features of the ad and message rather than just likability and purchase intent. And

don’t be afraid to use more advanced statistical analyses to work out these relationships.”

RESEARCH DESIGN AND STATISTICS

EYE TRACKING AND NEURO IMAGINGDr Cyril LatimerAssociate Professor of PsychologyUniveristy of Sydney

Eye movement tracking, along with neu-ro-imaging are almost certainly the pre-test-ing methodologies least understood by agency and client alike. Dr Latimer is an expert on these very testing methods, so is the right person to determine if they have a role in the evaluating the effectiveness of a piece of advertising.

Can there be a link drawn between what the eye fixates on and what is being interpreted?

“Some would say yes but I’m sceptical of that. There’s often a disjunction between

what the eye is looking at and what is being attended to.”

If pre-testing is about measuring the per-suasiveness of messaging, can it be said that either eye movement tracking or neuro-tes-ing can do that?

“Going beyond that and saying what the person is doing is a whole other world. Peo-ple try to infer that kind of thing from PET scans and EEG and it’s very difficult if not

impossible.” “Sure the brain is going to be ac-tive in different parts, but it’s active in many

areas, across millions of neurons.”

“You could say the brain reacting represents an element of surprise. But that’s about

all you can say. You can’t say what the people were doing or thinking.”

This raises the question, is a fluctuation inbrain activity, which could be interpreted as surprise, an adequate indicator of engage-ment, effectiveness, or persuasion?In other words, is; is there any way to predict how a person will behave in the future based on the activity in their brain, or the movements of their eye when exposed to a piece of communication?

“You may be able to predict where they will look next, and from that you may infer

what they are attending to. But it wont tell you what they’re thinking or what their

judgements would be of what they’re looking at.”

It appears then, that modern pre testing methods like neuro-imaging and eye movement tracking can at best be thought of as providing feedback on art directional elements, not measurement of cut through or engagement, and certainly not thematic or conceptual testing as they are often presented.

Ultimately, Dr Latimer is particularly suc-cinct in highlighting the difference between scientific and market research.

“There are two ways to go about this. You can have a theory that determines what

you look for and what you do. Or you can have what I call a fishing expedition where

you just collect data and hope that some theory is going to emerge.”

In light of this, it’s alarming that market research very rarely begins with a stated hypothesis.

PERSONALITYBeatriz Lopez PortilloPHD Candidate Sydney UniversityIndividual differences in personality

Beatriz explains that in discussion groups, people are asked to speak on behalf of their demographic group, but they are in fact speaking as individuals. Their response is based on their environment and personality, and their preference is influenced as much by those as it is by their belonging to a certain demographic group. The fact that there is a human moderator also exacerbates these differences in personality.

“Men and women will respond to an adver-tisement differently if it includes a face. They also respond to the gender of the moderator differently.” “We have seen in research that women are more hostile and are not that

open with other women.” “Even heterosex-uals and homosexuals are different when it comes to the answers they give in research.

Homosexuals are typically more open to ex-perience and respond more positively.”

When asked if certain personality traits are over represented in certain demographic groups Beatriz responds

“Low SEC demographic are likely to be more neurotic, but usually it’s really random.”

“Usually people with introversion and neu-roticism will respond to facts and prefer a

more serious moderator that looks academ-ic. People with extroversion will prefer someone who is casual and gregarious.

The association with the moderator is a significant influence”

The take out from this is, recruiting a group based on their belonging to a target demo-graphic is good for insight mining, but that method is inherently less effective when we are asking people to subjectively evaluate a piece of communications because we don’t currently control for different personality traits in the group.

How would Beatriz improve the discussion group? “

Follow up is key. Doing a follow up with respondents gives you the certainty that your idea has long term potential. It eliminates the research effects. More stability, more certain-ty, and will save you money in the long term. With a follow up you test the concept in the

real world.”

So what can we do with this advice? The discussion group will continue to be a useful market research technique. It’s not always the answer, but it is a pretty versatile method-ology. Beyond that, we now have a few easy things we can do to ensure we aren’t mislead by a flaw in the design of any research we do.

It’s easy to see why narrowing down the ob-jective to to a central question, and develop-ing a hypothesis would lead to better results. Taking into account individual differences in respondents is another easy win. Ultimately it’s about designing our research to be more like experiments, and fortunately we can do that without additional time or cost.

DATA DATINGThe application of ‘Moneyball’ economics to dating has al-

ready been introduced. In July of 2014 Psychologist Ty Tashi-ro proved that focusing on undervalued traits will help singles

find better dates, and more successful relationships. For in-stance, “Moneyball dating” advocates seek out likability in a date over physical attractiveness, because likability is a more

accurate predictor of long term relationships success, and any date that is both highly likeable and highly attractive will be

highly valued and hard to find.

Of course, it takes longer to determine likability than it does to assess attractiveness, so match making sites like eHarmony or Match.com employ algorithms to produce a score that esti-mates likability. Now if we are looking online for a successful romantic relationship, we see the valued traits, and we also

know a little about the less valued “Moneyball” traits.

This is all useful, except that for the fastest growing segment of digital daters, this isn’t what online dating is about. Tinder and similar apps are all about the most highly valued traits.

Rapid decisions, made in bulk, based on the most fleeting and superficial impressions make the proven benefits of “Money-ball dating” difficult to apply. It’s fair to say that many Tinder

users are not necessarily looking to meet the love of their lives, and that Tinder was designed for that particular user,

but new alternative apps are emerging which suggest an appe-tite for some measurement of a potential dates possession of

undervalued traits.

Last week saw the soft release of Tinder-style dating app ‘The League’, which adds more layers to the purely visual. Geo-location data and Linkedin connectivity, among other sources are collated to cater to mobile daters who want to use Tinder, but also want to

know more about the people they decide to meet in the real world.

So what does this all mean? We have proof that the more we know about the people we meet online, the more likely we are to find a successful relationship. We also see that the growing popularity of Tinder has attracted users who are not necessarily using it for the design of the app, but the age of the users. If Tinder users are

young, and you want to date young people, you’re dates are likely to be on Tinder.

So what happens now? Do all the dating audiences fragment into niche mini-Tinders? Do all the users just suck it up and repeat-

edly take punts on good looking strangers? One strategy could be to take advantage of the increasing amount of personal data the mobile daters are already generating. All the health, fitness, and geo-location data tracked by devices like Nike’s Fuel band, Peb-ble watches, and now Apple’s iPhone are measures that do relate to the desirability of a potential date. The addition of this data to Apps like Tinder represent an easy way for users to get a Money-ball-like advantage, but the really significant implications are for

users in tech hungry, developing markets, where chastity may still be a valued commodity, or what we think of as a traditional dating scene may be less accessible. The ability for users in these cultures to simultaneously meet new people, and fast track their identifica-

tion of quality potential dates really increases the power and value of these mobile dating apps.

The addition of this kind of quantified self data to mobile dating surely will be a developing trend in 2015.

THE SUPPOSED WISDOM OF THE CROWD

The central assumption of the phenomenon known as the wisdom of the crowd is that the average of a group of independent predictions is likely to be more accurate than any randomly selected individual prediction. The bigger the crowd, the better the prediction. It’s part

observable effect, part misleading heuristic, and it’s the reason we so rarely challenge consensus among respondents in consumer research.There are plenty of reasons why a group of consumers might struggle

to accurately determine the potential effectiveness of an advertise-ment, and some of them have been on show recently, as the collective wisdom of Twitter repeatedly swung and missed when predicting the

outcomes of both 2014 football grand finals.

In the Australian rules final we had an excellent opportunity for the wisdom of the crowd to work as it is supposed to. The large upset

means that the collective wisdom in the crowd of predictions should be seen in (if not an accurate prediction) a more accurate, or moderate

prediction than that made by a bookmaker. The opposite happened.

Collating every tweet appearing the hours leading up to the opening bounce hashtagged #AFLGF resulted in an average prediction errone-ously in favour of the Swans, that predicted a larger winning margin than that predicted by the major online bookmakers. Twitter had the

Swans by 15.5 and the bookies on average had the Swans by 10.5.

The crowd was wrong, so were the bookies, but in an instance where you would expect the crowd to be more

accurate, they were more wrong.

Not surprisingly, Twitter also got the first goal scorer and man of the match predictions wrong.

Turning to the NRL grand final and we have another surprising result. Not an upset this time, but a larger winning margin than the bookies

and tipsters predicted. In this case the twitter hive-mind correctly

picked the favourite (Souths) as the winner, but consensus was less clear. Twitter predicted a Rabbits victory by 5.1, while the bookies

margin was 8.5. The actual margin was a much larger 24 points, but again, the crowd was less accurate than the bookies.

The mistakes these two groups made are actually typical of groups reaching consensus, and they happen in consumer research all the

time. The AFL crowd reached a consensus that was too strong, and so their overconfidence in the Swans led them to overestimate the score

line. On the other hand, the NRL crowd didn’t reach a strong consensus, and so they were more conservative in their prediction.Of course, this all assumes the predictions made by bookmakers are

not also decided by groups. Andy Lulham, Head of marketing at odds comparison website Oddschecker.com.au was kind enough to offer a

detailed explanation of the process.

In the case of large online bookmakers odds are set by a member of a trading team. The individuals who make up the trading team each car-ry specialist knowledge and expertise in specific sports. “These traders

will have their own individual expertise, plus a huge set of previous data, and analytical programs that will help them establish the proba-

bility of an outcome happening.”

Once the odds are set, the bookmaker will build in a profit margin ($2 may go to $1.90 for instance). “Typically speaking, in an event with fewer outcomes (head to head), the bookie will have a lower margin

than in a market with numerous potential outcomes (first try scorer).” “Once punters start betting on an event, the prices are quite likely to

change as the bookmaker manages their position.” “It’s not uncommon for bookies to effectively hedge some of their liability by betting them-

selves with other bookmakers as another way of managing their level of risk”

“Typically, if a lot of favourites win big matches, then the bookies win less money off punters than when the lesser backed outsiders win. That said, the bookmakers obviously come out on top over the long run, no

matter what the trend of results are.”

RATES

WEEKLY ONLINE OMNIBUS

100 responses at $100 per question1000 responses at $250 per question

COMMISSIONED RESEARCH

Ethnographic, Qualitative, Quantitative projects$500 per day

@THE_STRATEGIES

INFO@THESTRATEGIES.COM.AU