Questionnaire Writing

PROFESSIONAL DEVELOPMENT SERIES - 2009

Note to users of this presentation.• Market Researchers are freely welcome to use this presentation which I put together for

a professional Development session in New Zealand in April 2009.

• It is a long presentation – so you might be wise to trim it to suit your needs.

• Please credit those who have been quoted, and an acknowledgement to the MRSNZ would be suitable too as well as to Tom H. C. Anderson who hosts the Linked In group Next Generation Market Researchers: whose members kindly contributed much thought and discussion to this presentation.

• Otherwise go for it! We ought to share more notes professionally.

• One other request. If you find this presentation useful, and it saves you valuable time – then please consider making a donation to my cause – a school I sponsor in Cambodia. Savong’s School is a great project www.savong.com

Duncan Stuart FMRSNZ

Disclaimer

• While every care has been taken, etc etc... Responsibility for any errors all mine...blah blah...usual caution... Please use with care, failure to do so...result in sudden death ...sacking from employer etc etc... No hurting of children or animals.... Unintentional damage to property... Public humiliation...Will try not to run over-time in this presentation... No liability...good luck etc....better check with the lawyers

Duncan Stuart accepts full blame for any errors, omissions or oversights in this presentation. If you have any suggestions or questions please feel free to contact me.

duncan@kudos-dynamics.com

Introduction

• There is no element of market research so difficult as writing a questionnaire.

• You face a blank sheet of paper and you are required to “architect” a full experience for the respondent, a solid platform for the analyst, and do these things in a way that will deliver the solutions needed by the client. Underneath it all, gnawing away at the exercise, are two ghastly truths.

1. The way you write your questionnaire is doomed to be a compromise of length, style, demands and human biases.

2. GIGO. Garbage in. Garbage out. If you write an inferior questionnaire the whole project will deliver bad value.

• Good luck!

PROFESSIONAL DEVELOPMENT SERIES - 2009

Preparation for this Presentation

• Three main streams of input were combined to put this presentation together.

• Without proper theoretical training in research methods

and statistics, and practical experience to know what

does and doesn't work in practice, I don't know how

anybody can really conduct sound research except by

chance...which I believe we see a lot of in the market

research industry

Peter Hill Vice President of Research & Analysis

“

Personal Experience

• Involvement with phone surveys since 1986

• Face to face surveys since 1993• Online surveys since 1998

Desk Research

• The Psychology of Survey Response. Roger Tourangeau, Lance J. Rips and Kenneth Rasinski.

• Improving Survey Questions. Floyd J. Fowler. • The threat of Satisficing in Surveys: The Shortcuts

Respondents take in Answering Questions. Jon Krosnick. • Cognitive Interviewing. Gordon B.Willis.

Colleague Input – online discussions

• Great input from the research colleagues on two Linked in groups.

• Next Gen Market Research (NGMR)• Market Research Knowledge Forum.

BEFORE YOU EVEN START WRITING THE QUESTIONNAIRE

Think!Think!

BEFORE YOU EVEN START WRITING THE QUESTIONNAIRE

NEEDS OF FIELD

A clear roadmap that

assists

in communicating clearly.

You need to think about all the stakeholders...

NEEDS OF CLIENTSI want answers to my problem

NEEDS OF ANALYSTS

I need data that is str

uctured

in a useful way to test

hypotheses.

NEEDS OF RESPONDENTS

Please make it short,

relevant , clear and give me

room to share my true

opinion.

YOU ARE HERE

Minimise Response Effects

Think about the client

• For the client we are designing a process that delivers measurement, guidance or inspiration – sometimes all three.– Are they after results that measure something?

– Or results that guide them on a decision?

– Or is this an idea generation process?

• Be very clear about what you’re trying to achieve. Then work backwards from there...

...to think about the analyst

• For the analyst we need to design a document that generates data of the highest quality through questions that are relevant meaningful and answer-options that are crunch-able in order to deliver the measurement, guidance or inspiration required by the client.

• What will the analysis look like? What kinds of slides will you be generating? What format do they need the data in?

• Will they have a lot of time to work with the data – or just a short timeframe? (You can simplify things for them if necessary.)

• And part of that comes back a step...

...which is why you need to think about the field process

• For the field team (phone or online or face to face) we need to deliver a document that acts as a well designed roadmap that allows us to contact the right people and then doesn’t lead anyone astray or into grief.

• Who are we talking to? • How do we filter out those we don’t want?• What limitations does the field process put on

the questionnaire design? (Length? Language? Ability to show pictures?)

• And of course we need also to think about...

... the respondent

• For the respondent we need to deliver an experience that respects where they’re at – and the fact that they’re neither stupid or tolerant of being bored.

• We need to communicate (instruct, ask and listen) in a manner that is as clear and unambiguous as possible.

The questionnaire is the key to a whole process...

In the next section I’m going to discuss...

• The introduction of the questionnaire.• How to manage the flow of the questionnaire

experience.• Question writing – the art and the science, the

good and the bad (scales, question styles...)• Respondent fatigue and how to minimise it.• Respondent biases – how to reduce these.

THE INTRODUCTION IS VITAL

Minimise Response Effects

Rapport to win co-operation.

• Respondent is judging us based on our preliminary contact.– Are we relevant?– Are we approachable? (Will we bite?)– Will this be an engaging experience?– Should I agree to do this?

The sense of contract in a questionnaire.

• I’ll give you the answers you require so long as:– You don’t start probing into issues that I find too

deeply personal.– You don’t take undue time or cause me undue

inconvenience.– You promise to report my feelings fairly – and

that also means you’d better ask fair questions that don’t paint me into a box.

– You attempt to make the exercise even a bit entertaining or engaging.

– You promise I will not be harmed, bothered or put on some damned list as a result of taking part in this exercise.

Putting people into the frame.

– An introduction also serves to get people ramped into the subject matter.

– This is about opening up their memory-bank and giving them the opportunity to “tune-in” to the topic.

– A significant part of whole survey design is about this cognitive “tuning in process” –and it begins right here.

Recommendations.

• Introduce yourself warmly using natural language.• Give your name (if its online, add your photo, make it a

personal contact.) Dial up the rapport! Give a little of yourself.

• Share the purpose of the survey. “I have a client who makes peanut butter – and he wants to find out x, y and z. I hope you can help me.”

• Cover off the elements that are in the implied contract. (Privacy, confidentiality.)

• Hint at some of the subjects you’ll raise, and what to expect in the survey. “First I’ll ask what you currently eat – and then later, I’d like to show you some new pack designs...”

Online surveys are inherently more personal than CATI or even Face to Face. They communicate with a single voice – your own, while other surveys are conducted by multiple voices that need to be “scripted neutral.”

MANAGE THE FLOW OF THE QUESTIONNAIRE EXPERIENCE

• Before anything else, a questionnaire must first be a

communication tool. If respondents fail to understand

the questions as intended, everything else falls apart.

I work with a lot of suppliers and see a lot of

questionnaires, and this is clearly one of the greatest

weaknesses in the world of research today. Are we so

focused on the back-end analytics that we forget

about this? Dan Womack Senior Manager, Insights; AFLAC Worldwide HQ

“

Minimise Response EffectsMANAGE THE FLOW OF THE QUESTIONNAIRE EXPERIENCE

Structure your survey carefully.

• Yes, keep it as short as possible – but short or long, develop a smooth logical flow.

• Don’t zigzag or hopscotch. Try to be linear – for example start macro and slowly zoom in to the heart of the subject. E.g. start with the category, then the brand, then the specifics of the brand.– This is smoother and also helps cognitive flow.

• If any sections are a bit arduous, create a change of activity – so there’s rhythm in the experience. – For example if there’s a somewhat repetitive task in the

questionnaire, provide a “breather activity” that is easier and lighter and helps you gear-shift into the next activity.

Verbal or visual signposts.

– Clearly communicate the structure of the survey. “This survey will ask about x, y, and z and then we’ll wrap up by asking your opinion of some new pack designs.”

– Let respondents know that they’re making good progress and that the end is in sight. Progress bars and verbal cues help.

– Set localised expectations. “Next I’m going to ask you 6 questions about x – and then we’ll look at those new pack designs I want to show you.”

– Provide encouragement. “Thanks for that – in fact we’re 80% of the way through the survey now.”

My own signposts tend to be a bit wordy – and respondents tell me that! That doesn’t mean exclude signposts – but keep them succinct...

Better quality, fewer drop outs.

– Research is consistently showing that big arduous batteries of questions deliver dulled data quality. They encourage respondents to offer less discriminant answers.

– Surveys that “feel” too long result in respondent drop-out: which is a waste of everybody’s time.

– Surveys that get repetitive, or too zig-zaggy or simply too arduous all trigger the same warning light to the respondent: “Warning – no light, ever, at the end of this tunnel.”

Big batteries are too easy to write! They seduce us into covering off the brief with big long lists. They consistently produce inferior data.

Recommendations.

• Introduce yourself warmly using natural language.• Give your name (if its online, add your photo, make

it a personal contact.)• Share the purpose of the survey. “I have a client

who makes peanut butter – and he wants to find out x, y and z. Are you able to help me?”

• Cover off the elements that are in the implied contract.

• Hint at some of the issues you’ll raise, and what to expect in the survey. “Later, I’d like to show you some new pack designs...”

Online surveys are inherently more personal than CATI or even Face to Face. They communicate with a single voice – your own, while other surveys are conducted by multiple voices that need to be “scripted neutral.”

• The farther we get away from paper questionnaires administered in a face-to-face methodology by an interviewer (usually a housewife) using a clipboard and pencil, which are then manually edited and coded, perhaps on a kitchen table -- and the closer we get to programmed self-administered questionnaires that are never actually touched or seen when completed -- the more likely it is that we have questionnaire writers who don't know how to write an effective and engaging (and "respectful") questionnaire.

Dan Gersten Owner, Dan Gersten & Associates, LLC

“

• You definitely hit on a complete pet peeve that I forgot about! There is nothing more obvious or worse than a survey that has been defined for one mode and administered in another. There are significant question and answer/scale construction differences for telephone, on-line, face-to-face and self-administered. It really bugs me when I get a survey intended for the internet delivered by phone. Some survey software provides "seamless administration of surveys across multiple modes"...but this definitely doesn't mean that the same survey should be used.

Peter Hill Vice President of Research & Analysis

“

ASK THE RIGHT QUESTIONS

Minimise Response EffectsASK THE RIGHT QUESTIONS

The basics of a good question

1. Questions need to be consistently understood.2. Questions need to be administered or

communicated in a consistent fashion.3. What it is that constitutes an adequate answer

needs to be consistently communicated.4. All respondents should have access to the

information needed to answer the question. (Unless we’re actually trying to measure level of knowledge.)

5. Respondents must be willing to provide the answers called for in the question.

6. It must be reasonably engaging. (See over...)

1. KNOWLEDGE QUESTIONS remembering memorizing recognizing recalling identification recalling information who, what, when, where, how ...? describe

2. COMPREHENSION QUESTIONS interpreting translating from one medium to another describing in one's own words organisation and selection of facts and ideas retell...

3. APPLICATION QUESTIONS problem solving applying information to produce some result use of facts, rules and principles how is ... an example of ...? how is ... related to ...? why is ... significant?

4. ANALYSIS QUESTIONS subdividing something to show how it is put together finding the underlying structure of a communication identifying motives separation of a whole into component parts what are the parts or features of ...? classify ... according to ... outline/diagram ... how does ... compare/contrast with ...? what evidence can you list for ...?

5. SYNTHESIS QUESTIONS creating a unique, original product that may be in verbal form or may be a physical object combination of ideas to form a new whole what would you predict/infer from ...? what ideas can you add to ...? how would you create/design a new ...? what might happen if you combined ...? what solutions would you suggest for ...?

6. EVALUATION QUESTIONS making value decisions, opinions, judgements or decisions. Do you agree that ...? what do you think about ...? what is the most important ...? place the following in order of priority ... how would you decide about ...? what criteria would you use to assess ...?

BEHAVIOURAL QUESTIONS

IDEA GENERATION QUESTIONS

EVALUATION QUESTIONS

ATTITUDINAL QUESTION

EVALUATION QUESTION

DEEP PROBE - INTERPRETATION

DEEP PROBE - INTERPRETATION

Some rules of thumb about questions.

• For accuracy – make sure behavioural questions involve easily recalled timeframes. – Last week is better than “last month” For that matter

“In a typical week” is more accurate than “last week.”– Focus on the immediate past for behaviour –

predictive questions are notoriously unreliable. They’re an indicator at best.

• Avoid double barrelled questions. • Avoid value judgements or implied judgements. • Provide alternatives. Do you support stricter

gun control – or less strict gun control?

How many litres of milk have you purchased in the past decade?

Do you feel the church and politicians should speak out against tobacco and crime?

Do you favour higher taxes?

But the bigger problem is scales and code frames.

• What is the best way to code the respondents’ answers?

• When do I use a five point scale?• Should I label every point?• Is a bi-polar scale worse or better than a uni-polar

scale (Don’t agree to fully agree?)• Should I use rating – or ranking?

Some rules of thumb about scales.

• 5 and 7 point scales are considered optimum* Labelling every point on the scale (5pt – 7pt) is desirable.

• 10 point scales are good intuitive ways of rating something overall “out of 10...” because it is part of our vernacular.

• 10 point scales are better when they start at 0 and are actually 11 point scales. (They have a mid way point.)

• 100 point scales offer little if any useful advantage – and are more complex to analyse. (They probably need recoding.)

• Having a neutral or half way point is strongly desirable because many people feel this way.

• A three point scale (Agree/Neither-Nor/Disagree) is almost always way too blunt and not discriminant enough.

• *There’s debate about this still. In Europe 11 pt scales are much more widely used.

Typical Likert Scale1. Strongly disagree 2. Disagree 3. Neither agree or disagree 4. Agree 5. Strongly Agree

Where did the Likert scale come from? Invented by Rensis Likert (1903–1981). In 1932 he received his Ph.D in psychology from Columbia University. For his thesis work, Likert produced a survey scale (Likert Scales) as a means of measuring attitudes, showing that it captured more information than competing methods. (Wikipedia)

• A process of consciously going through a questionnaire and sense-checking everything.

The poodle has 9 puppies.The collie dog has 5 puppies.How many more puppies does the poodle have?

Answer. None I guess. Perhaps she stopped at 9.

Cognitive testing covers off 8 sets of potential problems – per question!Structure• How easy is the question to read?• What assumed information is missing?Content• Is the intent of the question clear?• Is the word precise and clear – or mushy and

ungrammatical?• Is the question unambiguous.• If time is an element (“how often have you

recently...”) is the reference point clear. How recent is recent?

Assumptions.• Does the question assume anything about the

respondent?• Double-barrelling: is more than one question

explicit or implied?Instructions• Are the instructions complicated, unclear, or

conflicting?

Knowledge/Memory• Are we assuming respondent has attitude or

knowledge of subject?• Are we assuming respondent can recall

details or information?• Are we assuming they can compute the

answer? Sensitivity/Bias• Embarrassing topics• Social Acceptability issuesResponse Categories• Difficult open enders?• Overlapping categories?• Missing categories?• Too many or hard to decipher categories?Other problems• Ordering problems• Technical issues

Guard against Irrelevancy.Guard against Irrelevancy.• The survey feels “nothing to do with me.”• Make sure you’re surveying the right people.• Pique their interest about the topic. Paper towels or

toilet cleaners or politics – it can all be engaging.

Shorten the Length – real and perceivedShorten the Length – real and perceived

• Surveys feel long if they zig-zag, feel repetitive, laborious or are...actually long!

• A long experience is more tolerable if it is engaging and even entertaining.

• Vary the activities, vary the complexity and give respondents a sense of direction. (See over)

Show some clear directionShow some clear direction

• In today’s parlance, a questionnaire isn’t a questionnaire...it’s a journey.

• Give it a clear beginning, middle, mountain to climb...and then ...an ending.

Laborious or forbidding questionsLaborious or forbidding questions

• Batteries, repetitive sets – cognitively hard questions. Too much can get respondents to quit.

• Intersperse harder questions with easier questions.• Break down hard issues into smaller chunks.• Explain what you’re doing. “I’d like to ask you this from

another angle – imagine you were serving wine and a wine snob was one of your guests...now what would you serve?”

Avoid RepetitionAvoid Repetition• There’s a fine line between having a rhythm to the

questionnaire (you know the drill – now let’s do it again for Brand Y) versus tedious repetition.

Slim it down. Avoid too much textSlim it down. Avoid too much text

• However succinct you are – see if you can take more flab out. (My copy can usually lose 25-30%). Do a word count on the questionnaire – be assiduous about paring out unnecessary verbiage. Shorter, simpler copy is easier for everyone.

Don’t treat respondents like they’re stupid.Don’t treat respondents like they’re stupid.

• Any questionnaire that confuses simpleton for simplicity is pissing off the respondent – big time.

• Caused by inadequate answer options, forced choices. sing song La la La “Low IQ” links and stupid questions – do you think we should have better government?

• Treat symptoms with intelligence. Confide with the respondent – talk to them off the record between questions – let them know what issues the client is thinking about. (They want to test a new pack to help lift sales – but they need your opinion...)

• Avoid jargon and marketing speak.

The Cognitive process of answering a question

Tourangeau and Rasinski have identified four basic cognitive steps in the way we process questions.

The Cognitive process of answering a question

Jon Krosnick has modified this outlook – suggesting that respondents might tend to “satisfice” – that is, give a ‘good enough’ answer by shortcutting the process.

EFFICIENT RESPONSE

The “satisficing instinct” can greatly reduce the precision of the data. We need to systematically address this to lift survey performance.

Get the gist.

Save effort.

Response order effectsResponse order effects• Primacy and recency – can effect the way we deal with and

retrieve our thoughts. • Primacy effects occur more in written surveys. First mentioned

gets more votes.• Recency effects occur more often in phone surveys. Last

mentioned gets more votes.

Acquiescence BiasAcquiescence Bias• A tendency to agree with unbalanced statements.• The satisficing theory suggests that respondents tend

to assume that the main option on offer is probably the “correct” option. The easiest response is simply to “agree.”

Ranking testsRanking tests• Ranking exercises impose a significant cognitive

burden – and are not very popular.• The results tend to be more accurate however –

compared to batteries of rating scales.• The answer is to use them sparingly – and keep them

relatively simple. (5 or 6 items is okay.)

The presence of a “No Opinion” filter.The presence of a “No Opinion” filter.

• Traditionally we’re trained to include “No Opinion” as an option.

• Findings suggest that most people who offer “no opinion” do have an opinion. By removing the option you encourage a more thoughtful answer.

Timeframe – the burden of memory recallTimeframe – the burden of memory recall

• People have faulty memories.• If the subject is not recent then memories can have an

unreliable telescoping effect. • The cognitive burden can be reduced through using

reference points (since Christmas...) and landmarks.• Other ways to reduce burden are branching formats (you say

you played more last year...was that before or after September?) as well as simple probes: (Any other?)

• Don't ask respondents to recall specifics about their shopping behaviour

more than a week or two in the past. For example, how many of your

purchases in the past year were made with a coupon? It's unrealistic to

expect anyone to remember that. I can barely remember what coupons I used

on my shopping trip earlier today. Similarly, asking shopping questions like

"how did you chose that brand of toothbrush?" ... unless someone is in the

store in front of the shelf, they won't correctly recall why they chose that

brand out of a hundred options on the shelf.

Valerie Skala Walker Marketing / Product Management / Market Research

“

Cognitive booster 1. Maximising Respondent Motivation

Cognitive booster 2. Minimise task difficulty

Cognitive booster 3. Minimise Response Effects

Avoid huge batteries.Avoid huge batteries.

• Tiresome to fill in.• Encourages satisficing – just tick

the boxes.• Anchoring effect: your answer for

the first question influences your answers for the rest of the battery.

Thinking back to your experience with Shop X, please indicate your degree of agreement with the following statements.

Strongly agree

Agree Neutral Somewhat disagree

Disagree Strongly disagree

Not sure

I prefer making a purchase from Shop X to using local offices, malls or stores. Ο Ο Ο Ο Ο Ο Ο

I prefer Shop X over other home shopping services (i.e., catalogs, "1-800" services or television).

Ο Ο Ο Ο Ο Ο Ο

Shop X doesn't just sell products or services--it entertains me. Ο Ο Ο Ο Ο Ο ΟI received special rewards and discounts from doing business with Shop X Ο Ο Ο Ο Ο Ο Ο

I say positive things about Shop X to other people. Ο Ο Ο Ο Ο Ο Ο

I consider Shop X to be my first choice when I need products or services of this type.

Ο Ο Ο Ο Ο Ο Ο

The "look" of the Shop X web site is appealing to me. Ο Ο Ο Ο Ο Ο Ο

I really like doing business with Shop X Ο Ο Ο Ο Ο Ο ΟI intend to continue to visit the Shop X site in the future. Ο Ο Ο Ο Ο Ο ΟI intend to purchase from Shop X in the future. Ο Ο Ο Ο Ο Ο ΟShop X is one of the first places I intend to look when I need the type of merchandise or services it provides.

Ο Ο Ο Ο Ο Ο Ο

It would require a lot of time and effort on my part, to set up an account with another shop .

Ο Ο Ο Ο Ο Ο Ο

It would take a lot of time and energy to look for another retailer for this type of product.

Ο Ο Ο Ο Ο Ο Ο

The products and/or services I purchased from Shop X were a good value. Ο Ο Ο Ο Ο Ο Ο

NONE OF THESE

BRANDS

THIS DRAG AND DROP EXERCISE IS MORE INTUITIVE – AND ACTUALLY HANDLES MORE DATA THAN THE PREVIOUS PAGE.

Here’s a sorting exercise: which words go with which brand?Simply drag the words into the appropriate bucket below.

Open endersOpen enders

• The best listening tool there is.• Captures opinions and ideas outside of

our initial scope.• Gives the human nuance to the data

(helps contextualise the numbers.)• Ideal for opinion, decision-focused and

idea generation research.

Don’t ask them if you’re not going to use them.

Don’t ask them if you’re not going to use them.

• People LIKE filling in the occasion open-ender – but don’t like the burden of too many.

• Don’t make these compulsory. If they’ve got nothing to say – don’t force them to say it.

• Usually (not always) these questions are for colouration – not numbers.

• The best project folk I know are those that started out in coding

departments (which don't really exist anymore) or field

departments or in some other area like graphics. Those are the

basics, that all too often were never learned in order to have

been forgotten. IMO, it is extremely helpful and valuable for a

person who is writing a questionnaire to have an

understanding of how the questionnaire is going to be

administered, how the data is to be collected, how the data is to

be processed and managed and analyzed and presented.

Dan Gersten Owner, Dan Gersten & Associates, LLC

“

What is analysis?What is analysis?

• Basically analysis is the process of converting raw numbers or words back into the format they existed in when the respondents were asked the questions.

• We ask people to express their feelings “out of 10” – and the analyst’s job is to translate those collective numbers back into feelings – and to make sense of what they’re hearing.

Talk with them about the outputs for the client.

Talk with them about the outputs for the client.

• What sort of data will they be able to analyse?

• Is it – about measurement, – about decision-making or – about idea generation?

• Will you have an adequate timeframe for analysis?

Some questions create more work. Others less.

Some questions create more work. Others less.

• Single choice questions are way easier to work with than multichoice (tick any that apply) questions.

• Big wide scales (out of 10 or 100) probably need recoding into a chunkier scale.

• Open enders require coding.• Complex branches or skips can create

patchy spreadsheets with ever-changing bases. Much harder to analyse.

Advanced analytics have special requirements.

Advanced analytics have special requirements.

• Factor analysis requires biggish batteries of questions. Or is that vice versa?

• Perceptual maps usually require at least 5 or 6 variables asked on the same scale.

• Segmentations can require similar batteries – 5 or six variables for example.

Then there’s the basics...Then there’s the basics...

• Keep all your scales running in the same direction. (Big = good is most intuitive for most people.)

• Try and standardise your scales to an appropriate extent.

Before you Before you go into go into field!field!

NEARLY THERE....

1. Develop and review the questionnaire as a separate document. I've seen too many people

trying to program before the questionnaire is nailed down. That's inefficient and likely to

lead to a survey that doesn't hit the spot.

2. Don't ask for Day/Month/Year of birth. I haven't yet found any valid reason why

researchers would do this. It just seems like laziness, and it raises concerns about

person information that will reduce response rates. It is generally best to offer a range of

ages. If you absolutely must ask for the year (because you don't know how to cut the

results?) just stick to the year.

3. Do your very best to make sure that options are realistic for the respondent. This

probably means putting in enough effort into pre-testing among outsiders, and possibly

adding Other/Specifies. I see far too many surveys - and distressingly even from

professionals - where the results are compromised because the respondent is forced to

choose an option that isn't accurate. Overall, to paraphrase others' comments - be

respectful of the respondent, and think of the survey as a conversation.

Mike Pritchard Experienced Insightful Market Researcher - Principal at 5 Circles Research

“

Test and test again.Test and test again.

1. Allow time for adequate testing.2. Check that the logic works.3. Check that your spelling is correct.4. Check that you didn’t cut and paste the

wrong scales to the wrong questions.5. Check that non-marketers understand

the questions.6. Conduct cognitive interviews to

thoroughly sound out the survey.7. For on-line, do technical runs to make

sure it works on low res, works on slow bandwidth etc.

• Prepare a detailed outline of how you want the report to look,

this is like a great insurance policy against the main mistakes

that you can make in questionnaire design and scripting. It

will ensure that you ask all of the relevant questions that are

required to deliver the results and insights that you need.

By and large, the challenges that you then face in

constructing individual questions will be smaller as you will

have a crystal clear idea of what you are trying to measure.

Mark Solonsch Customer Insights and CTran Leader at GE Money

“

Keep Keep learning!learning!

Expose yourself to feedback.

• For a listening profession, we don’t always listen.• At the end of questionnaires invite feedback – either a

rating scale or open ended, or both.• This provides really good guidance, as well as ammunition

to throw at cynics who say that nobody enjoys doing surveys.

Get a grip Duncan – this was not a good experience.

This was a very engaging, interesting experience.

1 2 3 4 5 6 7 8 9 10

Actual scale I used on a long survey for feedback.

Expose yourself to feedback.

• “Great survey, it is not until you do one of these you realise how much money you spend on things, made me feel good, with a tad of guilt, but you only live once i say, live it like its your last. Cheers!”

• ”I love this style that you did, kept me going, wasn’t all formal and stiff (i.e. boring) GOOD JOB keep up the good work and I look forward to participating in another of your quiz things that aren’t quizzes soon :)

• Thanks for giving me a taste of luxury on a day when I worked too hard to go out tonight! it was a little frisson for my soul.

• Too long!• Interestingly presented survey. Didn’t go on too long with

bulk "matrix" type questions e.g. what wine where and why. Images work well

• A great simplified survey, keep up the good work.

This is feedback after an on-line wine survey – average completion time 25-30 minutes.

Keep up your personal research.

• There are plenty of books on the subject.• There is limitless material on the web –including

useful academic studies on questionnaire best practice.

• There are professional chat groups where you can talk shop with colleagues.

• Don’t stop. The mark of a professional researcher – of any professional for that matter – is the continuous challenge of the status quo, and the continuous belief that “better” is possible.

Summary!Summary!

YOU THOUGHT WE’D NEVER GET HERE!

Summary

• Questionnaires are the single most difficult and riskiest task of the researcher.

• Questionnaires must need the needs to several competing stakeholders - and that’s what makes writing them so tricky.

• For the respondent we need to deliver an experience that respects where they’re at – they’re neither stupid or tolerant of being bored.

• For the field team (phone or online or face to face) we need to deliver a document that acts as a well designed roadmap that doesn’t lead anyone astray or into grief.

• For the analyst we need to design a document that generates data of the highest quality through questions that are relevant meaningful and answer-options that are crunch-able.

• And the for the client we are designing a process that delivers measurement, guidance or inspiration – sometimes all three.

• There are well established principles of questionnaire design that most of us regularly embrace – but there are also some well proven principles that may be new to us.

• We owe it to ourselves as professionals to regularly listen to all our stakeholders, and to continue on our own quest to lift our game. By way of example, there are many, many researchers using “CATI Questionnaire” approaches to the online environment – and this is not necessarily desirable.

• Finally – there is no one right way. A good questionnaire also contains your own DNA – your personal sense of style and character. Be yourself and keep working on it.

1. Construct questionnaire from general to specific - e.g. overall brand/concept ratings following by specific ratings of certain features, attributes.

2. Use simple language, no industry jargon.

3. Review each question to see if it assumes something on part of respondent - i.e. have done or seen something before, think a certain way...

4. Response options must be exhaustive and mutually exclusive

5. Response options must be balanced - i.e. can't have three positive response options and only two negative ones

6. Add "or not" to end of questions where it makes sense - Do you think this brand is hip or not? Does not dramatically alter the response distribution to not have "or not" but still worth adding.

7. Open debate on whether should have neutral response options (e.g. neither agree nor disagree) and it really does depend on circumstance

8. Also some open debate on whether should have "don't know" as explicit response option. For knowledge questions you certainly should, for opinion/attitude probably not but again, circumstance based.

9. No double barrel questions meaning asking two questions in one. Having "and" in a question is usually a dead give away but this can be very subtle.

Kevin Schulman Senior Vice President at TRG iSky

“

Duncan Stuart FMRSNZ

• 2004 - elected as a Fellow of the MRSNZ.• Twice judge, NZ marketing Awards.• Five times best paper or people’s choice winner at

MRSNZ Conferences. 1995, 97, 99, 01 and 03.• Extensive background in advanced analytics, text

analysis and mixture of qual and quant approaches.• 16 years in research.• 2009 – elected as a Life Member of the MRSNZ

• Telephone 366 0620• duncan@kudos-dynamics.com• www.kudos-dynamics.com

Thank you!

Duncan Stuart FMRSNZ

duncan@kudos-dynamics.com

Telephone 64 9 366 0620

www.kudos-dynamics.com

If you find this presentation useful please support the language school we built and support in Siem Reap Cambodia. www.savong.com The website has PayPal.