Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner...

Survey Design 201

Getting from “Doing a survey” to “Doing a survey right.”

Andy ZehnerOffice of AssessmentYoung Hall, Rm 521

alzehner@purdue.edu

Basic tips

1. Don’t ask what you already know2. Avoid biased or leading questions3. Make choice options:

1. exhaustive2. exclusive3. balanced

4. Avoid double-barrelled questions5. Don’t assume knowledge of terms

6. Avoid jargon7. Be brief 8. Be specific9. Do/Don’t include a “neutral” option10. Avoid complex negative constructions11. Order questions strategically12. Save controversial questions for last

Spot the flaw

1. “Rate the quality and variety of the food at Wiley Dining Court”

2. “CCO Career Services are:”• Excellent Outstanding Very good Good Average

3. “Have you participated in any flipped courses during the past year?”• Yes No

4. Were the clerical staff who served you in [office] courteous?1. Yes2. Neutral3. No

5. What portion of the material did you master by the end of the training session?• All Most Some Very little None

New considerations

• Vague quantifiers• Individual response style• Dunning-Kruger Effect

Dunning-Kruger Effect

Kruger & Dunning, Journal of Personality and Social Psychology, 1999

What evidence of Dunning Kruger Effect looks like

Very poor Poor Fair Good Very good Excellent0%

10%

20%

30%

40%

50%

60%

Freshmen 1Freshmen 2Seniors 1Seniors 2

2014 Purdue SERU survey

Vague quantifiers

Vague quantifiers

How often have you…

Had a class where the professor knew your name

Never

Rarely

Occasio

nally

Somewhat o

ftenOften

Very often

0%

5%

10%

15%

20%

25%

30%

Communicated with an instructor by email/in person

Never

Rarely

Occasio

nally

Somewhat o

ftenOften

Very often

0%

5%

10%

15%

20%

25%

30%

2014 SERU Survey, Purdue data

Vague quantifiers

• “Always” almost always means always• “Never” almost always means never• Any other vague quantifier is . . . vague• Vague quantifiers convey relative position only• Meaning varies from one person to another• Meaning varies from one question to another• Grammatical opposites aren’t perceived equally

“Sometimes”

• “asking questions in class” 2 times per week• “working with other students outside of class” 1 time per month• “making a class presentation” 2 times per

semester• “participating in a community-based project” 1 time per year

How Often is Often Revisited: the Meaning and Linearity of Vague Quantifiers Used on the National Survey of Student Engagement -- Laird, et al., Indiana University, 2008

Vague quantifiers of behavioral frequency, Marincic, 2011

“Stay away from using words like “often,” “usually,” “generally,” etc. Each person’s thought process is different and some people may infer a different meaning.”

-- Smart Survey Design, Survey Monkey, 2011

Retrieved 3/11/2015

Consider using idiosyncratic terms!

2013 Purdue Student Satisfaction Survey

Response style

Moderate & Extreme Response

Bursar’s drop-box

Very unim

portant/d

issati

sfied

Unimporta

nt/diss

atisfi

ed

Neither

Importa

nt/Sati

sfied

Very im

portant/s

atisfi

ed0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Importance Satisfaction

Registering for classes

Very unim

portant/d

issati

sfied

Unimporta

nt/diss

atisfi

ed

Neither

Importa

nt/Sati

sfied

Very im

portant/s

atisfi

ed0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Importance Satisfaction

2010 Purdue Student Importance & Satisfaction Survey

Quotes from focus groups, Purdue, 2012-2013

“I feel like here in the US people use stronger

language, like ‘Oh, this is so wonderful.’ But for me,

I’m not so willing to say it.”

“Americans use a lot of hyperbole: ‘It’s super, it’s

wonderful!’ That is not language we use where

I come from.”

Factors that determine acquiescence

• Individualism -.445 *• Power distance .372 *• Extraversion .446 **• Uncertainty avoidance .533 *

Response styles in cross-national survey research: a 26-country study, Harzing, et al, 2006

Response styles by nationality

30 35 40 45 50 55 60 655

10

15

20

25

30

Acquiescence

Extr

eme

Resp

onse

Sty

le

Japan

India

Germany

Holland

Mexico

Turkey

China

Some words don’t translate well

“good” v. 良い / いい

“normal” v. нормальный

Does America spend too little, about the right amount, or too much money on…

Too little About right Too much0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

"Welfare" "Assistance to the poor"

General Social Survey, 2014

Again: Does America spend too little, about the right amount, or too much money on…

Liberal respondents

Too little About right Too much0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

"Welfare" "Assistance to the poor"

Conservative respondents

Too little About right Too much0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

"Welfare" "Assistance to the poor"

Order of choice options matters

Purdue’s favorite pizza:

Hot Box / Mad Mushroom / Villa@PMU / Domino’s

Domino’s / Villa@PMU / Mad Mushroom / Hot Box

Biasing Effects of Scale-Checking Style in Response to a Likert Scale, American Statistical Assn, Friedman, et al., 1994

Numbers on the scale matter, as much as the words do

0 1 2 3 4 56 7 8 9 10

Not at allExtremely

Successfulsuccessful

-5 -4 -3 -2 -1 01 2 3 4 5

Not at allExtremely

Successfulsuccessful

Rating the Rating Scales, Journal of Marketing Management, Friedman & Amoo, 1999

You can get reliable results with a moderate sample size or response rate

Stronglydisagree

Disagree Neutral Agree Stronglyagree

0%5%

10%15%20%25%30%35%40%

True distribution Sample of 5000

Stronglydisagree

Disagree Neutral Agree Stronglyagree

0%5%

10%15%20%25%30%35%40%

True distribution Sample of 1000

Stronglydisagree

Disagree Neutral Agree Stronglyagree

0%5%

10%15%20%25%30%35%40%

True distribution Sample of 500

Stronglydisagree

Disagree Neutral Agree Stronglyagree

0%5%

10%15%20%25%30%35%40%

True distribution Sample of 50

After the shot went off, my

training took over.

Resources and links:

A good overall lesson in 106 slides

Survey Monkey’s guide

Cross–national response styles

Rating the Rating Scales

Vague quantifiers: How Often is Often?

Vague quantifiers: Marincic dissertation

More about importance/performance analysis

Tex Grebner’s video (language)