Research Method Lab Designing and Writing Effective Surveys

Research Method Lab

Designing and Writing Effective Surveys

Overview

This talk will address the following:

Types of survey questions 14 tips for improving your survey

writing Common survey pitfalls 10 criteria for a good survey

Types of Survey Questions

Open-ended or Verbal The expected response is a word,

phrase, or an extended comment Responses can produce useful

information but analysis can present problems

Some form of content analysis may be required unless the information obtained is for special purposes

Use sparingly or stick with interviews


Closed-ended, Likert Scale Not very important Very

Important 1 2 3 4 5 6 7

The respondents must indicate by selecting the number that best represents their attitude

A quick and easy way to measure more nuanced ranges of opinions


Closed-ended, Multiple Choice Which learning activity did you like best?

A. Small group B. Role play C. Mapping D. Computer modeling

When you want respondents to pick the best answer or answers, consider this type

Be sure to include specific directions


Closed-ended, Ordinal Please write a number between 1 and 5 next to each item below.

Put a 1 next to the item that is most important to you in selecting an on-line university course. Put a 5 next to the item that is LEAST important. Please use each number only once.

___ availability of instructor for assistance ___ tuition cost for the course ___ ability to work in groups with other students ___ quality and quantity of instructor feedback ___ number of students enrolled

Great for rating things in relation to other things

Again, be specific with instructions


Closed-ended, Categorical How many hours do you study on a school night?

0-1 2-3 3+

Be sure that categories do not overlap

Be sure to break down items into distinct and clear categories

Each respondent must “belong” in a specific category


Closed-ended, Numerical How many years have you taught

full time? When the answer must be a real

number, ask a numerical question Use these, especially with large survey

populations, to do an analysis of age range, years taught, etc.

Not as effective with smaller samples


Cumulative/Guttman Scale Here the respondent checks each item with which they agree The items are constructed so that they are automatically

cumulative– if you agree to one, you probably agree to all of the ones above it on the list

Can be a good way to gauge how people feel about controversial topics

Requires care when writing so that it doesn’t seem leading Example (from a survey guide): Please check each statement that you agree with: __ Willing to permit immigrants to live in the U.S. __ Willing to permit immigrants to live in your community. __ Willing to permit immigrants to live in your neighborhood. __ Willing to have an immigrant as a next door neighbor. __ Willing to let your child marry an immigrant.

Which One Do I Use?

Use Open-ended questions…when you want to get the respondent’s own words

Use Likert-Scale questions to assess a person’s feelings about something

Use Multiple-Choice questions when there are a finite number of options

Use Ordinal questions to rate things in relation to other things

Use Categorical questions when the respondent must fall into one section

Use Numerical questions for real numbers like age, number of months, etc. for large surveys

Use Cumulative questions to gauge degrees of feelings about controversial or complex issues

Contingency Questions

Contingency questions can be any of the styles previously discussed

Typically used when you want to filter out certain respondents

Respondents are directed to other questions in the survey

Too many jumps can confuse the reader Example: Have you ever been bullied in school?

Yes NoIf “yes,” go to question #4

14 Tips for Improving Survey Writing

1. Remember your survey’s purpose All other rules and guidelines are based

on this one There is a reason you decided to spend

time and effort doing a survey and you should ensure that every question you ask supports that reason

If you start to get lost while writing your questions, refer back to this rule


2. If in doubt, throw it out This is another way of stating the first rule A question should never be included in a

survey just because you can’t think of a good reason to discard it

If you cannot come up with a concrete research benefit that will result from the question, don’t use it


3. Keep your questions simple Compound sentences force respondents

to keep a lot of information in their heads, and is likely to produce unpredictable results

The following question is too complex for a clear, usable answer and needs to be broken down into component parts

Example: Imagine a situation where your department chair is out sick, two new students have enrolled mid-term, and the district’s test scores are about to be released next week. How supported do you feel by the administration?


4. Stay focused—avoid vague issues If you ask “please rate your satisfaction

with the school’s discipline policy” the answers will not lead to any specific action steps

Particular elements of the school’s discipline policy must be probed if responses are to result in specific recommendations


5. If a question can be misinterpreted, it will be

“What time do you normally eat dinner” will be answered differently by people living in different regions because it can refer to the midday or evening meal

Be clear, concise, always be aware of imprecise language, and avoid double negatives


6. Include only one topic per question “Please rate your satisfaction with the

implementation and enforcement of your school’s dress code” combines two issues

You need to break this question into two smaller ones in order to get responses that can lead to recommended actions


7. Avoid leading questions It is easy, and incorrect, to write a

question that the respondent believes has a “right” answer

“Most P.E. teachers believe that exercise is good for you. Do you agree?” is an example of a leading question

Even the most well-meaning researcher can slant results by including extra information in a question


8. Consider alternative ways to ask sensitive questions

Income, drug or alcohol consumption, sexual habits, religious beliefs, and political views are obvious examples of sensitive topics

But even topics like teaching styles, identifying biases (racial and gender), classroom management enforcement, and coworker relations can be sensitive

Questions like “did you vote in the last election” forces respondents into a corner—they might be unwilling to admit they did not vote because of civic pride or embarassment


9. Make sure the respondent has enough information

Asking respondents “How has the school board affected your classroom” isn’t as effective as “Last month the school board passed a ruling that requires a district committee composed of parents and teachers to review all textbooks. Did you know this?” followed by “What kind of changes to your department’s curriculum have you seen?”

Break these kind of questions into two parts- a screening item and a follow-up question


10. Response questions need to be mutually exclusive and exhaustive

If response questions are not mutually exclusive, the respondent will have more than one legitimate place for their answer

The response choices “1-2,” “2-3,” and “more than 3” pose a problem for someone who answers “2”

You must also ensure that the response options you provide cover every possibility

Providing a list of “reading,” “math,” and “science” for “What is your toughest class?” isn’t exhaustive enough


11. Keep open-ended questions to a minimum

While open-ended questions are a valuable tool, they should not be overused

They can result in respondent fatigue where you’ll only get short answers

These short answers can be avoided by using a set of well-designed, closed-ended questions

Open-ended questions also pose problems in terms of coding and analysis


12. People interpret things differently, particularly when it comes to time

Trouble spots include “always,” “sometimes,” and “never”

You must build in a temporal frame of reference to ensure that all respondents are answering in the same way

Example: “I am going to give you a list of magazines. For each one, please indicate whether you have read it regularly. By regularly I mean at least twice this week.”


13. Consider a “don’t know” response It is useful to allow people to say they simply do

not have an opinion about a topic Some researchers worry that people will opt for

that choice, reducing the ability to analyze responses

Evidence shows this fear is largely unfounded If you only want information from those with an

informed opinion about an issue or interest in a topic, offer a “don’t know” choice


14. Provide a meaningful scale The end points of Likert scales must be anchored

with meaningful labels The number of scale points can have little effect

on the conclusions you draw later An odd number of points provides a middle

alternative and provides a good way for respondents to head for the center

If measuring extreme opinions, use a scale with a greater number of points

You generally gain nothing by having a scale with more than 7 points

Common Survey Pitfalls: Bad Questions Stink!

Well written questions are critical Participants must stay interested If your respondents start to feel alienated by

threatening, emotional, lengthy, or difficult questions, response rates are likely to go down

Respondents can get frustrated if your questions do not provide answer choices that match their opinions or experiences

The quality of your collected data will suffer, your analyses will be less meaningful, and the whole research process will be useless

Common Pitfalls: Writing Good Questions is an Art

There are infinitely good ways to create surveys- no one person holds all the knowledge on the subject

There are also a number of common pitfalls, which if avoided, will increase the usefulness of survey data

Nothing compensates for poorly worded questions

The objective is to keep survey questions focused on single issues and topics, to keep them short, and make sure everyone can understand the questions

Barreling vs. Specificity

Barreled questions ask respondents to rate 2 or more behaviors or issues in a single question

It is impossible to attribute one answer to either issue

This is the most common problem with survey questions

Let’s correct the barreling…

Bad example: “When making assignments, my supervisor gives clear, achievable goals that are within my control”

How can we improve this question? Good example: “When making

assignments, my supervisor gives me achievable goals”

Jargon vs. Clarity

Specialized terminology is not generally understood

Specialized or unfamiliar words prevent those not familiar with the terminology from understanding and answering the question in a responsible way

Using acronyms is a possible use of jargon Stick with common speech

Let’s correct the jargon…

Bad example: “How would you rate your child’s WISC-III assessment experience?”

How can we improve this question? Good example: “How would you

rate your child’s experience taking the Wechsler Intelligence Scale test?”

Loose Bundling vs. Anchoring

This occurs when questions are too general and not grounded in specific behaviors

To improve loosely bundled questions, a more specific behavior must be identified in the question

What does the behavior look like?

Let’s correct the loose bundling…

Bad example: “I am interested in my students”

How can we improve this question? Good example: “I regularly share

information about my students’ progress with their parents”

Complexity vs. Simplicity

Complexity occurs when sentence phrasing is so long and labored that respondents become confused

Keep questions short and focused on single issues

Let’s correct the complexity…

Bad example: “What is the frequency of the overall interpersonal, informal, and formal communication between the focal group and the targeted semi-autonomous, functionally specialized groups?”

How can we improve this question? Good example: “What is the frequency of

the formal communication between the focal group and your group?”

Lack of Knowledge vs. Knowledge

Refers to when respondents either lack the knowledge necessary to accurately answer the question or have never bothered to organize their knowledge around the subject

Never ask people more than they know

Let’s correct the lack of knowledge…

Bad example: “The in-service presenter was an expert in differentiated instruction”

How can we improve this question? Good example: “The in-service

presenter effectively answered questions from the audience”

Social Desirability vs. Realism

This occurs when questions have an obviously “correct” or socially desirable answer

When respondents worry about giving the “right” answer, the data will be distorted

Make sure the question prompts respondents to answer with a reasoned opinion instead of an emotional response

Let’s correct the social desirability…

Bad example: “I frequently belittle and talk down to my fellow employees”

How can we improve this question? Good example: “People in my work group

frequently belittle and talk down to their fellow employees”

Or “There are times when the work pressure

gets so bad that I have been known to belittle and talk down to my fellow employees”

Leading & Loaded Questions vs. Fact

Leading and loaded questions are worded so that they influence respondents’ answers, creating bias

Avoid using emotionally charged wording

Let’s correct the leading & loaded questions…

Bad example: “Don’t you think that the Pledge of Allegiance should be required in school because it teaches patriotism?”

How can we improve this question? Good example: “The Pledge of

Allegiance should be required in school”

Halo Effect vs. Respondent Autonomy

Watch out for questions that link a position with a particular person or group

The respondent’s attitude about the person or group may influence their attitude about the position

Eliminate the link to a specific person or group by creating an unnamed group or dropping the link

Let’s correct the Halo effect

Bad example: “Do you agree with President Bush that schools need to be accountable to taxpayers?”

How can we improve this question? Good example: “Do you believe

that schools need to be accountable to taxpayers?”

Conclusion: 9 Criteria for a Good Survey

1. The target population (the group of people from whom you want feedback) is well-defined

2. The actual people you survey matches your target population

3. The group you survey is large enough to gain useful data

4. Good follow-up minimizes non-response

5. The type of survey used is appropriate

Conclusion: 10 Criteria for a Good Survey

6. The questions are well-worded 7. The survey is properly timed,

both in the amount of time it takes to complete it, as well as when it is distributed

8. The people giving the survey are well-trained

9. The survey answers your research questions

Documents

Research Method Lab Designing and Writing Effective Surveys