STAT 472Survey Design

Constructing the Questionnaire

Constructing the Questionnaire

A good questionnaire forms an integrated whole.

The researcher weaves questions together so they flow smoothly.

He or She includes Introductory Remarks, Instructions for clarification and measure each variable with one or more survey questions

Principle of good questions There are two keys principles:

- Avoid confusion (Keep it clear and simple)

- Keep the respondent’s perspective in mind

Good survey questions give the researcher Valid and reliable measures Help respondents feel that they understand

the question and that their answers are meaningful

Do not mesh with a respondent's view point

Question writing is more of an art than a science. It takes skill, practice, patience, and creativity

A survey researcher must exercise extra care if the respondents are come from different life situations

Researchers face a dilemma

All respondents hear exactly the same questions, equally clear, relevant, meaningful and exact wording

If respondents have diverse back grounds and frames of reference, the exact same wording may not havethe same meaning. Yet, tailoring question wording to each respondent makes comparisons almost impossible

10 things to avoid when writing a question survey

(1) Avoid jargon, slang, and abbreviations:(unless a specialized population is being

surveyed)

Jargons and technical terms come in many forms: Plumbers talk about snakes

Slang: is a kind of jargon Skiers about a hotdogAbbreviations: for example NATO means North Atlantic Treaty Organization North African Tea Office National Auto Tourist Organization Native Alaskan Trade Orbit

(2) Avoid ambiguity, confusion and vaguenessAmbiguity and vagueness plague most question

writers For example: What is your income?

Could mean, weekly, monthly, or annual; family or personal; before tax or after; for this year or last year; from salary or from all sources

10 things to avoid when writing a question survey

"Do you jog regularly? Yes / No” Use of indefinite word regularly or response categories“hinges on the meaning of the word regularly. It may define as every day, others as once a week” Good Question: “To reduce respondent confusion and get more information”

Do you jog "about once a day”, “a few times a week”, “once a week”?

The question wording is an art that may improve with practice, patience, and pilot testing

Here are three survey questions written by experienced professional researchers. They revised the original wording after a pilot test discover that 15% of respondents asked for clarification or gave inadequate answers (e.g., don't know)

Q1. Do you exercise or play sports regularly?

Original question

Problem: What counts as exercise?Response to question: 48% saying “YES”

% asking for clarification: 5%

Q1. Do you do any sports or hobbies, physical activities, or exercise, including walking, on a regular basis?

Revised question

Response to question: 60% saying “YES”

% asking for clarification: 0%

Q2. What is the average number of days each week you have butter?

Original question

Problem: Does margarine count as butter?Response to question: 33% saying “none”

% asking for clarification: 18%

Q2. The next question is just about butter-not including margarine. How many days a week do you have butter?

Revised question

Response to question: 55% saying “none”

% asking for clarification: 13%

Q3. [Following question on eggs] What is the number of servingsin a typical day?

Original question

Problem: How many eggs is a serving? What is a typical day?Response to question: 80% saying “one”

% asking for clarification: 33%

Q3. On days when you eat eggs, how many eggs do you usually have?

Revised question

Response to question: 33% saying “one”

% asking for clarification: 0%

(3) Avoid emotional language and prestige bias

Example (emotional): What do you think about a policy to pay murderous terrorists who threaten to steal the freedom of peace-loving people?

Is full of emotional words (murderous, freedoms, steal, and peace)

Problem: respondents may react to the emotionally laden words rather than to the issue

Example 1 (Prestige bias) Most doctors say that cigarette smoke causes lung disease for those near a smoker, do you agree?

Problem: tends to provoke "yes" answers because people trust doctors

Example 2 (Prestige bias) "Do you support the president's policy on Zimbobutu?

Problem: tends to provokes "yes" answers because people trust president

(4) Avoid double-barreled questions

Double-barreled questions consists of two or more questions joined together.

Example: How do you evaluate the work of the Jordanian parliament and prime minister?

The two variables are: Jordanian parliament : prime minister

The wrong way is to ask one question:How do you evaluate the work of the

Jordanian parliament and prime minister?

The correct way is to ask two questions:1- How do you evaluate the work of the Jordanian parliament?2- How do you evaluate the work of the prime minister?

Example: Tell me whether you like summer and winter in Irbid or not?

The correct way is to ask two questions:1- Do you like summer in Irbid?2- Do you like winter in Irbid?

(5) Avoid leading (Bias) QuestionsMake respondents feel that all responses are legitimate. Do not let them become aware of an answer that the researcher wants.

A leading question is one that leads the respondent to choose one response over another by its wording.

Example: The government should force you to pay higher taxes.No one likes to be forced, and no one likes higher taxes. Alternative: The government should increase taxes, or the government needs to increase taxes.

Example: Don’t you think that suffering terminal cancer patients should be allowed to be released from their pain?

Researchers should never try to make one response option look more suitable than another.

Alternative: The suffering terminal cancer patients should be allowed to be released from their pain, do you agree?

Example: Wouldn’t you like to receive our free brochure?Example: You don't smoke, do you?

(6) Avoid asking questions that are beyond respondents capabilities.People have cognitive limitations, especially when it comes to memory of past eventsExample: (asking adult) How do you feel about your brother when you were 6 years old?is probably useless

It is pointless to ask people about things that are not natural ways for them to think. Example: How many gallons of gasoline did you buy for your car last year?

"Yet, respondents may be able to answer a question about gasoline purchases for a typical week, which the researcher can multiply by 52 to estimate annual purchases

Constructing the QuestionnairePrinciple of good questions

(7) Avoid false premises (assumption/ hypothetical )

Since the assumption is not correct, the conclusion drawn may be in error.

For example, false premise:If the streets are wet, it has rained recently.

(premise)

The streets are wet. (premise)

Therefore it has rained recently. (conclusion)

Example: The post office is open too many hours. Do you want it to open four hours later or close four hours earlier each day?

Do not begin a question with a premise with which respondents may not agree, then ask choice regarding it.

leaves those who either oppose the premise or oppose both alternatives without a meaningful choice

Respondents who disagree with the premise will frustrated and not know how to answer

A better question explicitly asks the respondent to assume a premise is true, then asks for a preference

A better question: Assuming the post office has to cut back its operating hours, which you find more convenient, operating four hours later or closing four hours earlier?

Answers to a hypothetical situation are not very reliable, but being explicit will reduce frustration

Example: When did you stop beating your kids?

A better question: Have you ever slapped, punched, or hit your kids?

(8) Avoid asking about future intentions (hypothetical )

Avoid asking people about what they might do under hypothetical circumstances far in the future. Responses are poor predictors of behavior removed far from their current situation or far in the future

Example: Suppose a new grocery store opened down the road, Would you shop at it?

Respondents answer specific, concrete questions that relate to their experiences

It is better to ask about current or recent attitude and behavior

What do you do if you get 10 millions JD a day?

Imagine you are married and have a 16 year old son who wants to give up his apprenticeship in order to become a professional footballer. Would you support him or would you advise him to complete his apprenticeship?

(9) Avoid double negative

Double negative is ordinary language are grammatically incorrect and confusing.

Examples: I am not got no job?logically means that the respondent does have

a job, but the second negatives used in this way for emphasis

Do you agree or not with changing your study plan or not?

Tell me whether you like or dislike having you exam online or not?

Do you agree or not with changing your study plan or not?

They arise when respondents are asked to agree or disagree with a statement. For example, respondents who disagree with the statement, "Students should not be required to take a comprehensive exam to graduate“ are logically stating a double negative because they disagree with not doing something.

(10) Avoid overlapping (unbalanced response categories)

Make response categories or choices mutually exclusive and exhaustive and balanced.

Mutually exclusive means that response categories do not overlap

Numerical ranges( e.g., 5-10, 10-20 , ..etc) should be as ( 5- 9, 10-19, …etc)

Exhaustive means that every respondent has a choice a place to go

"Are you working or unemployed?" leaves out respondents who are not working but do not consider themselves unemployed (e.g., full-time home makers, people on vacation, students, people with disabilities, retired people, etc.)

A case of unbalanced choices is the question What kind of job is the mayor doing: outstanding, excellent, very good, or satisfactory? Asking respondents to rate whether a mayor is highly, somewhat, or not very honesty isn't the same as asking them to rate the mayor's level of dishonesty

Which of the five candidates running for mayor do you favor: Eugene Oswego or one of the others?

Do you think the mayor is:very honest, somewhat honest, neither honestnor dishonest, somewhat dishonest, or very dishonest?Did you find the service at our hotel to be, Outstanding, Excellent, Superior, or good?

Please rate the service at our hotel:outstanding, very good ,inadequate, or poor.

Q. What is the composition of you household?1. One single adult 2. Two adults3. Two adults and one child4. Two adults with two children5. Two adults with three children6. One adult with one child

The categories are not exhaustive since there are several variations of adults and children possible

Q. What is the composition of you household?1. Number of adults …..2. Number of children ….