Survey Design and Survey Design and Data CollectionData Collection

Most Of the Error In Surveys Comes From

Poorly Designed Questionnaires And Sloppy Data Collection Procedures

Questionnaire Design

• Preliminary Decisions– What information is required?

– Who are the target respondents?

– How will we communicate with the respondents?

• Decisions About Question Content

– Is the question needed?

– Is the question sufficient?

– Can the question be answered?

– Will the question be answered?

– Are there external events that could influence response?

Questionnaire Design

• Decisions Concerning Question Phrasing – Do the words have the same

meaning to all respondents?– Are the words misleading?– Will the respondents adopt the

same frame of reference as the researcher?

• Decisions About Format Response – How should the respondents be

allowed to answer?

Questionnaire Design

• Decisions About Question Sequence– Does the question order make

sense?

• Decisions About Layout– Is the questionnaire easy to read

and answer?

• Pretest and Revise

Nonsampling Error

• In Terms of Total Survey Error Nonsampling Error:

– Is the Most Serious

– Is the Hardest to Detect

– Is the Toughest to Fix

Total Error

• Sampling Error• Increasing

Sample Size (n) Will Decrease

• Improved Sampling Procedure Will Decrease

• Probabilistic Bounds Can Be Placed on Our Level of Precision

• Nonsampling Error

• Increasing Sample Size (n) May Increase

• Improved Research Methods Will Decrease

• Difficult to Determine the Magnitude

Nonsampling Error

• Nonobservation

– Noncoverage• Sampling

Frame Doesn’t Match Population

• Nonresponse• Respondents

Do not Provide the Information You Need

• Observation

– Field • Information Is

Not Valid (Poor Questionnaire, Bad Interviewing, Misrepresentat

ion)

– Office• Survey

Processing

Sloppy

Noncoverage Error: How To Correct

• Think Hard About Who Our Population Is- This Isn’t As Easy As It Seems

• Search For A Complete Sampling Frame

• Fix the Sampling Frame We Have

Nonresponse Error: How To Correct

• Making Contact: Continue to Make Contact With Sampled Respondents (e.g. 5 call backs).

• Obtaining Cooperation: Respondents Must be Approached in a Professional Manner.

• Obtaining Desired Information: Questionnaire and/or Interview Must Appear Reasonable, Understandable, and Confidential.

Field Error: How to Correct

• The Entire Data Collection Process Must Be Thought Through Carefully.

• Devote Time to Developing Survey Instrument.

• Train All People Connected With the Research Project.

• Pretest Procedures

Office Errors: How To Correct

• Conduct Checks in the Field to Head Off Potential Problems Early.

• Check for Deletions, Inconsistencies, and Wrong Answers.

• Use Closed Ended Questions if Possible.

• Use Same Coder Per Question

• Use Statistical Procedures to Catch Mistakes.