Survey Design and Data Collection Most Of the Error In Surveys Comes From Poorly Designed...

Survey Design and Survey Design and Data CollectionData Collection

Most Of the Error In Surveys Comes From

Poorly Designed Questionnaires And Sloppy Data Collection Procedures

Questionnaire Design

• Preliminary Decisions– What information is required?

– Who are the target respondents?

– How will we communicate with the respondents?

• Decisions About Question Content

– Is the question needed?

– Is the question sufficient?

– Can the question be answered?

– Will the question be answered?

– Are there external events that could influence response?

Questionnaire Design

• Decisions Concerning Question Phrasing – Do the words have the same

meaning to all respondents?– Are the words misleading?– Will the respondents adopt the

same frame of reference as the researcher?

• Decisions About Format Response – How should the respondents be

allowed to answer?

Questionnaire Design

• Decisions About Question Sequence– Does the question order make

sense?

• Decisions About Layout– Is the questionnaire easy to read

and answer?

• Pretest and Revise

Nonsampling Error

• In Terms of Total Survey Error Nonsampling Error:

– Is the Most Serious

– Is the Hardest to Detect

– Is the Toughest to Fix

Total Error

• Sampling Error• Increasing

Sample Size (n) Will Decrease

• Improved Sampling Procedure Will Decrease

• Probabilistic Bounds Can Be Placed on Our Level of Precision

• Nonsampling Error

• Increasing Sample Size (n) May Increase

• Improved Research Methods Will Decrease

• Difficult to Determine the Magnitude

Nonsampling Error

• Nonobservation

– Noncoverage• Sampling

Frame Doesn’t Match Population

• Nonresponse• Respondents

Do not Provide the Information You Need

• Observation

– Field • Information Is

Not Valid (Poor Questionnaire, Bad Interviewing, Misrepresentat

ion)

– Office• Survey

Processing

Sloppy

Noncoverage Error: How To Correct

• Think Hard About Who Our Population Is- This Isn’t As Easy As It Seems

• Search For A Complete Sampling Frame

• Fix the Sampling Frame We Have

Nonresponse Error: How To Correct

• Making Contact: Continue to Make Contact With Sampled Respondents (e.g. 5 call backs).

• Obtaining Cooperation: Respondents Must be Approached in a Professional Manner.

• Obtaining Desired Information: Questionnaire and/or Interview Must Appear Reasonable, Understandable, and Confidential.

Field Error: How to Correct

• The Entire Data Collection Process Must Be Thought Through Carefully.

• Devote Time to Developing Survey Instrument.

• Train All People Connected With the Research Project.

• Pretest Procedures

Office Errors: How To Correct

• Conduct Checks in the Field to Head Off Potential Problems Early.

• Check for Deletions, Inconsistencies, and Wrong Answers.

• Use Closed Ended Questions if Possible.

• Use Same Coder Per Question

• Use Statistical Procedures to Catch Mistakes.