Survey methodology

ReferencesGroves R.M. et al. Survey methodology, Wiley-interscience, 2004Canada national statistical agency: http://www.statcan.gc.ca/Eurostat methodological publicationsEuropean Values Study (EVS)

A survey usually originates when an individual or institution is confronted with an information need and the existing data are insufficient A survey is a systematic method of gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of wich the entities are membersINTRODUCTION : What is survey methodology ?What is a survey ?Survey methodology is the study of survey methods

How survey work to produce statisticsRespondent answers to quenstionsInferenceCharacteristics of a respondentStatistical computingCharacteristics of the sampleCharacteristics of the populationInference

The life cycle of a survey

A survey from a process perspective

Define research objectivesChoose mode of collectionConstruct and pretest a questionnaireChoose sampling frameDesign and select sampleRecruit and measure sampleCode and edit dataMake postsurvey adjustmentsPerform analysis

The life cycle of a survey from a design perspective

ConstructMeasurementResponseEdited ResponseSurvey statisticsTarget PopulationSampling FrameSampleRespondentPostsurvey AdjustmentsThe Measurement dimension describes what data are to be collected about the observational units in the sampleThe Representational dimention concerns what population are described by the surveyWhat is the survey about?Who is the survey about?

The measurement dimension

Constructs are the elements of information that are sought by the researcher :How many incidents of crimes with victims there were in the last year;The consumption of beer in the last month;The degree of knowledge of mathematics of childrensMeasurements are ways to gather information about constructs :Questions posed to a respondent (During the last 6 month, did you call the police to report something that happened to you that you thought was a crime?)NB: the critical task for maesurement is to design questions that produce answers reflecting perfectly the construct we are trying to measure.Response could be produced in a variety of meansBut in general the nature of the response is determined by the nature of the measurementEditing of data may examine the full distribution of answers and look for atypical patterns of responsesEdited responses are the data from wich inference is made about the values of the construct for an individual respondentConstructMeasurementResponseEdited Response

The representational dimension

The target population is the set of unit to be studiedThe adult population living in households in 2009;The frame population if the set of target population members that has chance to be selected into the survey sample :In a simple case it is a list of all units in the target population, but sometimes it is a set of units imperfectly linked to population members.i.e. a list of telephone numbers when the target population is the adult populationThe sample is the group from wich measurement will be sought. In many case it is a very small fraction of the the sampling framePostsurvey adjustments consist on weighting up the underrepresented groups in order to improve the survey estimateBecause of mismatches of the sampling frame and the target population (coverage problems) statistics based on the respondents can differ from caracteristics of the target population. Examination of non response patterns may suggest an underrepresentation of some groupes relative to the sampling frameTarget PopulationSampleRespondentPostsurvey AdjustmentsSampling FrameRespondents are the elements successfully measures. Non respondents is the complement

The life cycle of a survey from a quality perspective

ConstructMeasurementResponseEdited ResponseSurvey statisticsTarget PopulationSampling FrameSampleRespondentPostsurvey AdjustmentsValidityMeasurement errorProcessing errorCoverage errorSampling errorNonresponse errorAdjustmentserror

Coverage of a target population by a frame

Frame populationTarget populationUndercoverageElements in the target population missing from the framei.e.:non telephone household, using a telephone frame to cover the full household populationCovered populationIneligible unitsIneligible unitsElements in the frame that are no member of the target populationi.e.:business telephone numbers, using a telephone frame to cover the full household populationUndercoverage

Evaluating survey questions:Are the answers good measures of the intended construct?Exemple of methods that can be used to evaluate draft survey questions Expert reviews

The substantive expert review the wording, the order and the structure of questions, the response alternatives etc. A small number of target population participate in a systematic discussion about the survey topic. The researcher learn about the nomenclature of the concept, the common perspective taken by the target population on key issues etc Focus groups

Questionnaire pretest

Researcher test how questions are read and answered. A behaviour coding is often used

Evaluating survey questions:Interview Questioning Behavior (choose one)Reads questions exacty as wordedReads questions with minor changesReads questions so that meaning is altering

Respondent Behaviors (check as many as apply)Interrupt question readingAsks for clarification of questionGives adequate answerGives answer qualifies about accuracyGives answer inadequate for questionsAnswers dont knowRefuses to answerExemple of behavior codes for interviewer and respondent behaviors

Methods of data collection

Traditional data collection methods

Mailing paper questionnaires to respondents, who fill them out and mail them backHaving interviewers call to respondents on the telephone and ask them the question in a telephone interview Sending the interviewers to the respondents home or office to administer the questions in face-to-face (FTF) interviews

Alternatives methods of data collectionFace to faceTelephoneMailCATIcomputer assisted telephone interviewingCAPI computer assisted personal interviewingTDETouchtone data entryOCR/ICROptical/intelligent caracter recognitionFAXDisk by MailE-mailWebComputerised Self Administered QuestionnairesIVRInteractive voice responseSAQSelf administered questionnaireWalkmanText CASIAudio CASIVideoCASI

Alternatives methods of data collection (a)

OCR/ICROptical/intelligent caracter recognitionFAXDisk by MailE-mailWebMail

Alternatives methods of data collection (b)

CATIcomputer assisted telephone interviewingTDETouchtone data entryIVRInteractive voice responseTelephone

Alternatives methods of data collection (c)Face to faceCAPI computer assisted personal interviewingSAQSelf administered questionnaireWalkmanText CASIAudio CASIVideoCASI

Chapter III

DESIGNING A QUESTIONNAIRE

QUESTIONS AND ANSWERS IN SURVEYS

A questionaire is a standardised set of questions administered to the respondents in a surveyRespondents are required to interpret a preestablished set of questions and to supply the information these questions seek.

The cognitive processes in answering questions

Comprehension of the questionRetrieval of informationJudgment and estimationReporting an answerA simple model of the survey response processResponses could be biased by acquiescence (the tendency to agree) social desirability (the tendency to present oneself in a favourable light by underreporting undesirable attributes and overreporting desirable one)

BUTRespondents often take shortcuts to get through the interview more quicklyORthey have motives that override their desire to provide accurate information

The cognitive processes in answering questions (2)

The satisficing model (Krosnik and Alwin, 1987) Some respondents try to satisfy (to take a low road answering more superficially) whereas others try to optimise (to take an high road by careful answering questions)Satisficing respondents do not seek to understand the question completely, just well enough to provide a reasonable answer

Problems in answering survey questions

Failure to encode the information sought

Misinterpretation of the questions

Forgetting and other memory problems

Estimation strategies

Problems in formatting answer

More or less deliberate misreporting

Failure to follow instruction

FORMATTING THE ANSWERSurvey items can take a variety of formats; the most common are:Open-ended qustions that call for numerical answersClosed questions with ordered response scalesClosed questions with categorial response options

1 - Open-ended qustions that call for numerical answers Now, thinking about your physical health, which includes physical illness and injury, for how many days during the past 30 was your physical health not good?Note that:Open-ended items yield more exact information than closed items

2 - Closed questions with ordered response scales

Would you say that in general your health is:ExcellentVery goodGoodFairPoorThe interviewer is instructed to please read the answer categories, but not the number attached to them!Note that :with some type of rating respondents seem to shy away from the negative end of the scaleWhen the scale points have numerical labels, the label can affect the answer (e.g. if respondents are asked to rate their success in life)

3 - Closed questions with categorial response options

Are you:MarriedDivorcedWidowedSeparatedNever marriedA member of an unmarried coupleNote that :The respondent may not wait to hear or read all the option; they may select the firs reasonable answer they consider (primacy effect)The opposite coul happen: the last option the interviewer read may be the first one that respondent think about (recency effect)

GUIDELINES FOR WRITING GOOD QUESTIONS (Sudman and Bradburn)Non sensitive questions about behavior The key problem with many questions about behavios is that respondents may forget some or all of the relevant information, or that their answer may reflect inaccurate estimate In order to reduce memory problems it is essential to play attention to the wording of the question and to provide memory helpAttitude questions Attitude questions are a very commen class of survey questions. The most frequent problems deals with the wording of questions, the question order and the format of response scales

Non sensitive questions about behavior Play attention to the wording Are you:MarriedDivorcedWidowedSeparatedNever married

With closed questions, include all reasonable possibilities as explicit response optionsAre you:MarriedSingle

Non sensitive questions about behavior Play attention to the wording In a tipical week, how often do you read a newspaper?Make the question as specific as possible (about who it covers, what time period, which behaviours) Over the last month, that is .. how often do you read a newspaper in a tipical week?

Non sensitive questions about behavior Play attention to the wording

Use words that virtually all respondents will understand Have you ever had a miocardial infarction? Have you ever had a heart attack?

Non sensitive questions about behavior Provide memory help

Uses aided recall (or ask separate questions about subcategories) Please look cerefully at the following list of volountary organisations: which, if any, do you belong to?

AReligious organisationsBCultural organisationsCPolitical groupsDOther To which volountary organisation do you belong to?

Attitude questionsPlay attention to the wording

Clearly specify the attitude object of interest

Do you think the Government is spending too litte, about the right amount, or too much on education? Do you think the Government is spending too litte, about the right amount, or too much on higher education?

Attitude questions* play attention to the wording

Measure the strength of the attitute using a response scale, a separate item or multiple items that can be combined into a scale

*note that an attitude have generally a direction (agree or disagree) and an intensity (strongly disagree.strongly agree) Do you agree or disagree with the following statement?Government is spending too little on education1Agree strongly2 Agree3Neither agree nor disagree4Disagree5Disagree strongly

Attitude questions reduce impact of question order

When asking general and specific questions about a topic, ask the general question first (otherwise, the answer to the general question is likely to be affected by the number and content of specific questions)

When asking questions about about multiple items, start with the least popular (the unpopular questions are likely to seem even less appealing when they follow more popular questions)

When asking general and specific questions about a topic, ask the general question first Please tell me whether or not you think it should be possible for a pregnant woman to obtain a legal abortion if: the woman wants it for any reason? 1. Yes 2. No 3. Don't know Please tell me whether or not you think it should be possible for a pregnant woman to obtain a legal abortion if: there is a strong chance of a serious defect in the baby? 1. Yes 2. No 3. Don't know USA GSS

Attitude questions play attention to the response scale

Use closed questions for measuring aptitudes;(open answers are difficult to code)

Use five-to seven-point response scales and label every scale point(verbal label ensure that interpret the scale in the same way)

Now is up to you!