Research Methods —Part III. Research Methods Survey

Research Methods — Part III. Research Methods

Survey Research Methods & Sampling林芳如

2021/10/13

Learning Objectives• To describe survey research process

• To describe types of information sought in survey research

• To understand how to write survey questions properly

• To compare and contrast major methods of survey administration

• To describe types of sampling strategies

2

Types of Quantitative Study Design (Review of previous lectures)

3

• Cross-sectional• Group comparison• Longitudinal

• Cross-sectional• Case-control• Retrospective cohort • Prospective cohort

What is survey?• Systematically collect information on

a topic by asking individuals questionsØ Purpose: to generate statistics on

the group or groups that those individuals represent

Survey data collection—Primary or secondary data?

http://www.hpa.gov.tw/Pages/List.aspx?nodeid=64

4

Example of National Health Surveys http://nhis.nhri.org.tw/

5

Steps in Designing & Conducting a Survey• Thinking about topics for health survey

• Matching the survey design to survey objectives

• Defining and clarifying survey variables (may involve focus group)

• Choosing the methods of data collection

• Formulating the questions: choose question & answer types, draft questions, check wording

• Formulating the questionnaire: Check order, format layout

• Questionnaire pretest

• Drawing the sample

• Monitoring and carrying out the survey

• Preparing data and analysis

• Writing research report

6

Framework for Classifying Topics in Health Survey

Type of Information Collected in SurveyInformationcategory

Description Question examples

Behavior What do people do? Have you personally experienced an abortion?

Knowledge What do people know? Are you aware that abortion can lead to inability to have children in the future?

Attitude What do people think? Do you think abortion in the first trimester should be legal for all women?

Beliefs What people strongly think or believe is true?

Do you believe abortion is murder?

Attributes What people are? (e.g. demographics)

Are you male or female? What is your age?What is your religion if you have one?

8

Study Variables• Demographics• Independent variables (e.g. exposure)• Dependent variables (e.g. diseases/ outcomes)• Related factors or potential confounders

e.g. Survey about risk factors for breast cancer:Sociodemographics (age, race, education level, income)Family health historyPersonal health history (previous diagnoses, last screening mammogram)Reproductive history (gravidity, menstrual cycles, hormone use)Lifestyle factors (alcohol, exercise, working the night shift)

9

• Whenever possible, use of a previously validated instrument without modification is preferable to developing a new instrument or modifying questions

→ Ensures that the questionnaire is reliable, valid, and responsive to change• Modifying an established instrument may reduce its validity and reliability, although

it may be necessary in some instances

10

Developing a Survey Questionnaire• Questionnaire:

– Tool for systematically gathering information from study participants– Can be designed for self-reporting or as scripts for interviews

• Rucker’s Law:“If one asks the wrong question,

the odds are very high that one will receive the wrong answer”• Cognitive testing:

– Ensures that a survey question successfully captures the scientific intent of the question and makes sense to respondents

Always remember to ensure—Reliability & validity!!!

11

First, let’s understand possible reasons for suboptimal accuracy

• Communication: respondents do not understand the question

• Knowledge: lack of knowledge to answer the question

• Memory: cannot recall information

• Motivation: do not want to report the information

12

Types of Questions• Close-ended questions:

– Allow limited number of answers– Usually easier to statistically analyze then open-ended questions– May force respondents to select answers that do not truly express their status

or opinions• Open-ended questions:

– Allow participants to explain their selections and qualify their responses, to give multiple answers, to provide responses not anticipated by the researchers

– Take longer to ask and answer– Often most useful when being used to get initial impressions or to clarify

responses to close-ended questions– May result in irrelevant answers

13

Type Sample question Response options for the sample question

Date What is your birth date? __ __ - __ __ - __ __ __ __m m - d d - y y y y

Numeric What is your height? __ __ __ cm

Yes/no During your lifetime, have you smoked more than 100 cigarettes?

□ Yes □ No

Categorical/ multiple-choice: nominal (no rank)

What is your favorite type of film? □ Action/drama□ Comedy/musical□ Documentary□ Other: ____________

Categorical/ multiple-choice: ordinal (ranked)

What is the highest level of education you have completed?

□ Less than high school□ High school□ Some college but no degree□ College degree or more advanced

Paired-comparisons Do you prefer to drink coffee or tea? □ I prefer coffee□ I prefer tea□ I like coffee and tea equally□ I do not drink coffee or tea

Rank-ordering List the following four political issues in order from most important to you (1) to least important to you (4): crime/safety, environment/energy, foreign policy/defense, taxes/revenue

Number from 1 to 4:___ Crime/safety___ Environment/energy___ Foreign policy/defense___ Taxes/revenue

Open-ended/free-response

What is your biggest personal health concern at present?

______________________________________________________________________

More on Types of Response• For categorical response questions:

– Sometimes an “other” category should be included• For ranked response questions:

– Whether there will be a neutral option (usually 5 or 7 categories)• More response options to provide:

– For self-reported survey: “not applicable”, “do not know”– For questionnaires to be used as scripts: “refuse to answer”– Others: “no opinion”, “not sure”, “hard to say”, “no answer”, “I forget”

15

Wording of QuestionsCheck the questions—• Does each question ask what it is intended to ask?• Is the language of each question clear and neutral? • Will members of the study population understand the language? • Is the question sensitive to potential cultural issues related to language?Check the responses—• Is the choice of response clear?• For scaled questions, is the rank order clear? • For questions with unranked categories, is the order of possible responses

alphabetical or otherwise neutral?

16

Problem with questions ExampleBig words/jargon Have you ever had a myocardial infarction?

Undefined abbreviations Have you ever been told that you have BPH?

Vague, ambiguous words What kind of house do you live in?Do you exercise regularly?Do you read New England Journal of Medicine?

Double negatives I do no find this visit with my doctor to be unpleasant.□ Disagree □ Neutral □ Agree

Faulty assumptions Do your gums bleed during regular dental cleanings?□ Yes □ No

Two-in-one/ Double-barreled questions

Do you exercise at least 3 times a week and eat a healthy diet?

Impossible to recall accurately

How many servings of carrots did you eat most weeks when you were a child?

Too much detail List any prescription medications you have taken for 1 month or longer in the past 10 years.

Problems with Questions to Avoid (1)

17

Problem with questions ExampleSensitive questions Have you ever hit, scratched, bruised, or otherwise physically

injured an intimate partner?

Hypothetical questions Have you ever thought that you would like to lose 10 or more pounds?

Leading questions What is your impression of the quality of work done by the dedicated public servants who work at the county health department?

Leading answers What is your impression about the quality of services provided by NTUH?□ Fair □ Good □ Great □ Excellent

Answer with a poor scale How many hours a week do you watch television?□ 0 □ 1-3 □ 4-7 □ 8 or more

Lack of specificity What is your income?

Missing answer options What color are your eyes?□ Black □ Brown

Overlapping answer options

In a typical week, how many days do you eat fish?□ 0 □ 1-3 □ 3-5 □ 5-7

Problems with Questions to Avoid (2)

Other Issues to Consider (1)• Anonymity

– Must decide which question type is most appropriate and will best protect participants’ anonymity and privacy

– Example: several ways to ascertain participants’ age—ask for date of birth, current age in years, age range they belong to

• Order of Questions– Start with easy or at least general questions– In a order that flows naturally from one topic to another– To group similar questions with similar response types

vs. to mix up such questions to prevent habituation

19

Other Issues to Consider (2)• Layout & Formatting

– Organized, easy to read, easy to record answer on• Indicate where and how responses should be marked• Use white space to separate sections• If necessary, with very clear instructions for skips

– For a self-reported written questionnaire, include a cover letter and instructions about how to indicate answers• Examples:

– “Select one answer that best describes you.”– “Circle all options that apply to you.”– “Write your answer in capital letters, as shown in the example below.”

20

Pretest & Pilot Testing• Both to examine how sample of people respond to a questionnaire

(before full extension to the study population)

• Pretest:– Used to test and refine questions– To ascertain if instructions clear, questions clear, whether questions can be

answered, approximate length of time necessary– A minimum standard, even if cannot do pilot testing

• Pilot Testing:– Ideally sample larger group so that possible within and across-group

differences can be analyzed

21

Mode of Survey Administration (Data collection methods)

• Two broad categories: written questionnaire & interview

• Considerations & general questions to ask:– Which method is most appropriate, given the study question and population of interest?p Costs (e.g., budget? require trained personnel?)p Responsep Timep Efficiency

Example: Ø Low-income countries (accessibility to telephone or internet, literacy)Ø Employee survey (confidentiality of responses)Ø Student projects (low budget and no staff available)

Interviewing Self-Administered Questionnaire

Paper-and pencil (including mail)

Electronic (web-based & E-mail)

Personal(face-to-face) Telephone

22

Comparing the data collection methodsPersonal surveys

Telephone surveys

Mail surveys Electronic surveys (web-based & E-mail)

Costs

Data collection period

Sampling limitation

Require level of literacy

Question complexity

Social desirability bias

Response rate

Item-completion mistake

Data collected easily imported for analysis

Why Sampling?

Population

Sample

[Inference]Generalize conclusions from the sample to the

population

Select a sample from the population

Difference between sample vs. census—Sample: representative fraction of populationCensus: every member of the population

25

Types of Research Populations

Target populationThe general population that the study seeks to

understand

Source population(sampling frame)

The specific individuals from which a

representative sample will be drawn

Sample population Individuals asked to

participate

Study populationEligible participants

26



understand





participate


Representative

27



understand





participate


Sampling

28

Probability Sampling Technique

N = the number of cases in the sampling framen = the number of cases in the samplef = n/N = the sampling fraction

Say that we want to randomly select n patients from a source population of N

• Probability sampling: every individual in the population has a knownchance of being selected

Sampling frame: the list of the source population from which the sample will be drawn

29

Probability Sampling TechniqueSimple Random Samplingv Each individual in the source population has an equal chance of being selected

for the sample population• Procedure: Use a table of random numbers, a computer random number

generator, or a mechanical device to select the sample• Limitation:

• Not the most statistically efficient method of sampling• May not get good representation of subgroups in a population

How to Create a Random Sample

in Excel

30

Probability Sampling TechniqueSystematic Sampling• After a random start point, every kth person is selected

• k = N/n = the interval size• Advantage: easy to do• The units in the population need to be randomly

ordered, at least with respect to the characteristics you are measuring

http://www.socialresearchmethods.net/kb/sampprob.php

Probability Sampling TechniqueStratified Samplingv Divide population into homogeneous strata and draw random sample

separately from all strata

• Proportionate: same sampling fraction in each stratum• Disproportionate: different sampling fraction in each stratum

4

1

3

9 12

2 6

87 11

5 10

Proportionate stratified sampling

1

9 12

8

711 5

sample

32

Example of disproportionate sampling: oversampling of American Indians

Probability Sampling TechniqueCluster Sampling

http://essedunet.nsd.uib.no/cms/topics/weight/2/6.html

v Single-stage sampling/ cluster sampling: Divide total population into heterogeneous clusters and select a random sample of the clusters

• Difference from stratified sampling: use the entire cluster as a sample• Example: an area is divided into geographic clusters and some clusters are

selected for inclusionv Multi-stage sampling:

Divide population into heterogeneous clusters and draw random sample separately from all sample of clusters

https://www.hpa.gov.tw/Pages/List.aspx?nodeid=3998

34

35

http://nahsit-form.ibms.sinica.edu.tw/node/7

Probability Sample Design

Advantages Disadvantages

Simple random sampling

• Requires little knowledge of population in advance

• May not capture certain groups of interest

• May not be very efficient

Systematic sampling

• Easy to analyze data and compute sampling (standard) errors

• High precision

• Periodic ordering of elements in sample frame may increase biases

• May not capture certain groups of interest

• May not be very efficient

Stratified sampling

• Enables certain groups of interest to be captured

• Enables disproportionate sampling and optimal allocation within strata

• High precision

• Requires knowledge of population in advance

• May introduce more complexity in analyzing data and computing sampling (standard) errors

Cluster sampling • Lowers field costs• Enables sampling of groups of

individuals for which detail on individuals themselves may not be available

• Introduces more complexity in analyzing data and computing sampling (standard) errors

• Lowest precision

36

Non-probability Sampling TechniqueDoes not involve random selection• Convenience sampling

– Based on ease of access to those individuals (simple and inexpensive)– High risk of creating sampling bias (occur if the sample is not

representative of the source or target population)

• Quota sampling• Snowball sampling• Judgmental sampling (or purposive sampling)

37



understand





participate


Participation/Response rate

38

What is response rate?• The proportion of people selected and deemed eligible for the study that

actually completed the questionnaire or interview

Situations where individuals selected for sample do not provide data:• Survey never reached intended respondent• Refusals• Incapacity (e.g. lack of literacy)

39

Less than 100% participation rate is usually not a problem if:

• Uses suitable and carefully explained sampling methods• Takes appropriate steps to maximize the participation rate• Recruits an adequately large sample size• Reports the number of potential participants at each stage

Non-response bias: if the members of the same population who agree to be in the study are systematically different from non-participants

http://rampages.us/benliam1/2015/07/05/24-hour-news-and-social-media-can-threaten-internal-validity/ 40

Use weights in analysis of survey data• Weights

– A value assigned to each case– Normally used to make statistics more representative of population

• Two most common types– Design weights: normally used to compensate for over- or

under-sampling of specific cases or for disproportionate stratification

– Post-stratification or non-response weights: used to compensate for that fact that persons with certain characteristics are not as likely to respond to the survey

41

References• Jacobsen KH. Introduction to health research methods: a practical guide. Sudbury,

MA: Jones & Bartlett Learning; 2012.

• Aday LA, Cornelius LJ. Designing and conducting health surveys: a comprehensive guide. 3rd ed. San Francisco, CA: Jossey-Bass; 2006.

• Dillman DA. Mail and telephone surveys: the total design method. New York, NY: Wiley; 1978.

• Brasel K, Haider A, Haukoos J. Practical guide to survey research. JAMA Surg2020;155(4):351-352.

Documents

Research Methods —Part III. Research Methods Survey