Survey Research: TTM531 Class Presentation
1
SURVEY RESEARCHSURVEY RESEARCH
TTM531 Fall 2001 Class Presentation
Nancy WhelchelCoordinator for Survey ResearchUniversity Planning and Analysis
515-4184http://www2.acs.ncsu.edu/UPA/survey/survey_research/
Survey Research: TTM531 Class Presentation
2
Major TasksMajor Tasks
ID Purpose of Research Decide on Survey Design and Format Select Sample Design Survey Instrument Pre-test Instrument Collect Data Analyze Data Report Findings
Survey Research: TTM531 Class Presentation
3
Purpose of Your SurveyPurpose of Your Survey
Ask Yourself: WHY do I want to/need to do a survey?
WHAT do I want to learn?
WHO is my audience?
HOW will I use the information I gather?
Survey Research: TTM531 Class Presentation
4
Purpose of Your SurveyPurpose of Your Survey
Our Example: Why =
President of NCTextiles.com, where you work, told you to. What want to learn =
How does employee satisfaction and productivity compare between NCTextiles.com telecommuters and non-telecommuters.
Who is audience =Company president and executive board.
How use =Set new policies for telecommuting.
Survey Research: TTM531 Class Presentation
5
Purpose of Your SurveyPurpose of Your Survey Description
Discover the distribution of certain traits or characteristics
(e.g., #telecommuters, $revenue generated)
Explanation Understand why something happens
(e.g., is there a difference between telecommuters and non-telecommuters in job satisfaction and in productivity)
Exploration Help inform a research design
Survey Research: TTM531 Class Presentation
6
Think About Available Think About Available ResourcesResources
What information is already available from non-survey sources?
Have any surveys already been done on this question?
Who is going to:• Design the survey• Administer the survey• Enter the data• Analyze the data• Write up the results• Present the findings
Survey Research: TTM531 Class Presentation
7
Think About Available Think About Available ResourcesResources
Should I hire a professional???
What resources/skills are available in-house?
What types of decisions are going to be made based on your findings?
How much money do you have for the survey?
Survey Research: TTM531 Class Presentation
8
Basic Survey DesignsBasic Survey Designs Cross-sectional
• Data collected at one point in time from a sample selected to describe a larger population
Longitudinal• Trend
• Different samples from a general population studied at different points in time.
• Cohort• Same specific population studied over time (may be
different samples).• Panel
• Same sample studied over time.
Survey Research: TTM531 Class Presentation
9
Basic Survey FormatsBasic Survey Formats
Format determines much of research design (e.g., sampling, questionnaire design, administration, data analysis, etc.)
Basic formats include:• Self-administered mail survey• Telephone interview• Face-to-face interview• Focus group• Self-administered Web survey• Self-administered Email survey
Survey Research: TTM531 Class Presentation
10
The Survey SampleThe Survey Sample
WHO or WHAT do I want to survey?
Sampling allows you to study just a portion of the population/group and make estimated assertions about the population from the sample.
Warning: Bad Sample = Bad Data
Survey Research: TTM531 Class Presentation
11
Why Sample?Why Sample?
TimeShorter time in the field decreases the chance that external factors will influence results.
QualityDoing fewer surveys increases control over the entire process (interviewing, data entry, etc.) and therefore decreases error
Cost
Survey Research: TTM531 Class Presentation
12
What is a Sample?What is a Sample?
Some definitions: Element or Sampling Unit
Unit about which information is collected (e.g., employee)
Survey Population
Aggregate of all elements interested in from which sample is selected (must be clearly defined)
(e.g., NCTextiles.com non-executive employees with job tenure 3+ months )
Survey Research: TTM531 Class Presentation
13
What is a Sample?What is a Sample?
Some more definitions: Sampling Frame
Actual physical source listing elements in the survey population
e.g., Edited list from company employee directory . Sample Population
Collection of elements selected from the survey population sampling frame
Survey Research: TTM531 Class Presentation
14
What is a Sample?What is a Sample?
A Warning: Findings from a survey are only
representative of the aggregate of elements in the sampling frame.
In other words…You can’t use your results to say anything
about the folks who didn’t have a chance to be included.
Survey Research: TTM531 Class Presentation
15
Sample SizeSample Size
Depends on: Cost Level of precision want (“margin of error”) Level of confidence want Anticipated response rate Number of responses needed per
subgroup interested in
Survey Research: TTM531 Class Presentation
16
Sample SizeSample Size
Example: Margin of Error +/- points confidence interval 80% 90% 95% 5 150 275 400 10 40 70 100
If 45% of 400 respondents agree that NCTextiles.com provides “excellent” health benefits, this means that 95 out of 100 times between 40% (45%-5) and 50% (45%+5) believe the company provides “excellent” health benefits.
Source: McClelland, Lou (1993), “Survey Research Mechanics in Three Checklists,” (working paper). Office of Research and Information: University of Colorado at Boulder.
Survey Research: TTM531 Class Presentation
17
Selecting a SampleSelecting a Sample
Probability Sample• A representative sample
• Everyone in the sample population has an equal chance of getting in the sample.
• Results can be generalized to the sample population.
Non-Probability Sample• Non-representative sample
• Purposive, quota, who’s available• Results can not be generalized beyond the
sample.
Survey Research: TTM531 Class Presentation
18
Probability SamplingProbability Sampling
Simple random sample• List the elements in the sampling frame and
use a random number chart to select elements.
Systematic sampling• List the elements in the sampling frame and
take every Nth element, based on how many needed in sample.• Warning: Be attentive to any inherent
patterns in how sampling frame is ordered.
Survey Research: TTM531 Class Presentation
19
Probability SamplingProbability Sampling
Stratified sampling• Draw appropriate proportion of elements
from homogeneous groups interested in (based on theory).
• Note: Think about using disproportionate sampling then “weighting” the data. E.G., If there are relatively few numbers in one sub-group you are interested in, include proportionally more of them in your sample, then “weight” the data to reflect their true proportion in your population.
Survey Research: TTM531 Class Presentation
20
Probability SamplingProbability Sampling
Stratified sampling continued
An Example:
250 out of your 2000 employees in the sample population telecommute (12.5% of sample population).
With stratified sampling with sample size of 800, would have 100 telecommuters (800 * .125), and 700 non-telecommuters (800 * .875).
Survey Research: TTM531 Class Presentation
21
Probability SamplingProbability Sampling
Stratified sampling continued
An Example continued:BUT, you might want to “oversample”
telecommuters (even include all 250) to be able have less error in their results, and “undersample” non-telecommuters (e.g., 550). When analyzing all results, “weight down” telecommuters to reflect their true proportion in the sample population, and “weight up” non-telecommuters.
Survey Research: TTM531 Class Presentation
22
Instrument DesignInstrument Design
Start with: Conceptualization
• Define the concepts you want to study
Operationalization• Specify indicators of your concepts.
• Make exhaustive list of everything the concept could include
Survey Research: TTM531 Class Presentation
23
Instrument DesignInstrument Design
ConceptualizationJob Satisfaction =
“A pleasurable or positive emotional state resulting from the appraisal of one’s job.”
Includes both extrinsic and intrinsic aspects.Has several dimensions:
• importance to employee• motivation of employee• current level of attainment • satisfaction with attainment
Survey Research: TTM531 Class Presentation
24
Instrument DesignInstrument Design
Extrinsic:• pay• promotion• supervision• co-workers• company• physical conditions• security• social status• work itself
Intrinsic:• variety of work• autonomy and freedom• responsibility• feedback• creativity• recognition• meaningfulness of work
Operationalization: Job Statisfaction
Survey Research: TTM531 Class Presentation
25
Questionnaire Questionnaire ConstructionConstruction
The way a question or statement is worded and the response options offered determine
the nature of the data received.
Bad question = Bad data
Survey Research: TTM531 Class Presentation
26
Questionnaire Questionnaire ConstructionConstruction
Some jargon… Questions need to be VALID
“…an empirical measure adequately reflects the real meaning of the concept under consideration.” (Babbie, 1990)
• face validity = the measure jibes with common agreement of what a concept entails.
• predictive validity = the measure accurately predicts behavior• content validity = the measure covers the range of meanings
included within the concept.
• construct validity = the measure relates to other measures
theoretically related to concept.
Survey Research: TTM531 Class Presentation
27
Questionnaire Questionnaire ConstructionConstruction
Some more jargon… Questions need to be RELIABLE
“…a particular technique, applied repeatedly to the same object, would yield the same result each time.” (Babbie, 1990)
Survey Research: TTM531 Class Presentation
28
Questionnaire Questionnaire ConstructionConstruction
Open-end Questions• Respondents write question response
in own words• probably need to enter data by hand• develop coding scheme for responses• content analysis ?
Suggestion: It’s a good idea to always include an open-end question giving the respondent the opportunity to add any additional comments they might have.
Survey Research: TTM531 Class Presentation
29
Questionnaire Questionnaire ConstructionConstruction
Closed-end or Forced-choice Questions• Respondent selects question answer
from among a list of possible responses
Suggestion: It’s a good idea to ask multiple indicators of a single concept, and then create a scale by combining those indicators.
Survey Research: TTM531 Class Presentation
30
Good Questions...Good Questions...
Good Questions... Are clear and use simple language Are concise Are specific Are possible to answer Are relevant to the respondent Do not use negatives Avoid biased terms Have only 1 part (not “double-barrel”)
Survey Research: TTM531 Class Presentation
31
Good Questions...Good Questions...
Use simple language...
My job is usually interesting enough to keep me from getting bored.
Is better than :
My job is usually engaging enough to permit me from reaching a banausic state of mind.
Survey Research: TTM531 Class Presentation
32
Good Questions...Good Questions...
Are concise...How interested would you be in a flexible work schedule in
which you would have to work an 8 hour day in the office, but would be able to arrive at work anytime between 6:00 and 10:00 AM and leave between 2:00 and 7:00?.
Is better than:
How interested would you be in being able to leave your house either before the morning rush hour or after the morning rush hour, and then arrive at the office not at the standard 8:00 AM time that is required according to the employee handbook, but at whatever time you got there, and then work a 8 hour day at the office (excluding and hour for lunch), and then leave the office so you are not commuting during rush hour but either before or after rush hour?
Survey Research: TTM531 Class Presentation
33
Good Questions...Good Questions...
Are specific...
In what year did you first start working at NCTextiles.com?
Is better than:
When did you start working here?
Survey Research: TTM531 Class Presentation
34
Good Questions...Good Questions...
Are possible for the respondent to answer...
NO:My contribution to the company has had a direct
impact on its ability to control a broader share in the global market.
Survey Research: TTM531 Class Presentation
35
Good Questions...Good Questions...
Are relevant...
NO:Polyester doesn’t breathe as well as natural fibers.
Survey Research: TTM531 Class Presentation
36
Good Questions...Good Questions...
Do not use negatives...
The lack of flexibility in my work hours makes it difficult for me to spend the amount of time I would like to with my family.
Is better than:
The lack of flexibility in my work hours does not make it possible for me to spend the amount of time I would like to with my family.
Survey Research: TTM531 Class Presentation
37
Good Questions...Good Questions...
Do not use biased terms or suggestions...
Do you agree or disagree with the following statement: All employees need to be in the office during regular
business hours.
Is better than:
Don’t you agree with the president of the company when she says that all employees need to be in the office during regular business hours?
Survey Research: TTM531 Class Presentation
38
Good Questions..Good Questions..
Do not ask two questions in one (“double- barrel”)...
I find real enjoyment in the work I do.I like the hours I keep at work.
Is better than:
I find real enjoyment in the work I do and the hours I keep.
Survey Research: TTM531 Class Presentation
39
Response OptionsResponse Options
Response options should reflect the concepts you are trying to measure, and fit with the question wording.
Suggestion: Avoid simple “yes” or “no” type responses and try to measure intensity of feeling when possible.
Survey Research: TTM531 Class Presentation
40
Response OptionsResponse Options
Must be: Mutually exclusive
can select only one appropriate answer
Example: NOT mutually exclusive...Q. On most days, what time do you leave your house
for work?6:00 AM or earlier 7:30-8:00 AM6:00-6:30 AM 8:00-8:30 AM6:30-7:00 AM 8:30-9:00 AM7:00-7:30 AM 9:00 AM or later
Survey Research: TTM531 Class Presentation
41
Response OptionsResponse Options
Mutually exclusive continued
Example: IS mutually exclusive...Q. On most days, what time do you leave your house
for work?Before 6:00 AM 7:30-7:59 AM6:00-6:29 AM 8:00-8:29 AM6:30-6:59 AM 8:30-8:59 AM7:00-7:29 AM 9:00 AM or later
Survey Research: TTM531 Class Presentation
42
Response OptionsResponse Options
Must be: Exhaustive
all possible answers are listed (including e.g., “other, “ “don’t know,” etc.)
Example: NOT exhaustiveQ. On most days, what time do you leave your house
for work?6:00-6:29 AM 7:30-7:59 AM6:30-6:59 AM 8:00-8:29 AM7:00-7:29 AM 8:30-8:59 AM
Survey Research: TTM531 Class Presentation
43
Response OptionsResponse Options
Exhaustive continued
Example: IS exhaustiveQ. On most days, what time do you leave your house
for work?Before 6:00 AM 7:30-7:59 AM6:00-6:29 AM 8:00-8:29 AM6:30-6:59 AM 8:30-8:59 AM7:00-7:29 AM 9:00 AM or later
Survey Research: TTM531 Class Presentation
44
Question OrderQuestion Order
Be attentive: Initial questions effect answers to subsequent ones.
Ask less interesting questions (e.g., demographics) at end of self-administered survey and beginning of interview survey.
Survey Research: TTM531 Class Presentation
45
Instructions for Instructions for RespondentRespondent
Provide general instructions in beginning of self-administered survey• Brief explanation could include:
• why doing survey• why answers are important• how to answer questions• stress confidentiality (if appropriate)
• Do NOT ask permission - - assume respondent will want to participate.
Provide specific instructions to questions as necessary
Survey Research: TTM531 Class Presentation
46
Instructions for Instructions for RespondentRespondent
ExampleInstructions: The following statements are about some
features of your job. Please tell us how satisfied or dissatisfied you feel with the features of your present job by circling one of the response options listed below each statement.
There are no right or wrong answers. All replies are confidential so give your honest opinion on each of the statements. Please do not discuss your answers with other people. It is your opinion which matters.
Even though it may be hard to decide, please be sure to answer all questions. Thank you for your help.
Survey Research: TTM531 Class Presentation
47
Questionnaire FormatQuestionnaire Format
Should be visually attractive and nicely reproduced
Should be spread out and uncluttered• better to have more than 1 page than
cramped page Should be broken into logical sections
when possible Should have clear skip patterns for
contingency questions
Survey Research: TTM531 Class Presentation
48
Questionnaire FormatQuestionnaire Format
Should have clear spaces for respondents to mark answers• use boxes • parentheses ( X ) • or numbers to circle • avoid lines to put check on X_
X
1
Survey Research: TTM531 Class Presentation
49
Questionnaire FormatQuestionnaire Format
Helpful hint….
Be sure to seek advice on question wording, response options, and questionnaire format from those who will be doing the data entry and data analyses.
Survey Research: TTM531 Class Presentation
50
Pre-TestPre-Test
Test the survey instrument before doing the actual study.
Suggestion: It’s also a good idea to test out your sample design, data collection methods, data processing, and data analysis if possible.
Survey Research: TTM531 Class Presentation
51
Pre-Testing the Survey Pre-Testing the Survey InstrumentInstrument
Use respondents similar to those who will be in survey sample.
Test:• Question clarity
• failure to answer?• multiple answers?• “other” answers?• qualified answers?
Survey Research: TTM531 Class Presentation
52
Pre-Testing the Survey Pre-Testing the Survey InstrumentInstrument
Test continued:• Questionnaire format
• instructions?• flow?• layout?
• Variance in responses (if interested in relationships among variables…)
• do most respondents give the same answer to a question?
Survey Research: TTM531 Class Presentation
53
Pre-Testing the Survey Pre-Testing the Survey InstrumentInstrument
Revise survey instrument as necessary Pre-test again if possible
(ideally with some of original pre-test respondents, and some new respondents)
Survey Research: TTM531 Class Presentation
54
Data CollectionData Collection(Self-administered mail survey)(Self-administered mail survey)
Include: Survey instrument Return envelope
• self-addressed and stamped(either postage-paid or 1st class stamp)
Cover letter• on letterhead paper• signed by most important person possible
Survey Research: TTM531 Class Presentation
55
Data CollectionData Collection(Self-administered mail survey)(Self-administered mail survey)
Cover letter (continued)• brief explanation of
• purpose of survey• how results will be used• why respondent was selected• why all answers are important
• emphasize confidentiality (if applicable)• provide contact information for respondents
with questions about the survey• Describe/explain any incentives for
participating in the survey and/or consequences for non-participation if applicable.
Survey Research: TTM531 Class Presentation
56
Identifying Respondents Identifying Respondents and Non-Respondentsand Non-Respondents
If respondents are not asked to provide any identifying information on the survey, you can…• Number the surveys before they are sent out OR• Unobtrusively number the return envelopes
before they are sent in the survey packetAND• Keep a list of survey/envelope numbers
corresponding to respondent name or id.
Survey Research: TTM531 Class Presentation
57
Monitor Returned SurveysMonitor Returned Surveys
Open surveys as they are returned Assign unique ID number to each survey
(if not already assigned) Stamp date on survey Track the number of survey coming in
each day• Use this information to help inform timing of
follow-up reminders.
Survey Research: TTM531 Class Presentation
58
Follow-up RemindersFollow-up Reminders
Sending 2 follow-up reminders works best to increase response rates.
Timing• first follow-up 2-3 weeks after original• second follow-up 2-3 weeks after 1st follow-up
Materials• reminder card with information to have new
survey sent if necessary OR• entire survey packet
Survey Research: TTM531 Class Presentation
59
Follow-up RemindersFollow-up Reminders
Send to:• Non-respondents only if possible to identify• All respondents if not possible to identify
non-respondents• include “if you have already responded, thank
you for participating….”
Survey Research: TTM531 Class Presentation
60
Response RatesResponse Rates
Response Rate =n surveys completed and returned
net sample size
(net sample size = sample size - undeliverable surveys)
In general …• Acceptable response rate = 50%• Good response rate = 60%• Very good response rate = 70%
Survey Research: TTM531 Class Presentation
61
And then….And then….
Data Entry Data Analysis Report Writing Presentation
Survey Research: TTM531 Class Presentation
62
Some ReferencesSome ReferencesBabbie, Earl (1990), Survey Research Methods. Wadsworth Publishing: CA.Cox, B., et. al., (1995), Business Survey Methods. Wiley: NY.Dillman, D.A. (1991), “The Design and Administration of Mail Surveys.” Annual
Review of Sociology, 17:225-249.Fink, Arlene (1995), How to Sample in Surveys. Sage: CA.Fowler, Floyd J. (1993), Survey Research Methods. Sage: CA.Fowler, Floyd J. (1995), Improving Survey Questions. Sage: CA.Sudman, Seymour and Bradburn, Norman (1982), A Practical Guide to Questionnaire
Design. Jossey-Bass: San Francisco.Tourangeau, R., and Smith, T.W. (1996), “Asking Senstive Questions: The Impact of
Data Collection Mode, Question Format, and Question Context, “ Public Opinion Quarterly, 60:275-304.
See also:Course Descriptions and Syllabi: the Joint Program in Survey Methodology
(University of Maryland, University of Michigan, and Westat) at http://www.jpsm.umd.edu/classinfo.htm