SURVEY RESEARCH

Survey Research: TTM531 Class Presentation

1

SURVEY RESEARCHSURVEY RESEARCH

TTM531 Fall 2001 Class Presentation

Nancy WhelchelCoordinator for Survey ResearchUniversity Planning and Analysis

[email protected]

515-4184http://www2.acs.ncsu.edu/UPA/survey/survey_research/


2

Major TasksMajor Tasks

ID Purpose of Research Decide on Survey Design and Format Select Sample Design Survey Instrument Pre-test Instrument Collect Data Analyze Data Report Findings


3

Purpose of Your SurveyPurpose of Your Survey

Ask Yourself: WHY do I want to/need to do a survey?

WHAT do I want to learn?

WHO is my audience?

HOW will I use the information I gather?


4

Purpose of Your SurveyPurpose of Your Survey

Our Example: Why =

President of NCTextiles.com, where you work, told you to. What want to learn =

How does employee satisfaction and productivity compare between NCTextiles.com telecommuters and non-telecommuters.

Who is audience =Company president and executive board.

How use =Set new policies for telecommuting.


5

Purpose of Your SurveyPurpose of Your Survey Description

Discover the distribution of certain traits or characteristics

(e.g., #telecommuters, $revenue generated)

Explanation Understand why something happens

(e.g., is there a difference between telecommuters and non-telecommuters in job satisfaction and in productivity)

Exploration Help inform a research design


6

Think About Available Think About Available ResourcesResources

What information is already available from non-survey sources?

Have any surveys already been done on this question?

Who is going to:• Design the survey• Administer the survey• Enter the data• Analyze the data• Write up the results• Present the findings


7

Think About Available Think About Available ResourcesResources

Should I hire a professional???

What resources/skills are available in-house?

What types of decisions are going to be made based on your findings?

How much money do you have for the survey?


8

Basic Survey DesignsBasic Survey Designs Cross-sectional

• Data collected at one point in time from a sample selected to describe a larger population

Longitudinal• Trend

• Different samples from a general population studied at different points in time.

• Cohort• Same specific population studied over time (may be

different samples).• Panel

• Same sample studied over time.


9

Basic Survey FormatsBasic Survey Formats

Format determines much of research design (e.g., sampling, questionnaire design, administration, data analysis, etc.)

Basic formats include:• Self-administered mail survey• Telephone interview• Face-to-face interview• Focus group• Self-administered Web survey• Self-administered Email survey


10

The Survey SampleThe Survey Sample

WHO or WHAT do I want to survey?

Sampling allows you to study just a portion of the population/group and make estimated assertions about the population from the sample.

Warning: Bad Sample = Bad Data


11

Why Sample?Why Sample?

TimeShorter time in the field decreases the chance that external factors will influence results.

QualityDoing fewer surveys increases control over the entire process (interviewing, data entry, etc.) and therefore decreases error

Cost


12

What is a Sample?What is a Sample?

Some definitions: Element or Sampling Unit

Unit about which information is collected (e.g., employee)

Survey Population

Aggregate of all elements interested in from which sample is selected (must be clearly defined)

(e.g., NCTextiles.com non-executive employees with job tenure 3+ months )


13


Some more definitions: Sampling Frame

Actual physical source listing elements in the survey population

e.g., Edited list from company employee directory . Sample Population

Collection of elements selected from the survey population sampling frame


14


A Warning: Findings from a survey are only

representative of the aggregate of elements in the sampling frame.

In other words…You can’t use your results to say anything

about the folks who didn’t have a chance to be included.


15

Sample SizeSample Size

Depends on: Cost Level of precision want (“margin of error”) Level of confidence want Anticipated response rate Number of responses needed per

subgroup interested in


16

Sample SizeSample Size

Example: Margin of Error +/- points confidence interval 80% 90% 95% 5 150 275 400 10 40 70 100

If 45% of 400 respondents agree that NCTextiles.com provides “excellent” health benefits, this means that 95 out of 100 times between 40% (45%-5) and 50% (45%+5) believe the company provides “excellent” health benefits.

Source: McClelland, Lou (1993), “Survey Research Mechanics in Three Checklists,” (working paper). Office of Research and Information: University of Colorado at Boulder.


17

Selecting a SampleSelecting a Sample

Probability Sample• A representative sample

• Everyone in the sample population has an equal chance of getting in the sample.

• Results can be generalized to the sample population.

Non-Probability Sample• Non-representative sample

• Purposive, quota, who’s available• Results can not be generalized beyond the

sample.


18

Probability SamplingProbability Sampling

Simple random sample• List the elements in the sampling frame and

use a random number chart to select elements.

Systematic sampling• List the elements in the sampling frame and

take every Nth element, based on how many needed in sample.• Warning: Be attentive to any inherent

patterns in how sampling frame is ordered.


19


Stratified sampling• Draw appropriate proportion of elements

from homogeneous groups interested in (based on theory).

• Note: Think about using disproportionate sampling then “weighting” the data. E.G., If there are relatively few numbers in one sub-group you are interested in, include proportionally more of them in your sample, then “weight” the data to reflect their true proportion in your population.


20


Stratified sampling continued

An Example:

250 out of your 2000 employees in the sample population telecommute (12.5% of sample population).

With stratified sampling with sample size of 800, would have 100 telecommuters (800 * .125), and 700 non-telecommuters (800 * .875).


21


Stratified sampling continued

An Example continued:BUT, you might want to “oversample”

telecommuters (even include all 250) to be able have less error in their results, and “undersample” non-telecommuters (e.g., 550). When analyzing all results, “weight down” telecommuters to reflect their true proportion in the sample population, and “weight up” non-telecommuters.


22

Instrument DesignInstrument Design

Start with: Conceptualization

• Define the concepts you want to study

Operationalization• Specify indicators of your concepts.

• Make exhaustive list of everything the concept could include


23


ConceptualizationJob Satisfaction =

“A pleasurable or positive emotional state resulting from the appraisal of one’s job.”

Includes both extrinsic and intrinsic aspects.Has several dimensions:

• importance to employee• motivation of employee• current level of attainment • satisfaction with attainment


24


Extrinsic:• pay• promotion• supervision• co-workers• company• physical conditions• security• social status• work itself

Intrinsic:• variety of work• autonomy and freedom• responsibility• feedback• creativity• recognition• meaningfulness of work

Operationalization: Job Statisfaction


25

Questionnaire Questionnaire ConstructionConstruction

The way a question or statement is worded and the response options offered determine

the nature of the data received.

Bad question = Bad data


26


Some jargon… Questions need to be VALID

“…an empirical measure adequately reflects the real meaning of the concept under consideration.” (Babbie, 1990)

• face validity = the measure jibes with common agreement of what a concept entails.

• predictive validity = the measure accurately predicts behavior• content validity = the measure covers the range of meanings

included within the concept.

• construct validity = the measure relates to other measures

theoretically related to concept.


27


Some more jargon… Questions need to be RELIABLE

“…a particular technique, applied repeatedly to the same object, would yield the same result each time.” (Babbie, 1990)


28


Open-end Questions• Respondents write question response

in own words• probably need to enter data by hand• develop coding scheme for responses• content analysis ?

Suggestion: It’s a good idea to always include an open-end question giving the respondent the opportunity to add any additional comments they might have.


29


Closed-end or Forced-choice Questions• Respondent selects question answer

from among a list of possible responses

Suggestion: It’s a good idea to ask multiple indicators of a single concept, and then create a scale by combining those indicators.


30

Good Questions...Good Questions...

Good Questions... Are clear and use simple language Are concise Are specific Are possible to answer Are relevant to the respondent Do not use negatives Avoid biased terms Have only 1 part (not “double-barrel”)


31


Use simple language...

My job is usually interesting enough to keep me from getting bored.

Is better than :

My job is usually engaging enough to permit me from reaching a banausic state of mind.


32


Are concise...How interested would you be in a flexible work schedule in

which you would have to work an 8 hour day in the office, but would be able to arrive at work anytime between 6:00 and 10:00 AM and leave between 2:00 and 7:00?.

Is better than:

How interested would you be in being able to leave your house either before the morning rush hour or after the morning rush hour, and then arrive at the office not at the standard 8:00 AM time that is required according to the employee handbook, but at whatever time you got there, and then work a 8 hour day at the office (excluding and hour for lunch), and then leave the office so you are not commuting during rush hour but either before or after rush hour?


33


Are specific...

In what year did you first start working at NCTextiles.com?

Is better than:

When did you start working here?


34


Are possible for the respondent to answer...

NO:My contribution to the company has had a direct

impact on its ability to control a broader share in the global market.


35


Are relevant...

NO:Polyester doesn’t breathe as well as natural fibers.


36


Do not use negatives...

The lack of flexibility in my work hours makes it difficult for me to spend the amount of time I would like to with my family.

Is better than:

The lack of flexibility in my work hours does not make it possible for me to spend the amount of time I would like to with my family.


37


Do not use biased terms or suggestions...

Do you agree or disagree with the following statement: All employees need to be in the office during regular

business hours.

Is better than:

Don’t you agree with the president of the company when she says that all employees need to be in the office during regular business hours?


38

Good Questions..Good Questions..

Do not ask two questions in one (“double- barrel”)...

I find real enjoyment in the work I do.I like the hours I keep at work.

Is better than:

I find real enjoyment in the work I do and the hours I keep.


39

Response OptionsResponse Options

Response options should reflect the concepts you are trying to measure, and fit with the question wording.

Suggestion: Avoid simple “yes” or “no” type responses and try to measure intensity of feeling when possible.


40


Must be: Mutually exclusive

can select only one appropriate answer

Example: NOT mutually exclusive...Q. On most days, what time do you leave your house

for work?6:00 AM or earlier 7:30-8:00 AM6:00-6:30 AM 8:00-8:30 AM6:30-7:00 AM 8:30-9:00 AM7:00-7:30 AM 9:00 AM or later


41


Mutually exclusive continued

Example: IS mutually exclusive...Q. On most days, what time do you leave your house

for work?Before 6:00 AM 7:30-7:59 AM6:00-6:29 AM 8:00-8:29 AM6:30-6:59 AM 8:30-8:59 AM7:00-7:29 AM 9:00 AM or later


42


Must be: Exhaustive

all possible answers are listed (including e.g., “other, “ “don’t know,” etc.)

Example: NOT exhaustiveQ. On most days, what time do you leave your house

for work?6:00-6:29 AM 7:30-7:59 AM6:30-6:59 AM 8:00-8:29 AM7:00-7:29 AM 8:30-8:59 AM


43


Exhaustive continued

Example: IS exhaustiveQ. On most days, what time do you leave your house

for work?Before 6:00 AM 7:30-7:59 AM6:00-6:29 AM 8:00-8:29 AM6:30-6:59 AM 8:30-8:59 AM7:00-7:29 AM 9:00 AM or later


44

Question OrderQuestion Order

Be attentive: Initial questions effect answers to subsequent ones.

Ask less interesting questions (e.g., demographics) at end of self-administered survey and beginning of interview survey.


45

Instructions for Instructions for RespondentRespondent

Provide general instructions in beginning of self-administered survey• Brief explanation could include:

• why doing survey• why answers are important• how to answer questions• stress confidentiality (if appropriate)

• Do NOT ask permission - - assume respondent will want to participate.

Provide specific instructions to questions as necessary


46

Instructions for Instructions for RespondentRespondent

ExampleInstructions: The following statements are about some

features of your job. Please tell us how satisfied or dissatisfied you feel with the features of your present job by circling one of the response options listed below each statement.

There are no right or wrong answers. All replies are confidential so give your honest opinion on each of the statements. Please do not discuss your answers with other people. It is your opinion which matters.

Even though it may be hard to decide, please be sure to answer all questions. Thank you for your help.


47

Questionnaire FormatQuestionnaire Format

Should be visually attractive and nicely reproduced

Should be spread out and uncluttered• better to have more than 1 page than

cramped page Should be broken into logical sections

when possible Should have clear skip patterns for

contingency questions


48


Should have clear spaces for respondents to mark answers• use boxes • parentheses ( X ) • or numbers to circle • avoid lines to put check on X_

X

1


49


Helpful hint….

Be sure to seek advice on question wording, response options, and questionnaire format from those who will be doing the data entry and data analyses.


50

Pre-TestPre-Test

Test the survey instrument before doing the actual study.

Suggestion: It’s also a good idea to test out your sample design, data collection methods, data processing, and data analysis if possible.


51

Pre-Testing the Survey Pre-Testing the Survey InstrumentInstrument

Use respondents similar to those who will be in survey sample.

Test:• Question clarity

• failure to answer?• multiple answers?• “other” answers?• qualified answers?


52


Test continued:• Questionnaire format

• instructions?• flow?• layout?

• Variance in responses (if interested in relationships among variables…)

• do most respondents give the same answer to a question?


53


Revise survey instrument as necessary Pre-test again if possible

(ideally with some of original pre-test respondents, and some new respondents)


54

Data CollectionData Collection(Self-administered mail survey)(Self-administered mail survey)

Include: Survey instrument Return envelope

• self-addressed and stamped(either postage-paid or 1st class stamp)

Cover letter• on letterhead paper• signed by most important person possible


55

Data CollectionData Collection(Self-administered mail survey)(Self-administered mail survey)

Cover letter (continued)• brief explanation of

• purpose of survey• how results will be used• why respondent was selected• why all answers are important

• emphasize confidentiality (if applicable)• provide contact information for respondents

with questions about the survey• Describe/explain any incentives for

participating in the survey and/or consequences for non-participation if applicable.


56

Identifying Respondents Identifying Respondents and Non-Respondentsand Non-Respondents

If respondents are not asked to provide any identifying information on the survey, you can…• Number the surveys before they are sent out OR• Unobtrusively number the return envelopes

before they are sent in the survey packetAND• Keep a list of survey/envelope numbers

corresponding to respondent name or id.


57

Monitor Returned SurveysMonitor Returned Surveys

Open surveys as they are returned Assign unique ID number to each survey

(if not already assigned) Stamp date on survey Track the number of survey coming in

each day• Use this information to help inform timing of

follow-up reminders.


58

Follow-up RemindersFollow-up Reminders

Sending 2 follow-up reminders works best to increase response rates.

Timing• first follow-up 2-3 weeks after original• second follow-up 2-3 weeks after 1st follow-up

Materials• reminder card with information to have new

survey sent if necessary OR• entire survey packet


59

Follow-up RemindersFollow-up Reminders

Send to:• Non-respondents only if possible to identify• All respondents if not possible to identify

non-respondents• include “if you have already responded, thank

you for participating….”


60

Response RatesResponse Rates

Response Rate =n surveys completed and returned

net sample size

(net sample size = sample size - undeliverable surveys)

In general …• Acceptable response rate = 50%• Good response rate = 60%• Very good response rate = 70%


61

And then….And then….

Data Entry Data Analysis Report Writing Presentation


62

Some ReferencesSome ReferencesBabbie, Earl (1990), Survey Research Methods. Wadsworth Publishing: CA.Cox, B., et. al., (1995), Business Survey Methods. Wiley: NY.Dillman, D.A. (1991), “The Design and Administration of Mail Surveys.” Annual

Review of Sociology, 17:225-249.Fink, Arlene (1995), How to Sample in Surveys. Sage: CA.Fowler, Floyd J. (1993), Survey Research Methods. Sage: CA.Fowler, Floyd J. (1995), Improving Survey Questions. Sage: CA.Sudman, Seymour and Bradburn, Norman (1982), A Practical Guide to Questionnaire

Design. Jossey-Bass: San Francisco.Tourangeau, R., and Smith, T.W. (1996), “Asking Senstive Questions: The Impact of

Data Collection Mode, Question Format, and Question Context, “ Public Opinion Quarterly, 60:275-304.

See also:Course Descriptions and Syllabi: the Joint Program in Survey Methodology

(University of Maryland, University of Michigan, and Westat) at http://www.jpsm.umd.edu/classinfo.htm

Documents

SURVEY RESEARCH