PPA 501 – ANALYTICAL METHODS IN ADMINISTRATIONLecture 3d – Survey research

INTRODUCTION

Government administrators and elected officials love to claim that they possess a profound understanding of their public’s needs, desires, and disaffection.

INTRODUCTION Unfortunately, the administrators and

officials are learning that storms of controversy provide meager evidence of the workaday values of the everyday people they govern.

Surveys of the public, conducted following the basic precepts of survey design and analysis, are fast becoming the vehicle for genuine connection to the public will.

INTRODUCTION

Uses. Evaluations of government services. Changing demographics that may signal shifts in

service demand. Patterns of service utilization. Problem identification. Customer service.

INTRODUCTION

Surveys have several important qualities. Anonymity to respondents. Point of view, characteristics, or use patterns can

be characterized with little confusion. Good surveys provide input from a

representative cross-section.

BEGIN BEFORE THE BEGINNING

The best surveys grow from well-conceived and well-articulated reasons for conducting them.

Resist the temptation to hit the ground running.

Be certain of the purposes of the survey.

BEGIN BEFORE THE BEGINNING

Identify the appropriate audiences. Identify the political and personal will for

doing the survey. Determine whether the questionnaire to be

developed is better as a one-time or periodic survey.

Think about the usefulness of comparative data.

GETTING STARTED

Convene a steering committee with key stakeholders.

Enlist the help of top government officials or administrators.

DESIGNING THE SURVEY

Sampling. Choose the appropriate sampling frame: about

what population do you wish to generalize? A sampling plan must give every respondent in

the sampling universe an equal chance of ending up in the sample. Simple random sample. Stratified sampling. Stratified random cluster sampling.

DESIGNING THE SURVEY

Targeting the individual in the household. If no list exists, you may only have addresses or

phone numbers. If so, use household member with most recent birthday.

MAIL, PHONE, OR IN-PERSON INTERVIEWS

The best ways to conduct surveys vary by accuracy, speed, and cost. Most common are mail and phone surveys.

MAIL, PHONE, OR IN-PERSON INTERVIEWS

Criteria Mail Phone In-Person

Accuracy

Response rate 45-55% 65-75% 75%+

Permits anonymity High Moderate Low

Is free from interviewer bias High Moderate Low

Handles various question types

Long/ complex Low Moderate High

Visual aids High Low High

Ensures question order Low High High

Permits widest coverage

Targets geographic areas High Low High

Avoids education bias Low Moderate High

Gives easy access to target population

High High Moderate

Speed of administration Month Week Month

Cost per interview $8-12 $15-20 >$20

MAIL, PHONE, OR IN-PERSON INTERVIEWS

Increasing response rates. Multiple mailings (up to three) with stamped,

return address envelope. Press coverage. Combination of methods often best: Mail survey

with telephone and in-person followup.

MAIL, PHONE, OR IN-PERSON INTERVIEWS Selecting sample size.

The size of sample depends on desired precision of estimates.

Generally speaking, if opinions are split as much as possible, than 100 residents will have a margin of error of +/- 10% with 95 percent confidence. Four hundred residents the margin is +/- 5%.

In general, 100 is a good minimum number, especially for subgroups.

QUESTIONNAIRE CONSTRUCTION

Each question should be judged against the purposes of the survey and the uses to which it will be put.

Steal widely. National Citizen Survey from International City

Management Association and National Research Center.

QUESTIONNAIRE CONSTRUCTION

Major principles.Consistency.Clarity.

Vague wording. Double-barreled questions. Assumed knowledge. Overlapping response categories.

Simplicity. Specificity. Brevity (30 min. Phone, 60 min. In-person, 10

page mail). Context sensitivity.

QUESTIONNAIRE CONSTRUCTION

Major principles. Security.

Demographic at end. General to specific.

Fairness. Option symmetry (balanced responses). Option wording and order.

Background info, pros and cons, opinion. Randomize pros and cons in a complicated survey.

QUESTIONNAIRE CONSTRUCTION

Open-ended versus closed-ended questions. Commonality versus depth.

Broad categories of questions. Factual. Opinion. Attitude. Motive. Knowledge. Action or behavior.

QUESTIONNAIRE CONSTRUCTIONA Classification of Question Types by Content

Cognitive Awareness Stage

Type of Question Information Acquired

Cognitive Stage Factual The facts about people and things

Knowledge What people know about things

Affective Stage Opinion What people say about things

Attitude What people believe about things

Motive Why people act the way they do

Action Stage Behavior How people act, what they do; how they will react to certain stimuli

QUESTIONNAIRE CONSTRUCTION Sections.

Title and identification of the survey sponsors (including human subjects information).

Instructions. Warm-up questions (simple and direct, factual or

knowledge). Body of the questionnaire (more complex)

First third – Awareness and knowledge of factors, indicators, and causes.

Second third – attitude, opinion, motive scales. Lifestyle information (nonresponse goes up).

Last third – Focused questions, controversial, personally embarrassing. Partially completed is better than not completed.

Classification items. Demographics.

Pretest and revise the instrument.

WRITING QUESTIONS Open-ended questions.

Unstructured (free to answer as they will). Projective.

Association – React to a particular stimulus. Construction – Create a story or self-portrait. Completion – Finish an already started stimulus or

picture. Ordering – Arrange or select items as important or

salient. Expressive – Freely express themselves by drawing a

picture or something similar. Closed-ended questions.

Structured Answer. Dichotomous. Multichotomous.

Scale (see next slide).

DEVELOPING AND USING SCALES The types of scales most commonly used in

public administration are attitude scales, importance scales, rating scales, and readiness-to-act scales.

The most common are attitude scales. Types of attitude scales.

Thurstone – 100 or more opinion statements ranked by informed judges. Assigned scale values by median ranking of judges. Time consuming and rarely used.

Likert Scales – Statements ranked on a five-point scale ranging from Strongly Agree, Agree, Neutral or Undecided, Disagree, or Strongly Disagree. Usually summed or average across multiple items.

Semantic differential – Five- or seven-point scales with opposing adjectives.

CONDUCTING THE SURVEY The survey steering committee.

Double check questionnaire with steering committee.

Frequency of surveys. For most multipurpose surveys, no more than

once per year. Pretest.

Test on twenty people at random. Ask questions about format and clarity.

CONDUCTING THE SURVEY

Training.Survey assistants must be trained. All

must operate uniformly, asking the questions in the same way, coding in the same way.

Consistent open-ended coding.10% recontact of survey respondents.

Trying hard and keeping track.Three contacts by telephone for each

number.Warning and at least two mailings for mail

surveys.

REPORTING RESULTS

Data analysis.For most government surveys, percentages,

average responses, simple cross-classifications.

The most complicated analysis will be to get accurate population estimates – weighting.

Report writing and presentation.Executive summary.Bulleted lists.Document survey methods in appendix.Augment tables with bar and pie charts.Powerpoint for in person presentation.

HIRING A CONSULTANT

Previous experience. Ability to communicate findings. Share work with in-house staff. Intuition.