An Insider’s Guide to Survey Design Amy Feder Annemieke Rice

An Insider’s Guide to Survey Design

Amy FederAnnemieke Rice

Integrated Assessment Solutions

Today’s Learning Outcomes

Participants will…. understand the concepts of local survey planning with

results in mind. gain knowledge of best practices regarding instrument

implementation. practice specific survey design and editing skills. be able to apply knowledge and skills to a campus

project.


Professional Equivalencies

ACPA ASK Standards• Ability to evaluate and create a rigorous survey with focus

on effective question wording, effective survey format for the intended sample population, appropriate administration method, along with ability to achieve appropriate response rate

• Ability to determine the manner in which participants with disabilities will utilize any assessment instruments selected

• Ability to review an instrument for inclusive and accessible language likely to be viewed as informed and respectful by those for whom use of the instrument is intended

NASPA’s Assessment Framework• Assessment 301: Home Grown Surveys for Assessment


Agenda

I. Why Survey Design is Important

II. Survey Design Concepts:Developing ContentResponse FormatsWriting QuestionsStructure & LengthArrangement & Design

III. Survey Administration


WHY SURVEY PLANNING AND DESIGN IS IMPORTANT

Section I:


Overview

“Questions and answers are part of everyday conversation; they are part of the fabric of our social life.”

“remote conversation”

“conversation with a purpose”


Steps in survey design


DEVELOPING CONTENTSECTION II:


Beginning with the end in mind

Is a survey the best method?

What data is already available?

Who should be involved?

What can we learn from past assessments?

Who will review results?

How will this data be used?


Develop objectives

Examples:• Age• Major• Preferred communication method• Satisfaction with program• Interest in future programs• Impact on overall college experience


RESPONSE FORMATSSECTION II:


Response Formats

Open ended responses• Free response - text• Numeric• Yes/No with please explain

Types of multiple choice responses• Yes/No• Single response• Multiple response (e.g., Check all that apply, Select 3)• Ranking• Scales (see handout)


What to consider

Scales that match

Mutually exclusive

Exhaustive

Neutral/Not applicable/Non-response optionsChoose not to respond Don’t know

Not applicable Unable to judge

No opinion Neutral

Neither ___ nor ___


WRITING QUESTIONS


Reliability & Validity

Reliability – yielding the same results repeatedly• Test/Re-test – consistency over time• Inter-rater – consistency between people

Validity – accurately measuring a concept• Internal – confidence results due to independent variable• External – results can be generalized • Face validity – does this seem like a good measure?

If a survey is valid, it is almost always reliable!


Language

Consistency

Simplicity

Spelling

Grammar

Q&A instead of statements


Pitfalls to avoid

Socially desirable responding – based on social norms• Can never be eliminated• Consider sensitive topics like race, drug and alcohol use, sexual

activity, and other areas with clear social expectations

Leading questions – suggesting there is a correct answer

Double-barreled questions – asking more than one question

Double negatives – including negative phrasing which makes responding difficult


STRUCTURE & LENGTHSECTION II:


Creating structure: Matrices


Creating structure: Display Rules


Creating structure: Paths


Evaluating Length

Greater attrition at 22 questions or 13 minutes

What to consider:• Excluding “nice to know”• Eliminate what you already know• Outlining how results will be used• Number of open-ended questions• Number of required questions


ARRANGEMENT & DESIGNSECTION II:


Organization of content


Design

Descriptive title Branding

Progress bar Visibility of scale

Questions per page Section headers

ADA compliant


SURVEY ADMINISTRATIONSECTION III:


Data Collection Methods

Pros Cons

Web

No data entry Accuracy is excellent Technology benefits (e.g., display

rules, required questions) Immediate results Anonymous

Audience is not usually captive Possible misinterpretation (can’t ask ?s) Technology issues Response sample unrepresentative

Mobile

No data entry Accuracy is good Technology benefits (e.g., display

rules, required questions) Captive audience Administrator is available for ?s

Technology issues Response sample unrepresentative Limited formatting Anonymity is questionable

Paper

Captive audience Administrator is available for ?s No technology issues

No benefits of technology Accuracy can be compromised Data entry necessary Anonymity is questionable


Piloting

1. Take it as if you were respondent

2. Seek reviews from colleagues with no prior knowledge

3. Administer to sample of actual population being studied• Focus group• Questions at end of survey• Observing


Confidentiality & Anonymity

Confidentiality:data kept contained to a specific set of reviewers

Anonymity: participants not individually

identified


Invitations

Importance/Purpose

Relevancy to respondent

Request for help

How and by whom results are used

How long it will take to respond

Deadline

Incentives/Compensation

Contact information


Response rate strategies

Preannouncement

Reminders

Timing

Incentives/Compensation

Sampling

Culture of sharing results


References & ResourcesBrace, I. (2004). Questionnaire design: How to plan, structure and write survey

material for effective market research. London: Kogan Page Limited.Bradburn, N.M., Sudman, S. & Wansink, B. (2004). Asking questions: The definitive

guide to questionnaire design – for market research, political polls, and social and health questionnaires. San Francisco: Jossey-Bass.

Fink, A. (1995). The survey handbook. Thousand Oaks, CA: Sage Publications, Inc. Fowler, F.J. (1993). Survey research methods. Newbury Park, CA: Sage

Publications, Inc. Fowler, F.J. (1995). Improving survey questions: Design and evaluation. Thousand

Oaks, CA: Sage Publications, Inc. Porter, S. (2004). Overcoming Survey Research Problems. New Directions for

Institutional Research, 121.Rea, L.M. (1992). Designing and conducting survey research: A comprehensive

guide. San Francisco: Jossey-Bass. Stage, F.K. (Eds.). (1992). Diverse methods for research and assessment of college

students. Alexandria, VA: ACPA.Suskie, L. (1997). Questionnaire survey research: What works (2nd ed.) Tallahassee,

FL: AIR. Tourangeau, R., Rips, L.J., & Rasinski, K. (2000). The psychology of survey

response. New York: Cambridge University Press.


Questions?

Annemieke Rice

[email protected]

716.652.9400

www.studentvoice.com

Amy Feder

[email protected]

Documents

An Insider’s Guide to Survey Design Amy Feder Annemieke Rice