Creating an Effective Survey Instrument

CREATING AN EFFECTIVE SURVEY INSTRUMENT

SUZANNE SANNWALD

ASHFORD UNIVERSITY

INSTRUCTIONAL DESIGN & DELIVERY

EDU 652

DR. KATHY ZIENTEK

MAY 4, 2013

CREATING AN

EFFECTIVE ONLINE SURVEY

INSTRUMENT

1. Online Survey Benefits2. Survey Process3. Survey Goals: What, Who, Why & How4. Online Survey Design Hints5. Designing Questions

1. Writing Effective Questions

2. Survey Question Types

3. Question Sequencing

6. Feedback7. Results Processing

Overview

Online Survey Benefits

Participation can be self-select, if openly available on a Web page

Participation can be targeted when distributed by e-mail

Quick turnaround, although potentially lower response rates

Cost efficient, particularly with large samples

Optional anonymity conducive to questions of a sensitive nature

(Schoniau & Fricker, 2002, p. xiv, viii-xx)

What are the basic steps of survey creation?

Click icon to add pictureWhen creating online surveys, “the basic steps in the process

should be the same as with any other survey

medium” (Schoniau & Fricker, 2002, p. 6).

Survey Process

Plan optimal survey timing considering time necessary for entire process

University of Wisconsin-Madison (2010) defines the survey process as follows:

An additional step is to apply findingsApply findings

(University of Wisconsin-Madison, 2010, p. 4)

Survey Goals: What

What is the purpose of the survey? What do you want/need to learn?” According to Barlow (2010), surveys often

have a goal of gathering information about: “Feelings, attitudes, reactions to products or

services” “Anticipated reactions to new kinds of products

or services” “Current levels of behaviors/activities for

planning of or to gauge impact of programs” (p. 3).

Survey Goals: Who

Who is your target population? Who will you use as your sample

(subgroup of your target population)? Sample Size Calculator

Reasons to use a sample: If the target population is too large or

widely dispersed To allow for more reminders and follow-up

contact Consider any relevant privacy issues

(Barlow, 2010, p. 4)(University of Wisconsin-Madison, 2010, p. 5, 17)

Survey Goals: Why & How Why do you need the information about

the particular target population? How will the results help you or others? How will you use the results? How can you apply your findings?

Will you use information to change current or future service delivery?

Will you re-evaluate the impact of any changes?

(Johnson, n.d., p. 3)

How else can we improve response rates?

Click icon to add pictureFrom the start, improve response rate by giving respondents

“clear indication of intent of survey and how information will

be used”(Barlow, 2010, p. 9).

Online Survey Design Hints

Ensure and communicate protection of privacy, as appropriate

Provide indication of survey progress Limit questions per screen Limit use of graphics, understanding graphics may cause

bias Limit use of complex or matrix questions Restrict and validate responses when possible to reduce

response errors Add and test all skip logic to streamline survey Only require responses when necessary Ease navigation with format (e.g., spacing, fonts, shading) Thoroughly test survey, including technical compatibility(Schoniau & Fricker, 2002, p. 41-53)

How can we improve reliability and validity?

Click icon to add pictureStrive for reliability and validity:• “Reliability is the extent to

which repeatedly measuring the same property produces the same result” and

• “Validity is the extent to which the survey question measures the property it is supposed to measure”

(University of Wisconsin-Madison, 2010, p. 6).

Writing Effective Questions

Survey goals should drive and shape questions According to Johnson (n.d.), surveys reflect a

solely personal expression and should thus avoid: “Asking secondhand information” “Asking hypothetical questions” “Asking about causation” “Asking for solutions to complex issues” (p. 14)

Language used should be familiar to respondents Do not use abbreviations Avoid technical language and jargon

Writing Effective Questions

“Questions are

effective only if all learners understand them and can answer

them the way you

intended”

(Horton, 2012, p. 243)

Focus on one issue per question Use simple, direct, neutral language

Keep both questions and choices short Avoid difficult, ambiguous, or leading language Avoid questions in the negative or with double

negatives Include clear instructions

Specify conditions such as time frames (e.g., within the past week)

Specify tasks required (e.g., select one, select all that apply)

Emphasize important word (e.g., “not”) Use parallelism with answer choices

Same length Equivalent grammar

(Barlow, 2010, p. 5, 7)(Horton, 2012, p. 244-

259)(Johnson, n.d., p. 11-

13)

What types of questions are there?

Click icon to add pictureTo pick the right question

type, “consider what kind of information

you are trying to gather” (Horton, 2012, p. 220).

Survey Question Types

May take more time for respondents to answer

Takes more time to process

May provide new or unique insight

Responses may be hard to interpret and analyze

Quick response time

Respondents are more limited in responses

Quick processing time

Insights are limited response options

Open-Ended “Fill in the blank” Responses

Closed or Structured Responses

(University of Wisconsin-Madison, 2010, p. 9)

Survey Question Types: Closed

Examples: Multiple Choice True/False Checklists Attitude scales

Options should be exclusive (non-overlapping)

Options should be exhaustive Include an “Other” option when appropriate

For frequency, use specific number ranges versus “often, seldom, never”(Barlow, 2010, p. 5-6)

Question Sequencing

Order questions for a logical flow Start with an easy question Start with questions that reflect the study topic

and/or pique interest Group similar questions For questions requiring recall, use the order in

which they happened Sequence questions about details before asking

for an overall assessment End with personal and/or demographic questions Enable forward and backward navigation through

questions (Johnson, n.d., p. 18)(University of Wisconsin-Madison, 2010, p.

11)

Feedback

During Survey: Provide indication of survey progress Provide clear error/warning messages Provide an avenue for respondents to report

problems. Incomplete Surveys:

Follow-up with reminders to complete surveys Completed Surveys:

Send thank you messages for survey completion Consider providing incentives for survey completion Update respondents with survey results Plan outreach to respondents based on survey

results (Schoniau & Fricker, 2002, p. 45, 47, 52-53)

Results Processing: By Survey Consider audience and purpose of result

sharing Define demographic characteristics of

sample Report survey response rate Determine handling of incomplete

surveys Provide general summary of results Attach survey as appendix

(Barlow, 2010, p. 13)

Results Processing: By Question Quantitative Processing:

For open-ended responses Report key themes and their frequency

Qualitative Processing For closed responses Report frequency, percentages, and by

categories Find ways to visually represent

responses

(Barlow, 2010, p. 12)

References

Barlow, P. (2010). Effective survey design and analysis [Presentation]. Retrieved from University of Wisconsin-La Crosse: http://www.uwlax.edu/catl/Presentations/T4TW/Effective%20Survey%20Design%20and%20Analysis+Worksheet%20-%20June%202010.pdf

Horton, W. (2012). E-Learning by design (2nd ed.). San Francisco, CA: Wiley.

Johnson, K. (n.d.). An introduction to effective survey design and administration [Presentation]. Retrieved from Penn State University Survey Research Center: http://studentaffairs.psu.edu/assessment/pdf/KurtJohnsonPresentationFA11.pdf

University of Wisconsin-Madison Office of Quality Improvement. (2010). Survey fundamentals: A guide to designing and implementing surveys. Retrieved from http://oqi.wisc.edu/resourcelibrary/uploads/resources/Survey%20Guide%20v%202.0.pdf

Schoniau, M., Fricker, R. D., & Elliott, M. N. (2002). Conducting research surveys via e-mail and the Web. [Adobe Digital Editions version]. Retrieved from http://site.ebrary.com/