Surveys 101: Methodology & Good QuestionsPart I of II

Joyce Chapman, Consultant for Communications & Data AnalysisState Library of North Carolina, 4-16-2014

1

AgendaPlanning a surveyDistributing a surveyBest practices for survey design

This webinar will be recorded and made available here:http://statelibrary.ncdcr.gov/ld/webinars.html

Much info for this webinar comes from the book "Designing & Conducting Survey Research: A Comprehensive Guide." By Louis Rea & Richard Parker.

2

What is a survey and when do we use it

?

3

Before you beginWhat problems are you trying to solve? What questions are you trying to answer? What data will help you answer the questions? Does any of this data already exist?For colleges and universities: do you need IRB approval?

4

Clearly define the target population for the survey

5

How do we determine sample size?

This is where it gets tricky for us non experts!

Try something simple like a sample size calculator:

http://www.surveysystem.com/sscalc.htm

6

How do I select participants who accurately reflect my population?

Ideally you want a random sample, not a convenience sample

7

Types of survey distributionMail-out surveysWeb-based surveysIn-library paper-based surveysTelephone surveysIn-person interviewsIntercept surveys

8

Types of survey distributionMail-out surveysWeb-based surveysIn-library paper-based surveysTelephone surveysIn-person interviewsIntercept surveys

Most common types used in libraries

9

Web-based surveys: advantagesConvenienceRapid data collectionCost-effectiveAmple timeConfidentiality and security

10

Web-based surveys

Web-based surveys: disadvantagesLimited respondent baseSelf-selectionLack of interviewer involvement

11

In-person surveys: advantagesAbility to clarify and probeAbility to contact hard-to-reach populationsAssurance the instructions are followedObserved data (watch out!)

12

In-person surveys

In-person surveys: disadvantagesInterviewer-induced biasLess anonymity

13

Survey design

14

15

The introductory statement

The introductory statement

Mention the organization conducting the survey, state the relationship between the sponsoring institution and the respondent. "The Duplin County Public Library is conducting a

survey of residents in order to assess community opinions about services provided by the library."

Make a general statement establishing the objectives and goals of the survey and the significance of results to the respondents. "The purpose of this study is to identify those needs

that undergraduate students feel should be addressed in order to maintain a competitive 21st academic library."

16

The introductory statement

Clarify the basis of the sample selection. "The library is particularly interested in the opinion of

patrons who live on the west side of town, and as such you have been selected at random."

State that participation is valued and answers are neither correct nor incorrect. Participation is confidential. “Your opinions are very important to us. You should know

that there are no right or wrong answers and that your responses will be treated confidentially. Survey results will in no way be traceable to individual respondents."

If you are conducting an in-person or phone survey, identify yourself by name and job title.

17

18

Question sequence

Question sequence

Begin with "introductory questions." Sensitive questions should be placed near the

end. Consecutive questions that tend to evoke

reflexive responses should be minimized or respondents should be sensitized to the differences.

19

20

Question format

Question format

Advantages of closed-responseAnswer sets are uniform; facilitates comparison.Makes the question clearer to the respondent. Categories might remind the respondent of alternatives that otherwise would have been forgotten.Sensitive questions are often better addressed this way because you are pre-establishing an implicitly "acceptable" range of answers.They are easier on the respondent, increasing the likelihood that the response rate will be higher.

21

Question format

Disadvantages of closed-responseParticipants may select a response randomly rather than in a thoughtful way.Increased possibility that simplicity of format leads to entry errors.Compel respondents to choose a "closest representation" of their actual response. Subtle distinctions among respondents cannot be detected within the pre-established categories.

22

Question format

Open-responseRecommended to use sparingly / only when needed.Analysis will requires time-consuming categorization of open-ended response data by the researchersRequires greater communicative skills for respondent (spelling, writing, computer usage)Good compromise: use a close-ended question with a follow-up that's open-ended.

Do you want X? Yes__ No__. If no, why not? ____________

23

24

Question phrasing

Jargon, colloquialisms, word choiceThink of your audience. Wording should be simple. Avoid technical words.

X "How many times in a typical week do you use the library's Internet-enabled public access

computers?"O "How many times in a typical week do you use the

library's computers [to access the Internet]?"

25

Abbreviations and acronymsAssume your audience does not know any of these!

X “CCPL is interested in replacing its OPAC."O “Cleveland County Public Library is interested in

replacing its online catalog.”X “How frequently do you use our ILL services?”

O “How frequently do you use our Interlibrary Loan services? [This service allows us to request

material from another library for you if we do not have it here].”

26

Avoid ambiguityLook at the survey from every angle: are there ways that someone could interpret a question to have two meanings?

X "With which ethnic group do you most closely identify?"

[someone might interpret this as "which group do I best get along with?" instead of "which ethnic group am I a

member of?"]

O "Please indicate your race or ethnicity."

27

Avoid ambiguityLook at the survey from every angle: are there ways that someone could interpret a question to have two meanings?

X "What is your income?"

O "What is your income before taxes? Include salary as well as other sources of income."

X "How many people are there in your household?"

O "Including yourself, how many people are there in your household?"

28

Confusingly phrased questionsMake sure your wording is not unnecessarily confusing. The respondent should not have to spend time re-reading/interpreting the question.

X "Does it seem likely or does it seem unlikely to you that you would use a Maker Space if the library had one?"

It seems likely __ It seems unlikely __

O "If the library had a Maker Space, would you use it?"

Yes__ No__ Unsure__

29

Avoid “double barreled" questionsA double barreled question introduces two or more issues with the expectation of a single responseX "Are you satisfied with opening hours at the West branch

and at the main library?

O "Are you satisfied with opening hours at the West branch?"

"Are you satisfied with opening hours at the main library?"

O "Are you satisfied with opening hours at the West branch?"

"Are you satisfied with opening hours at the main library?"

30

Avoid “double barreled" questionsA double barreled question introduces two or more issues with the expectation of a single response

X "Is our staff friendly and professional?"

O "Is our staff friendly?"

"Is our staff professional?"

31

Do not use manipulative informationCertain questions require some explanation/background. Be careful that explanatory statements do not unduly influence responses.

X The county government spends approximately $10 per resident on landscaping public areas. Do you believe that the county government is adequately allocating funds for

our library by designating only $1.15 per resident?"

O "Do you believe that the county government is adequately allocating funds for our library by designating $1.15 per

resident?"

32

Do not use manipulative informationWe often are interested to know how knowledge of the difference in spending might affect responses, first ask straightforward, then with additional info.

O "Do you believe that the county government is adequately allocating funds for our library by designating

$1.15 per resident?"

O "If you were to learn that the county government spends approximately $10 per resident on landscaping public

areas, would that change your opinion about the adequacy of allocating $1.15 per resident to the library?"

33

34

Response formatting

Order of response optionsOften there is a logical, inherent order. If order is irrelevant, list choices alphabetically so respondents don't assume answers at the top are more important to the interviewer.

X Group study rooms O Digital media lab

Digital media lab E-books

Laptop lending Group study rooms

E-books Laptop lending

Printing/copying Printing/copying

35

Interval scale categoriesShould be as equal as possible in size. You also want each category to have an equitable number of respondents. Do not allow to overlap. Provide an unbounded final category.

X Age 0-10 O Age 0-9

Age 10-15 Age 10-19

Age 15-20 Age 20-29

Age 20-50 Age 30-39

Age 50-75 Age 40-49

Age 50+

36

Multiple responseSometimes we allow respondents to choose only one option and sometimes we let them choose multiple. Be very clear that you are allowing multiple! Otherwise results are unclear.

X For which of the following reasons do you use the library?

X__ Y__ Z__

O For which of the following reasons do you use the library? Choose

all that apply.

X__ Y__ Z__

37

Appropriate response choicesSurveys can be very frustrating when the questions are fixed response but do not have all the appropriate answer choices.Make sure you provide answer choices such as “Don’t know,” “N/A,” “Unsure,” and “Other” where appropriate.

X Does the laptop lending program meet your needs?

Yes__ No__

O Does the laptop lending program meet your needs?

Yes__ No__ N/A__ [or “I’ve never used this program__”]

38

You've drafted the survey!

39

Review each question against a checklistDo you have an introduction with all the parts we talked about? Are the first questions easy/factual? Are sensitive questions near the end?Are reflexive response questions minimized or is respondent sensitized?Have you clarified multiple v. single choice responses?Are you using closed- v. open-ended responses appropriately? Do you have "other" open-ended responses where needed?Have you done away with all acronyms/abbreviations?Have you avoided jargon, ambiguously worded, confusing, and double barreled questions? Do you have any manipulative language?

40

41

Questions?

Contact: joyce.chapman@ncdcr.gov919-807-7421

Find this Powerpoint/recording: http://statelibrary.ncdcr.gov/ld/webinars.html

Sign up for part II of the Survey 101 webinar series here: http://statelibrary.ncdcr.libcal.com/event.php?id=399040

42