1

Gathering SoTL Gathering SoTL Evidence: Methods for Evidence: Methods for Systematic Inquiry into Systematic Inquiry into

Student LearningStudent LearningRenee A. MeyersRenee A. Meyers

Coordinator, UWS SoTL Coordinator, UWS SoTL Leadership SiteLeadership Site

Faculty College, May 2009Faculty College, May 2009

2

Using Questions to Using Questions to Gather SoTL EvidenceGather SoTL Evidence

Types of Questions Types of Questions Question DesignQuestion Design Question Issues and ProblemsQuestion Issues and Problems Types of Response ScalesTypes of Response Scales Design and Preparation of SurveyDesign and Preparation of Survey

3

Asking Questions To Asking Questions To Gather EvidenceGather Evidence

4

Methods for Gathering Methods for Gathering EvidenceEvidence

Evidence can include:Evidence can include:

Course-Intrinsic EvidenceCourse-Intrinsic Evidence

Course-Extrinsic EvidenceCourse-Extrinsic Evidence

Using your research question, what Using your research question, what is one type of course-intrinsic and is one type of course-intrinsic and one type of course-extrinsic evidence one type of course-extrinsic evidence you might collect?you might collect?

5

Asking QuestionsAsking Questions

All SoTL evidence gathering is All SoTL evidence gathering is focused on asking focused on asking questionsquestions of of students.students.

Research on writing Research on writing surveysurvey questions questions provides comprehensive information on provides comprehensive information on how to do this besthow to do this best

6

Survey EvaluationSurvey Evaluation

Look at survey in packet. Look at survey in packet. What works?What works? What doesn’t work? What doesn’t work? What is missing?What is missing?

7

What Survey Research Tells What Survey Research Tells Us About Asking Good Us About Asking Good

QuestionsQuestions

8

The Impact of Questions on The Impact of Questions on the Research Processthe Research Process

Poor questions lead to lousy results.Poor questions lead to lousy results.

How might answers to these questions How might answers to these questions differ? differ? Do you attend class?Do you attend class? Do you Do you regularlyregularly attend class? attend class? How many times per week do you attend this How many times per week do you attend this

class?class?

Asking survey questions is part science Asking survey questions is part science and part art.and part art.

9

Types of Survey Types of Survey QuestionsQuestions

Open Ended and Closed Ended Open Ended and Closed Ended Question FormatsQuestion Formats

Attitude, Knowledge, and Behavior Attitude, Knowledge, and Behavior Question FormatsQuestion Formats

10

Open-Ended QuestionsOpen-Ended Questions Open-ended questions ask respondents Open-ended questions ask respondents

to state in their own words their to state in their own words their response to the question.response to the question.

Closed-ended (or fixed alternative) Closed-ended (or fixed alternative) questions ask respondents to respond questions ask respondents to respond by picking one or more alternatives by picking one or more alternatives from a list of available options.from a list of available options.

Can you find any examples of open-Can you find any examples of open-ended questions on the survey? ended questions on the survey? Are these good open-ended Are these good open-ended questions? questions?

11

Open Ended QuestionsOpen Ended Questions

What are advantages and What are advantages and disadvantages to open- and close-disadvantages to open- and close-ended questions? ended questions?

12

Advantages and Disadvantages Advantages and Disadvantages

Are often less time consuming Are often less time consuming May provide more complete answersMay provide more complete answers May provide information May provide information

researchers did not anticipateresearchers did not anticipate Responses are often easier to Responses are often easier to

analyzeanalyze May produce better response rateMay produce better response rate

13

Surveys: Asking QuestionsSurveys: Asking Questions

Surveys are used most often in SoTL Surveys are used most often in SoTL research when we are interested in research when we are interested in asking questions about: asking questions about: how students FEEL or THINK about how students FEEL or THINK about

certain issues, topics, activities, related certain issues, topics, activities, related to their learning,to their learning,

what students KNOW about various what students KNOW about various topics, etc.topics, etc.

how often students engage in certain how often students engage in certain learning BEHAVIORSlearning BEHAVIORS

14

Attitude/Feelings QuestionsAttitude/Feelings Questions

Attitude/feelingsAttitude/feelings questions ask questions ask respondents for their views on respondents for their views on certain topics or issuescertain topics or issues There are no right or wrong answers to There are no right or wrong answers to

attitude/feeling questions.attitude/feeling questions. Attitude/feeling questions should be Attitude/feeling questions should be

“neutrally” written so that the “neutrally” written so that the respondent is not led to believe that a respondent is not led to believe that a certain response is desired.certain response is desired.

15

Knowledge QuestionsKnowledge Questions

KnowledgeKnowledge questions ask about what questions ask about what a respondent knows about a a respondent knows about a particular topicparticular topic

Knowledge questions can be asked as Knowledge questions can be asked as a screening question to determine if a screening question to determine if respondents are qualified to answer respondents are qualified to answer attitude questions.attitude questions. ““What do you know about evolution?What do you know about evolution?

“ “How do you feel about evolution as a How do you feel about evolution as a theory of creation?”theory of creation?”

16

Behavior QuestionsBehavior Questions

BehaviorBehavior questions ask respondents to questions ask respondents to report on their past or future behaviorreport on their past or future behavior Responses to behavior questions may Responses to behavior questions may

be biased because of faulty or biased be biased because of faulty or biased recall by respondentsrecall by respondents

17

Attitude, Knowledge, and Attitude, Knowledge, and Behavior QuestionsBehavior Questions

Can you find an attitude/feelings, Can you find an attitude/feelings, knowledge, and behavior question on the knowledge, and behavior question on the survey?survey?

Write one attitude/feelings, knowledge, or Write one attitude/feelings, knowledge, or behavior question for your own SoTL behavior question for your own SoTL project. project. Pre-assessment measurePre-assessment measure Question for an assignment you will collectQuestion for an assignment you will collect Post-assessment Post-assessment

18

Question Issues and Question Issues and ProblemsProblems

19

Recall QuestionsRecall Questions

Is there a question on survey that Is there a question on survey that asks students to “recall” some asks students to “recall” some behavior? behavior? How would you evaluate these How would you evaluate these

questions?questions? Would you know how to answer Would you know how to answer

them? them? Is there a better way to ask these Is there a better way to ask these

questions?questions?

20

Using Diaries for Recall Using Diaries for Recall QuestionsQuestions

To minimize recall errors, researchers To minimize recall errors, researchers can ask respondents to keep personal can ask respondents to keep personal diaries to record their actual teaching diaries to record their actual teaching or learning activities over a period of or learning activities over a period of timetime Caution: Respondents may forget to fill Caution: Respondents may forget to fill

out their diaries on a timely basis and out their diaries on a timely basis and instead “catch up” their diaries just instead “catch up” their diaries just before they are due to be collectedbefore they are due to be collected

21

Unaided vs. Aided Recall Unaided vs. Aided Recall QuestionsQuestions

Because respondents often forget, or fail to recall Because respondents often forget, or fail to recall accurately, their activities and behavior, you can use accurately, their activities and behavior, you can use aided recall questions to “jog” their memories.aided recall questions to “jog” their memories. Unaided recall: “Please list all of the Unaided recall: “Please list all of the

Communication classes in which you participated Communication classes in which you participated in learning groups last semester.” (open-ended in learning groups last semester.” (open-ended question)question)

Aided recall: “Please place a check by any of the Aided recall: “Please place a check by any of the Communication courses in which you participated Communication courses in which you participated in a learning group last semester.” (closed-ended in a learning group last semester.” (closed-ended question)question)

Public SpeakingPublic Speaking Introduction to Interpersonal CommunicationIntroduction to Interpersonal Communication Small Group CommunicationSmall Group Communication

22

Recall Questions and Time Recall Questions and Time FrameFrame

We often want respondents to focus on a We often want respondents to focus on a specific time frame when answering a specific time frame when answering a question.question. Avoid ambiguous terms like “past year” or Avoid ambiguous terms like “past year” or

“next year” because respondents may not “next year” because respondents may not know if this is a calendar year (i.e., 2006, know if this is a calendar year (i.e., 2006, 2007, 2008) or a 12-month period.2007, 2008) or a 12-month period.

Avoid time frames that are too large (for the Avoid time frames that are too large (for the recall question) or too far in the past.recall question) or too far in the past.

Write one recall question for gathering Write one recall question for gathering evidence for your SoTL research question. evidence for your SoTL research question.

23

Intimidating or Intimidating or Inappropriate QuestionsInappropriate Questions

Respondents will consider a question to Respondents will consider a question to be intimidating or inappropriate if:be intimidating or inappropriate if: It addresses highly personal and private It addresses highly personal and private

behavior (drinking behavior, sexual behavior (drinking behavior, sexual activity, illegal drug use, etc.)activity, illegal drug use, etc.)

It addresses subjects for which the It addresses subjects for which the respondent believes there are socially respondent believes there are socially desirable norms (communication with desirable norms (communication with instructors, behavior in class, spiritual instructors, behavior in class, spiritual behavior, etc.)behavior, etc.)

If respondents perceive a question to be If respondents perceive a question to be intimidating, they are more likely to intimidating, they are more likely to refuse to answer or to give an inaccurate refuse to answer or to give an inaccurate answer.answer.

24

Intimidating QuestionsIntimidating Questions

Would you use an open- or Would you use an open- or closed-ended question to ask a closed-ended question to ask a potentially intimidating question potentially intimidating question to gather evidence from your to gather evidence from your students?students? Examples? Examples?

25

Forms for Asking Potentially Forms for Asking Potentially Intimidating Questions About Intimidating Questions About

BehaviorBehavior If you ask a closed-ended question If you ask a closed-ended question

with a set of response options, with a set of response options, people typically do not choose the people typically do not choose the extreme options.extreme options.

Therefore, it is often better to use Therefore, it is often better to use an open-ended question and allow an open-ended question and allow the respondents to indicate how the respondents to indicate how often they engage in certain often they engage in certain behaviors, or define the terms as behaviors, or define the terms as they perceive them.they perceive them.

26

Forms for Asking Potentially Forms for Asking Potentially Intimidating Questions About Intimidating Questions About

BehaviorBehavior Third person questions ask Third person questions ask

respondents to report on the respondents to report on the behavior and activities of other behavior and activities of other people.people. Example: Instead of asking college Example: Instead of asking college

students directly about their alcohol students directly about their alcohol consumption behavior and its impact on consumption behavior and its impact on their learning, you could ask students to their learning, you could ask students to report how much alcohol their report how much alcohol their roommates consume and how that roommates consume and how that affects their learning behavior.affects their learning behavior.

27

Double-Barreled QuestionsDouble-Barreled Questions

Double-barreled questions ask Double-barreled questions ask respondents for their opinions about respondents for their opinions about two distinct issues in the same two distinct issues in the same question. question. ““Do you prefer working in groups and Do you prefer working in groups and

learning from your fellow students, or learning from your fellow students, or do you prefer lectures and learning do you prefer lectures and learning from the instructor?from the instructor?

Any other examples?Any other examples?

28

Leading and Loaded Leading and Loaded QuestionsQuestions

Some “surveys” seek to generate Some “surveys” seek to generate results that support a set of results that support a set of predetermined goals and predetermined goals and objectives.objectives.

How might we re-write a How might we re-write a question on the survey to make it question on the survey to make it a leading question?a leading question?

29

Survey Question DesignSurvey Question Design

30

Ordering of QuestionsOrdering of Questions Start with easy, salient, and non-Start with easy, salient, and non-

threatening questions first.threatening questions first. Complete questions on a single topic Complete questions on a single topic

before moving on to a new topic.before moving on to a new topic. When switching topics, use transitional When switching topics, use transitional

phrases to make it easier for phrases to make it easier for respondents to switch their train of respondents to switch their train of thought.thought.

Put demographic questions near the Put demographic questions near the end.end.

Do a quick evaluation of survey to see Do a quick evaluation of survey to see how it fits criteria.how it fits criteria.

31

Funnel QuestionsFunnel Questions Questions regarding a specific topic should usually Questions regarding a specific topic should usually

start with the most general questions and then start with the most general questions and then follow with increasingly more restrictive questions.follow with increasingly more restrictive questions. If you were required to work in groups in your If you were required to work in groups in your

class, what types of students would you want in class, what types of students would you want in your group?your group?

If you had to select just three key If you had to select just three key characteristics that you would use to choose characteristics that you would use to choose among these different students, what would among these different students, what would these 3 factors be?these 3 factors be?

Would “student’s GPA” be an important factor in Would “student’s GPA” be an important factor in your choice of students for your work group?your choice of students for your work group?

If your instructor indicated that she was going If your instructor indicated that she was going to put students in groups according to similar to put students in groups according to similar GPAs, would you be interested in working in GPAs, would you be interested in working in such a group?such a group?

32

Skipping and BranchingSkipping and Branching Surveys can instruct respondents Surveys can instruct respondents

to skip over certain questions if to skip over certain questions if these questions do not pertain to these questions do not pertain to them.them.

Question 7: “Have you worked in a Question 7: “Have you worked in a classroom learning group in the classroom learning group in the past semester?”past semester?” ____ Yes (please proceed to question 8)____ Yes (please proceed to question 8) ____ No (please skip to question 12)____ No (please skip to question 12)

Any examples on survey?Any examples on survey?

33

Demographic QuestionsDemographic Questions In most surveys, questions regarding In most surveys, questions regarding

demographic characteristics of the demographic characteristics of the respondents are included at end.respondents are included at end.

Typical demographic questions:Typical demographic questions: SexSex Ethnicity/Racial backgroundEthnicity/Racial background AgeAge Year in SchoolYear in School MajorMajor

Are there better ways to write these Are there better ways to write these demographic questions on the survey?demographic questions on the survey?

34

Wording of Survey Wording of Survey QuestionsQuestions

35

Avoid Ambiguity in Avoid Ambiguity in WordingWording

Some common words, such as “regularly” or Some common words, such as “regularly” or “often”, may be interpreted very differently by “often”, may be interpreted very differently by respondents.respondents. Do you go to talk to your instructor regularly Do you go to talk to your instructor regularly

(outside of class)?(outside of class)? How often do you go talk to your instructor (outside How often do you go talk to your instructor (outside

of class):of class): 1-2 times a week1-2 times a week 3-4 times a week3-4 times a week 5-7 times a week5-7 times a week More than 7 times a weekMore than 7 times a week

What might be the issue with these What might be the issue with these questions? questions?

36

Response Scales Used To Response Scales Used To Collect Survey DataCollect Survey Data

37

Features of Response Features of Response CategoriesCategories

Any response category must be Any response category must be mutually exclusivemutually exclusive.. There should not be any overlap in the There should not be any overlap in the

categories.categories. Any response category Any response category systemsystem should be should be

collectively exhaustivecollectively exhaustive.. All possible responses should be included in the All possible responses should be included in the

category category systemsystem.. Usually it is advisable to include an “other” Usually it is advisable to include an “other”

category to allow respondents to add a response category to allow respondents to add a response that is not included in the category system.that is not included in the category system.

On survey, evaluate questions for exclusivity On survey, evaluate questions for exclusivity and collective exhaustion. Any issues? and collective exhaustion. Any issues?

38

Semantic Differential Semantic Differential ScalesScales

A scale, typically used to measure A scale, typically used to measure attitudes or perceptions, that includes attitudes or perceptions, that includes end points that are anchored by bipolar end points that are anchored by bipolar adjectives.adjectives.

Outstanding _ _ _ _ _ _ _ Outstanding _ _ _ _ _ _ _ UnacceptableUnacceptable

StrongStrong _ _ _ _ _ _ _ _ _ _ _ _ _ _WeakWeakMoralMoral _ _ _ _ _ _ _ _ _ _ _ _ _ _ ImmoralImmoral

Any examples of a semantic differential Any examples of a semantic differential scale on survey? scale on survey?

39

Likert ScalesLikert Scales A scale, typically used to measure attitudes, that A scale, typically used to measure attitudes, that

usually includes end points of “strongly agree” and usually includes end points of “strongly agree” and “strongly disagree.”“strongly disagree.”

Abiding by page limits is important for writing a good Abiding by page limits is important for writing a good paper:paper:

Strongly AgreeStrongly AgreeAgreeAgreeNeutralNeutralDisagreeDisagreeStrongly DisagreeStrongly Disagree

Any examples of a Likert scale on the survey?Any examples of a Likert scale on the survey?

40

Numerical ScalesNumerical Scales A scale that uses multiple numbers, usually A scale that uses multiple numbers, usually

5 to 10, with anchored end points to 5 to 10, with anchored end points to represent the strength or position of represent the strength or position of respondents’ attitudes. respondents’ attitudes.

1 strongly disagree1 strongly disagree 2 disagree2 disagree 3 neutral3 neutral 4 agree4 agree 5 strongly agree5 strongly agree

Any examples of a numerical scale on Any examples of a numerical scale on survey?survey?

41

The Use of “Don’t Know” and “Not The Use of “Don’t Know” and “Not Sure” and “Not Applicable”Sure” and “Not Applicable”

Rather than have respondents feel Rather than have respondents feel forced to give an invalid response, forced to give an invalid response, researchers often add a category to researchers often add a category to scales that is labeled “don’t know” scales that is labeled “don’t know” or “not sure” or “not applicable”.or “not sure” or “not applicable”.

Write a question for your research Write a question for your research study that includes a response scale. study that includes a response scale. Then explain why you wrote the Then explain why you wrote the question using that response scale. question using that response scale.

42

43

Survey Design and Survey Design and PreparationPreparation

44

Length of SurveysLength of Surveys

How long respondents will spend How long respondents will spend answering a survey is directly related answering a survey is directly related to the saliency of the issues for them.to the saliency of the issues for them.

For salient topics, questionnaires of For salient topics, questionnaires of 10-15 pages are possible.10-15 pages are possible.

For non-salient topics, questionnaires For non-salient topics, questionnaires are usually limited to 2 to 4 pages.are usually limited to 2 to 4 pages.

How does this survey rate on these How does this survey rate on these criteria?criteria?

45

Introducing the SurveyIntroducing the Survey In a cover letter or introductory In a cover letter or introductory

comments, the following points comments, the following points should be addressed:should be addressed: What the study is about and its relevance to What the study is about and its relevance to

the participant;the participant; Why the respondent’s cooperation is Why the respondent’s cooperation is

important;important; Promise of confidentiality and security of Promise of confidentiality and security of

information (IRB rules and regulations must information (IRB rules and regulations must be followed);be followed);

Reward for participation (if relevant);Reward for participation (if relevant); Deadline for completion (if relevant);Deadline for completion (if relevant); Who to contact regarding any questions;Who to contact regarding any questions; Thank youThank you

Evaluate survey in terms of these criteriaEvaluate survey in terms of these criteria

46

Pretest, pretest, pretestPretest, pretest, pretest

You should always pretest draft versions You should always pretest draft versions of your questions with representatives of your questions with representatives from the target population.from the target population.

But never pretest your questionnaire with But never pretest your questionnaire with people who may be in your final sample of people who may be in your final sample of respondents.respondents.

Who might I ask to pretest this survey? Who might I ask to pretest this survey?

47

Online survey help sitesOnline survey help sites

Advanced SurveyAdvanced Survey ZoomerangZoomerang Web Online SurveysWeb Online Surveys SurveyMonkeySurveyMonkey

48

Final Comments on Survey Final Comments on Survey DesignDesign

Never underestimate the time to takes to Never underestimate the time to takes to create a high quality survey instrument.create a high quality survey instrument.

No survey is ever perfect.No survey is ever perfect. There is no substitute for pre-testing and There is no substitute for pre-testing and

soliciting feedback from other research soliciting feedback from other research experts.experts.

Shorter is always better.Shorter is always better. Only ask questions on surveys that are Only ask questions on surveys that are

directly related to specific and important directly related to specific and important research issues.research issues.

49

ReflectionReflection

How might I use questions to collect How might I use questions to collect evidence for my research project?evidence for my research project?

What will be most important for me What will be most important for me to remember about the questions I to remember about the questions I want to ask?want to ask?

50