Upload
sana
View
41
Download
0
Tags:
Embed Size (px)
DESCRIPTION
"Developing Surveys to Measure Student Satisfaction and Learning Outcomes“ Nathan Lindsay & Larry Bunce February 19, 2014. www.campuslabs.com /blog. @ CampusLabsCo # labgab. Like us on Facebook!. Satisfaction Surveys Learning Outcome Surveys Needs Assessments Exit Surveys Alumni Surveys - PowerPoint PPT Presentation
Citation preview
"Developing Surveys to Measure Student Satisfaction and Learning Outcomes“
Nathan Lindsay & Larry Bunce
February 19, 2014
@CampusLabsCo #labgab Like us on Facebook!www.campuslabs.com/blog
There are many types of surveys to consider… Satisfaction Surveys Learning Outcome Surveys Needs Assessments Exit Surveys Alumni Surveys User Surveys Non-User Surveys Student/Faculty/Staff/General Public Surveys Other?
Steps in survey design
Pilot test survey and revise
Review and revise survey
Determine physical characteristics of survey
Determine item sequence
Write and edit items
Choose response formats
Outline topic(s) and draft items
• Determine your purposeDESIGN
Begin with the end in mind…
What do you want/need to show?
Why do you need to show it?
Who is the source of your data?
How will you use the data?
Who will need to see results?
The purpose of this assessment is…
• To better understand what the needs of our veteran students are, and how the new Veteran’s Center can meet them
• To evaluate if students achieved the stated learning outcomes of our workshop, and what additional training needs they have
• To demonstrate to stakeholders the impact that living in the residence halls has on student development
• To assess student awareness of services in order to develop our marketing and communications plan
• Determine your purposeD• Examine past assessmentsE
SIGN
Examine past assessments
• Did you use the data?o If not, what kept you from examining it?
• If you used the data…o What was useful?o Was any of the data difficult to analyze?o Were there questions you wished you had asked?o Did any question wording make you unsure of what the data
meant?
• What feedback did you receive from those who participated?
Those who cannot remember the past are condemned to repeat it.Philosopher, George Santayana 1905
• Determine your purposeD• Examine past assessmentsE• Select the appropriate methodS
IGN
Select an appropriate method
Indirect vs. direct
Quantitative vs.
qualitative
Formative vs.
summative
Population vs. sample
Quantitative Qualitative• Focus on numbers/numeric values• Who, what, where, when• Match with outcomes about
knowledge and comprehension (define, classify, recall, recognize)
• Allows for measurement of variables• Uses statistical data analysis• May be generalize to greater
population with larger samples• Easily replicated
• Focus on text/narrative from respondents
• Why, how• Match with outcomes about
application, analysis, synthesis, evaluate
• Seeks to explain and understand• Ability to capture “elusive” evidence
of student learning and development
Sampling
PopulationThe whole group
Example: survey goes to entire campus
If entire campus: use sparingly and coordinate with Institutional Research
SampleA subsection of that group
Example: survey goes to 30% of campus
Sampling strategies
Simple Random Sample gives everyone in
sampling population an equal chance of
selection; a probability sample
Stratified Random Sample
breaks total sample into subpopulations and then
selects randomly from each stratum
Sample suggestionsNumber of Students in Population Random Sample Size (suggestion)
1,000 278
500 217
350 184
200 132
100 80
50 44
Based on 5% margin of errorSuggestion in: Assessing Student Learning by Linda Suskie
Sample size is the desired number of respondents NOT the number of individuals invited to participate.
Any process employed to gather data which requires
subjects to display their knowledge, behavior, or
thought processes.
Direct Methods Indirect Methods
Any process employed to gather data which asks subjects to reflect upon
their knowledge, behaviors, or thought processes.
Where on campus would you go or who would you consult with if youhad questions about which coursesto register for the fall?
I know where to go on campus if I have questionsabout which courses to register for in the fall.
Strongly agreeModerately agreeNeither agree nor disagreeModerately disagreeStrongly disagree
• Conducted during the program• Purpose is to provide feedback• Use to shape, modify or improve
program
• Conducted after the program• Makes judgment on quality, worth, or
compares to standard• Can be incorporated into future plans
FormativeSummative
Is a survey right for you?
Pros:• Include large numbers• Relatively fast and easy
to collect data• Lots of resources
available• Requires minimal
resources• Fast to analyze• Good for surface level or
basic data
Cons:• Survey fatigue and
response rates• Non-responsive • Limited in type of
questions asked• Lacks depth in data• Skills set in both
designing questions and analyzing data properly
Focus Groups
• Group discussions where the facilitator supplies the topics and monitors the discussion.
• The purpose is to gather information about a specific (or focused) topic in a group environment, allowing for discussion and interaction by participants.
• Similar to interviews, but use when the group interaction will give contribute to a richer conversation
Is a focus group right for you?
Pros:• Helps to understand
perceptions, beliefs, thought processes
• Small number of participants
• Focus groups encourage group interaction and building upon ideas
• Responsive in nature• Relatively low costs
involved
Cons:• Getting participants (think of
time/places)• Data collection and analysis
takes time• Data is as good as the facilitator• Beware of bias in analysis
reporting• Meant to tell story, may not help
if numbers are needed
• Data is not meant to be generalizable
Quick, 1-Minute Assessments
On a notecard, write a real-world example of how you can apply what you learned.
Pass an envelope containing notecards with quiz questions. Students pick one and have 60 seconds to answer and pass along.
At the end of a workshop, ask students to write down 1 thing they learned, and 1 lingering question.
Is a quick assessment right for you?
Pros:• Provides a quick summary of take
away from student perspective• Quickly identifies areas of
weakness and strengths for formative assessment
• Can track changes over time (short-term)
• Non-verbal (provides classroom feedback from all students)
• Captures student voice• Short time commitment
• Provides immediate feedback
Cons:• Non-responsive• Short (so you may lose
specifics)• Sometimes hard to interpret• Need very specific prompts
in order to get “good” data• Plan logistics ahead of time
and leave time during program/course
• May need to be collected over time
Mixed Methods
Look for same results across multiple data collections
Build upon or relate results from one assessment to another
Use data from one method (e.g., a survey) to inform another method (e.g., a focus group)
Able to increase scope, number, and types of questions
•Determine your purposeD•Examine past assessmentsE•Select the appropriate methodS•Identify ethical/logistical considerationsI
GN
Identify ethical/logistical considerations
Do you have the necessary resources and brain power? Do you need to go through IRB? Do you need to identify respondents for follow up,
merging of data, tracking of cohorts, or pre/post analysis? Do you need to include demographic questions to drill
down or separate data? Who needs to be involved at planning stage to avoid
problems when results are in? Does anyone need to approve the project?
Are there any political issues to be aware of?
•Determine your purposeD•Examine past assessmentsE•Select the appropriate methodS•Identify ethical/logistical considerationsI•Generate the best question and answer formatG
N
What to consider• Scales that match
• Mutually exclusive
• Exhaustive
• Neutral/Not applicable/Non-response options• Choose not to respond Don’t know• Not applicable Unable to judge• No opinion Neutral• Neither ___ nor ___
Pairing Question Text with Answer ChoicesQuestion text should be compatible with the answer choices
e.g., “How satisfied were you with the following?”
e.g., “Did you enjoy the Black History Month speaker?”Strongly agreeSomewhat agreeSomewhat disagreeStrongly disagree
Excellent Good Fair PoorMeals at the conferenceLocation of the conferenceDate of the conference
Mutually Exclusive Answer Choices
Response options should never overlape.g., How many hours per week do you work?
0-1010-2020-3030-40
Response options should exist independently of one another
e.g., Which of the following statements describes your peer mentor?He/she is helpful and supportiveHe/she is difficult to get a hold of
Exhaustive Answer Choices
Respondents should always be able to choose an answer
e.g., How often do you use the University website?Daily2-3 times a weekWeeklyMonthly
Non-response optionsAlways consider a non-response option
Choose not to respond Don’t knowNot applicable Unable to judgeNo opinion NeutralNeither ___ nor ___
Customize the non-response option when possiblee.g., How would you rate the leadership session?ExcellentGoodFairPoorDid not attend
Pitfalls to avoidSocially desirable responding – based on social norms
– Can never be eliminated– Consider sensitive topics like race, drug and alcohol use, sexual activity,
and other areas with clear social expectations
Leading questions – suggesting there is a correct answere.g., “Why would it be good to eliminate smoking on campus?”
Double-barreled questions – asking more than one questione.g., “What were the strengths and weaknesses of orientation?”
Double negatives – including negative phrasing which makes responding difficult
e.g., “I do not feel welcome in my residence hall.”
Response Formats• Open ended responses
– Free response - text– Numeric– Yes/No with please explain
• Types of multiple choice responses– Yes/No– Single response– Multiple response (e.g., Check all that apply, Select 3)– Ranking– Scales
Yes/NoWhen to Use:
– There is no response between “Yes” and “No”e.g., “Have you ever lived on campus?”
– You consciously want to force a choice even if other options might existe.g., “Would you visit the Health Center again?”
When Not to Use:– There could be a range of responses
e.g., “Was the staff meeting helpful?”
Single response
When to Use:– All respondents would only have one response
e.g., “What is your class year?”
– You consciously want to force only one responsee.g., “What is the most important factor for improving the Rec
Center?”
When Not to Use:– More than one response could apply to respondents
e.g., “Why didn’t you talk to your RA about your concern?”
Multiple responseOptions: “Check all that apply” or “Select (N)”
When to Use:– More than one answer choice might be applicable
e.g., “How did you hear about the Cultural Dinner?” (Check all that apply)
– You want to limit/force a certain number of responsese.g., “What were your primary reasons for attending?” (Select up to 3)
When Not to Use:– It’s important for respondents to only be associated with one
responsee.g., “What is your race/ethnicity?”
RankingWhen to Use:
– You want to see the importance of items relative to one another
e.g., “Please rank how important the following amenities are to you in your residence: (1=most important)”
– You are prepared to do the analysis and interpretation!
When Not to Use:– You want to see the individual importance of each item
e.g., “How important are the following amenities to you?”
ScalesWhen to Use:
– You want to capture a range of responsese.g., “How satisfied were you with your meeting?”
– When you would like statisticse.g., 4 = strongly agree
3 = agree2 = disagree1 = strongly disagree
When Not to Use:– The question is truly a Yes/No question
e.g., “My mother has a college degree.”
ScalesUnipolar – no negativeA great dealConsiderablyModeratelySlightlyNot at all
Bipolar – positive or negative (with or without a midpoint)
Very safeSomewhat safeSomewhat unsafeVery unsafe
Consider…• Number of points• Inclusion of neutral• Whether labels are needed• Order (e.g., 1, 2, 3, 4, 5 or 5, 4, 3, 2, 1)
Recommended Scales
•Determine your purposeD•Examine past assessmentsE•Select the appropriate methodS•Identify ethical/logistical considerationsI•Generate the best question and answer formatG•Note the purpose for each data pointN
Note the reason for each data point• Bubble next to question• Compare against
purpose to identify gaps• Look for overlap• Eliminate “nice to know”• Help with ordering• Retain for analysis step
DATA COLLECTION METHODS
Paper surveys
Things to consider• Captive audience• Administrator available for
questions• No technology issues or
benefits• Data entry necessary
Web surveys
Things to Consider:• No data entry• Technology issues and
benefits• Immediate results• Can be anonymous or
identified• Not a captive audience
Data Collection MethodsPros Cons
Web
No data entry Accuracy is excellent Technology benefits (e.g., display
rules, required questions) Immediate results Anonymous
Audience is not usually captive Possible misinterpretation (can’t ask ?s) Technology issues Response sample unrepresentative
Mobile
No data entry Accuracy is good Technology benefits (e.g., display
rules, required questions) Captive audience Administrator is available for ?s
Technology issues Response sample unrepresentative Limited formatting Anonymity is questionable
Paper
Captive audience Administrator is available for ?s No technology issues
No benefits of technology Accuracy can be compromised Data entry necessary Anonymity is questionable
SURVEY FATIGUE
General information
• Survey response rates have been falling o Difficult to contact peopleo Refusals to participate increasing
• Strategies for correcting low response rates:o Weight the data for non-responseo Implement strategies to increase response rates
Non-response may not be random• Correlation exists between demographic characteristics
and survey response• Higher response has been found among certain sub-
populations:o Womeno Caucasianso High academic abilityo Living on campuso Math or science majors
• Research is inconsistent
IMPROVING RESPONSE RATES
Specific techniques
• Survey length• Preannouncement• Invitation text• Reminders• Timing of administration• Incentives• Confidentiality statements• Salience• Request for help• Sponsorship• Deadlines
Survey length
• Greater attrition at 22 questions or 13 minutes
• What to consider:– Excluding “nice to know”– Eliminate what you already know– Outlining how results will be used– Number of open-ended questions– Number of required questions
Invitations
• Importance/Purpose• Relevancy to respondent• Request for help• How and by whom results are used• How long it will take to respond• Deadline• Incentives/Compensation• Contact information
Timing of contact/administration• Avoid busy times or
holidays• Send email/
preannouncement 2-3 days prior to survey mailing
• First half of semester/term may be better if you are surveying in an academic environment
Piloting
• 1. Take it as if you were respondent
• 2. Seek reviews from colleagues with no prior knowledge
• 3. Administer to sample of actual population being studied– Focus group– Questions at end of survey– Observing
Reliability & Validity
• Reliability – yielding the same results repeatedly– Test/Re-test – consistency over time– Inter-rater – consistency between people
• Validity – accurately measuring a concept– Internal – confidence results due to independent variable– External – results can be generalized – Face validity – does this seem like a good measure?
• If a survey is valid, it is almost always reliable!
USING THE SURVEY RESULTS
PR or advertising campaign
Results in survey invitations
Some Final Advice
• Google your area to see what other surveys have been conducted
• Contact Larry Bunce (Director of Institutional Research) or Nathan Lindsay for help in designing your survey
• Online surveys should be coordinated through the Office of Institutional Research
Nathan LindsayAssistant Vice Provost for AssessmentUniversity of Missouri-Kansas [email protected] 816-235-6084
QUESTIONS?