SURVEYS: GETTING THE RELUCTANT TO DO THE IRRELEVANT FOR THE (MOSTLY) UNINTERESTED

Patrick Beard

Coordinator,

Academic Reviews and Planning

Brock University

But First, A Word from our Sponsor:

AGENDA

Maximizing Participation(Getting the Reluctant motivated)

Dealing With the Issue of Relevance(Why is this Survey Important?)

Making Results Meaningful(Why a “Data Dump” is Not Enough)

Maximizing Participation

Maximizing Participation

Maximizing Participation

Maximizing Participation

Pina McDonnellAdministrative AssistantDepartment of Mathematics

Look for “secret weapons:”

Mathematics Academic Review Survey:

50.7% (vs. 41.3% overall)

Relevance

How do we make the survey relevant?

by

Making it important.

Relevance

“A survey is a transaction. To get students’ attention, administrators must entice them.”

Sara Lipka, Want Data? Ask Students. Again and Again.

Relevance IBM's e-business Innovation Centre in Toronto ran an online business-to-business survey between 2000 and 2001, and after tweaking a number of variables they managed to double their response rates in the second year of running the survey. Here are some of the lessons they learnt:

- Survey length - Be honest about how long the survey will really take - Provide value, value, value - In our experience, online survey response rates increase dramatically when the participant gains value from responding. - Send the survey mid-week, during mid-afternoon - Use 1 reminder e-mail to the survey invitation - Allow for some open ended questions

http://www.peoplepulse.com.au/Survey-Response-Rates.htm

Relevance

Hot Tip: Make it MatterCommunicate the relevance and value of your survey to your participants. In your invitation, let participants know what the survey’s purpose is, how the results will benefit them, and how much their opinions matter!

Jessica Foster on Maximizing Survey Response Rates(http://aea365.org/blog/?p=3302)

Interpreting Results

•Dangers of the “Data Dump”• Interpretation helps comprehension• Interpretation helps address issues• Interpretation takes time and effort

Interpreting ResultsAcademic Review Survey of Current Majors:

Philosophy The following are the results of a student survey conducted between October 4th and November 12th, 2010. A total of 82 students with the major code of PHIL were invited to participate. Excluded from the sample were new (in fall of 2010) full-time Year I students. A total of 42 students (51.2% of the sample) completed the survey. Forced-Choice Questions For virtually all questions (the exception is 14a), students are asked to respond to each question on the basis of a five-point scale: 1. Strongly Disagree 2. Disagree 3. Unsure/Neutral 4. Agree 5. Strongly Agree They are also given the option of responding “Not Applicable”. 1a. My academic program contribute to developing my strength in A broad

spectrum of knowledge Mean

N Valid 42

Missing 0

Mean 4.29

Frequency

Frequency Percent Valid Percent

Cumulative

Percent

Valid Strongly Disagree 1 2.4 2.4 2.4

Disagree 1 2.4 2.4 4.8

Unsure/Neutral 2 4.8 4.8 9.5

Agree 19 45.2 45.2 54.8

Strongly Agree 19 45.2 45.2 100.0

Total 42 100.0 100.0

Interpreting Results

Commentary on Philosophy Data Survey of Current Majors

Forced-Choice Questions Question 1 Academic program contributed to: Mean % Agree & Strongly Agree Broad spectrum of knowledge 4.29 90% Research skills 3.62 57% Problem solving skills 4.19 86% Analyzing ideas, experiences, methods, data 4.33 85% Synthesizing ideas, information, or experiences 4.31 88% Making judgments 4.38 91% Understanding application of knowledge 3.90 71% Working with other people 3.22 41% Written communication skills 4.33 93% Oral communication skills 3.71 74% Recognition of limits to knowledge 4.05 83% Comment: Anything with a mean less than 4.00 should be addressed.

Interpreting Results

Philosophy Self Study (Version 1.0):

c) Working with other people. Generally speaking, working with other people is not something which is encouraged by Departments within the Humanities; working independently is. Students, for example, are warned not to write their essays as members of a committee but to write them independently. And joint projects in which two or more students work together and the group as a whole receiving a single grade is not a strategy found within the Department of Philosophy.

You can lead a horse to water….

Interpreting Results

Don’t allow the numbers to speak for themselves. Numeric results and statistical comparisons are more accessible when accompanied by an explanation and interpretation of what can and cannot be concluded from them.

Using NSSE Datahttp://www.nsse.iub.edu/2011_Institutional_Report/pdf/Using_NSSE_Data.pdf

And remember….

If at first you don’t succeed,

destroy all the evidence that you tried.

and…

If you can’t convince them,

confuse them.

Questions/Discussion?