Upload
chessa
View
27
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Using Surveys to Support OD projects. Mick Verran for Boston Facilitators Roundtable 02-07-2014 Handout www.MickVerran.com Password BFR. This work is licensed under a Creative Commons Attribution- NonCommercial - ShareAlike 3.0 Unported License . . - PowerPoint PPT Presentation
Citation preview
Using Surveys to Support OD projects
You are free to copy and distribute the work under the following conditions: you must attribute the work to “Mick Verran [email protected]” ; you may not use the work for commercial purposes; if you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.
Mick Verran for Boston Facilitators Roundtable
02-07-2014
Handout www.MickVerran.com Password BFR
Issues in gathering data
Sensitivities about the role of an outside ‘expert’Impact of the intervention itself
Barriers to participation – distribution, trustChallenges in balancing the loudest voices
Availability & accessibility of usable dataWhole or representative population required
Degree of accuracy required
Funding & time available
2 Using Surveys to Support OD Projects Mick Verran
Data-gathering methods
3 Using Surveys to Support OD Projects Mick Verran
Pros Cons
Observation, Test, Simulation
High face-value.Allows for probing & clarification.Observe hidden dynamics. More accurate than e.g. recall. Comparability.
Skilled, knowledgeable observer.Time-consuming, high $.
Personal Interview
Follow-up to nuance data.Unexpected information.
> Quality < Quantity. Time consuming.Analysis and comparisons harder.
Focus Group Interaction between viewpoints.Buy-in.
Time-consuming.Quantifying data.Skillful facilitator.
(Electronic) Survey
Low $.Reach large or distributed population.Easy for respondents.Easy to nudge completion.Higher response than paper.Easy data analysis & reporting.
Requires internet access.No opportunity for clarifying or probing.(Rare) concerns about anonymity.
Survey tools
Constant Contact
SearchGizmo
Capricity
Qualtronics – QuestionPro – Vertical Response
‘… where I do not speak the language’http://qualtrics.com/ https://www.questionpro.com/
‘Analysis, vendors who specialize in analysis and reporting, current tools and trends, live poling, web-based live poling’www.DiscoverSurveys.com – Bruce Katcher
4 Using Surveys to Support OD Projects Mick Verran
‘How others are using surveys’
5 Using Surveys to Support OD Projects Mick Verran
Download Handout 1
‘Other ways to leverage surveys’
6 Using Surveys to Support OD Projects Mick Verran
Enter, Scope
Initiate, Analyze
Design, Plan
Implement
Sustain, Exit
M O V E
Motivation
Ownership
Voice Expectation
(Role = moving the process along)
Download Handout 2
‘What makes for a good survey?’
Everyone concerned agrees that the survey asks
an important question that each respondent you
need to hear from will understand, know their
answer to, and be willing and able to answer, to
produce information you can reliably interpret and
use.
7 Using Surveys to Support OD Projects Mick Verran
Online survey workload
Activity
Agree survey goals, measures, population, design considerations, follow-up actions, scheduleDraft questions, intro & nudge emails, example reportReview with client
Create online survey
Launch
Monitor & nudge
Draft results
Review results with client
Finalize report of results
8 Using Surveys to Support OD Projects Mick Verran
Additional Workload
-
2 hours
-
< 1 hour
-
< 1 hour
1 hour
-
< 1 hour
Activity Additional Workload Watch-out for ….
Agree survey goals, measures, population, design considerations, follow-up actions, schedule
- • Vague follow-up actions• Push for demographics• Shift patterns
Draft questions, intro & nudge emails, example report 2 hours
Review with client - • ‘Scope creep’
Create online survey < 1 hour
Launch -
Monitor & nudge < 1 hour
Draft results 1 hour
Review results with client - • Heavy-handed editing• Vague follow-up actions
Finalize report of results < 1 hour
Needs of interested parties1. Inform the content of what you’re doing
2. Move the process along
Client• Validation• Efficiency • > Participation• Temperature check
Survey Participants• A voice• Understand what the others involved think.
Consultant• Reach around the client to uncover hidden issues• Platform for success• Attune to the language and issues
9 Using Surveys to Support OD Projects Mick Verran
Applications – Goals
10 Using Surveys to Support OD Projects Mick Verran
Download Handout 3
Response rates
Online, ‘out of the blue’ market research = 30% averageOur pre-survey = 27/30 = 90%
What impacts the response rate?• Connected with issues of importance to the respondents.• Promise of report back on results. • Credible commitment of the sponsor to address issues.• Confidence in confidentiality.• Accommodation of e.g. shift schedules.• Easy-to-answer.
– Questions & scales– Goal; 20 questions or 5 minutes (5 closed or 2 open per minute)
11 Using Surveys to Support OD Projects Mick Verran
Choosing respondentsWho to ask?Stakeholders – Knowledgeable – Opinion Leaders
‘Gathering data from a large sample…’http://www.calculator.net/sample-size-calculator.html
‘Surveying groups for the same project with variants of the survey’Separate surveys – no value in complexities of creating too much conditional/skip logic/branching/piping
12 Using Surveys to Support OD Projects Mick Verran
Employee EngagementExperiencesWhich ‘drivers of engagement’ do you experience?CommitmentHow committed are you to our new strategy?ConfidenceHow confident are you that we will successfully execute our new strategy?ConcernsWhat concerns do you have about our new strategy?What concerns do you have about our ability to successfully execute our new strategy?
13 Using Surveys to Support OD Projects Mick Verran
Organization/Team EffectivenessExperienceHow often does the team ……?EvaluationHow well does the team do ……?SatisfactionHow satisfied are you with the way the team ………?Opportunities for improvementImprovements in which of the following would have a significant impact on the team’s effectiveness?
14 Using Surveys to Support OD Projects Mick Verran
Training Needs AnalysisRelevanceHow often do you use this skill at work? ImportanceHow important is this skill to success in your work?CompetenceHow skilled are you in ……?Development NeedI need to improve my ability to ….Commitment to ImproveHow likely are you to attend ….?
15 Using Surveys to Support OD Projects Mick Verran
Applications – Measures
16 Using Surveys to Support OD Projects Mick Verran
Download Handout 3
Survey sections
1. Why doing survey, benefits, how results will be used
2. Time commitment, when required3. Contacts – content AND completion4. Call for Honesty – use full scale, specific examples5. Confidentiality framing6. Demographic data7. Questions8. Open ended comments 9. Thanks
17 Using Surveys to Support OD Projects Mick Verran
‘Collecting demographic info’
Only those that are vital to the achievement of your objectivesNot so narrow that one can guessOptional space for self-identification if open to follow-up SurveyMonkey tracks IP address; individual tracking requires unique survey
Positioning Demographic questionsAt the end of the survey, when respondents have developed confidence in the survey and it’s objective
OR
Early, straight after the reassurance of anonymity, so don’t feel cheated if asked at the end
18 Using Surveys to Support OD Projects Mick Verran
So many options!How many times in the last (period of time) have you bought a Coca-Cola?01 – 56 – 12More than 13How much do you enjoy Coca-Cola?I enjoy Coca-ColaI neither enjoy nor do not enjoy Coca-Cola I do not enjoy Coca-Cola
Given a choice, which would you choose?Coca-Cola PepsiI prefer Coca-Cola to Pepsi; Strongly disagreeDisagreeNeither agree nor disagreeAgreeStrongly agree If you were to buy 10 soft drinks, how many of each would you buy?Coca-Cola Pepsi
I have chosen Coca-Cola over Pepsi;AlwaysOftenAbout half and halfRarelyNever I enjoy Coca-Cola;More than I enjoy PepsiAs much as I enjoy PepsiLess than I enjoy PepsiPut these soft drinks in order of your preference; Coca-Cola PepsiMountain DewAre you willing to; Drink Coca-ColaBuy Coca-ColaOffer Coca-Cola to your guestsBuy Coca-Cola for your children Consult to Coca-Cola Inc.Invest in Coca-Cola Inc.Work for Coca-Cola Inc.?
19 Using Surveys to Support OD Projects Mick Verran
Question types
20 Using Surveys to Support OD Projects Mick Verran
Download Handout 4
Respondent TendenciesAgree with statements as presented.Respond from general opinion of positives & negatives.
Have own definition of e.g. ‘frequently’ v. ‘occasionally’.Avoid using extreme response categories/poles.“I Tend to Disagree” is more negative than “I Tend to Agree” is positive.
Portray selves or group in a more favorable light when asked about; • Feelings of powerlessness, "kindness" or "benevolence“, honesty,
willingness to contribute.• Ability to do job, stress levels, company loyalty, value of own
contribution, compliance with procedures.
21 Using Surveys to Support OD Projects Mick Verran
How the question and the scale work togetherHow important are the following to the success of a restaurant?
Very Unimportant
Somewhat Unimportant
Neither Important Nor Unimportant
Somewhat Important Very Important
ServiceParkingCost
22 Using Surveys to Support OD Projects Mick Verran
How important are the following in attracting families to a restaurant?
Very Unimportant
1 2 3 4Very Important
5
Quality of ServiceEase of ParkingValue for MoneyI take the following into account when choosing a restaurant for a family outing;
I Strongly Disagree I Mostly Disagree I Neither Agree
Nor Disagree I Mostly Agree I Strongly Agree
Quality of ServiceEase of ParkingValue for Money
‘Preventing bias’ – ‘Examples of what not to do’
Reliability – Problem questions & scales
23 Using Surveys to Support OD Projects Mick Verran
Download Handout 5
Anchored rating scales
Process X produces an acceptable quality output.Strongly AgreeSomewhat AgreeNeither Agree nor DisagreeSomewhat DisagreeStrongly Disagree
24 Using Surveys to Support OD Projects Mick Verran
Effectively examine events issues, problems and generate solution alternatives.
Strongly Agree
Somewhat Agree
Neither Agree nor Disagree
Somewhat Disagree
Strongly Disagree
Behaviorally-anchoredEffectively examine events issues, problems and generate solution alternatives
Strongly Agree 1 Makes sound decisions on everyday issues and problems.
Somewhat Agree 2
Neither Agree nor Disagree 3 Makes sound decisions on complex functional issues and problems.
Somewhat Disagree 4
Strongly Disagree 5 Makes decisions that take into account broad, system-wide implications
Anchored thru question AND scaleThe output of Process X compared with our competitors is;Far Above AverageSomewhat Above AverageAverageSomewhat Below AverageFar Below Average
Rating Scale Template – ‘gold standard’
25 Using Surveys to Support OD Projects Mick Verran
Download Handout 6
Favorite questions & scales
26 Using Surveys to Support OD Projects Mick Verran
Download Handout 7
Questions about scales 1How many points should there be on the scale?• How many degrees of difference can respondents discriminate?• How many responses will give rise to different actions?
Should I include a mid-point?• Use is heavily influenced by “social desirability bias”.• “… a four point scale without a mid-point appears to push more respondents
towards the positive end of the scale.”• (a four point scale) “… can … (force) subjects to indicate their opinion when
they actually have no opinion and are not simply reluctant to reveal it.”• “This study found the opposite … the way in which people will respond to a
balanced Likert type scale without a mid-point is content-specific.”• (To minimize the use of the mid-point) “… either not include it at all or use
scales with many points.”
27 Using Surveys to Support OD Projects Mick Verran
Questions about scales 2Should I start with the “agree” or “disagree” options?• Some evidence of a small bias towards left-side options.• No evidence of significant bias whether start with positive option or
negative in a survey (different when face-to-face due to “social desirability bias”).
Should I give people the option to NOT respond to a question?• I Don’t Know – I Have No Evidence – I Have No Experience – I Have No
Opinion.• SurveyMonkey: “By allowing respondents to opt out due to non-familiarity
with the question or if it does not apply to their situation, this will help increase the response rate and quality of the collected data.”
• Exclude the question if too many respondents will be unable to respond. • How will high “don’t know” responses impact your interpretation of the
results?• Omit if it’s impossible NOT to have an opinion on or experience of the issue.
28 Using Surveys to Support OD Projects Mick Verran
Sequencing the questions“Respondents asked to answer questions dealing with fraud and waste in government programs, were subsequently MORE likely to oppose welfare programs than those who were asked the questions in the reverse order.”Meeting Review • Content we covered• Actions you will take • What’s left to be done• (Logistics)
29 Using Surveys to Support OD Projects Mick Verran
Team Assessment• Goal clarity • Results• Roles &
responsibilities• Goal clarity
• Processes • Roles & responsibilities• Resources • Processes• Results • Resources
Ok, …. one ‘Tip on using SurveyMonkey’
To allow multiple entries from same computer or IP address
1. Create the survey2. Click ‘Collect Responses’3. Select ‘Web Link’ & click ‘Next Step’4. Click ‘Change Settings’5. Under ‘Allow Multiple Responses’ click ‘Yes’6. Click ‘Save Settings’
30 Using Surveys to Support OD Projects Mick Verran
Deployment schedule (time zones, shift patterns)
You Sponsoring ManagerLaunch -2 days - Heads-up
Day 1 a.m. - Distribute link; 3-day deadline
Day 2 a.m. Send manager x/n - Day 2 p.m. - NudgeDay 3 a.m. Send manager x/n - Day 3 p.m. - Extend deadline by one dayDay 4 a.m. Send manager x/n - Day 4 noon - Count down to deadline
Day 5 a.m.Send manager x/n(Leave open for stragglers)
Thank you
Day 5 p.m. Draft report -
31 Using Surveys to Support OD Projects Mick Verran
Download Handout 8
Analyzing responses
‘Evaluating data based on … demographics’• SurveyMonkey filters
What to do about missing responses• What response rate & coverage is critical to the
goals?
32 Using Surveys to Support OD Projects Mick Verran
‘Communicating results to the event sponsor’ ‘Improving the presentation of data’
http://www.edwardtufte.comBoston, March 11–14 $380
Reporting principles
Separate results & interpretation.
Low response; how representative are the responses you got?
‘Use-by’ date on the report?
Summarize themes in open-ended comments.Respondents’ own words – including typos.
.pptx for the report, .docx for the record.Present all the data at end in a Table.
34 Using Surveys to Support OD Projects Mick Verran
Means, standard deviations, confidence interval
A = 2.60 : B = 2.95 : 5 point scale : 12 respondents
Very Strongly Agree is not 20% > Mostly Agree
“I can do that for you, but I’m afraid I’ll have to add $nx2 to my bill”.
35 Using Surveys to Support OD Projects Mick Verran
Presenting the results
Responses
Strongly Agree
Mostly Agree
Mostly Disagree
Strongly Disagree
Q1Q2Q3Q4Q5Q6Q7Q8Q9
36 Using Surveys to Support OD Projects Mick Verran
Q Mean/4.00
Q1 2.50Q2 2.50Q3 2.50Q4 2.75Q5 2.75Q6 3.00Q7 3.00Q8 3.25Q9 3.25
Number of Responses to each Rating Option
4 Strongly
Agree
3 Mostly Agree
2 Mostly Disagre
e
1 Strongly Disagre
e
0 2 2 02 0 0 21 1 1 10 3 1 01 2 0 10 4 0 02 1 0 11 3 0 03 0 0 1
Format options
All responses• Bar chart
% responses• Pie chart
Changes over time• Line graph• Side-by-side bar chart
Comments• Verbatim• Themes• Word frequency
www.Wordle.net
You can struggle with Excel & CSv or PDFs or …..Manual Templates!
37 Using Surveys to Support OD Projects Mick Verran
Priorities for a new CEOBusiness Development
Creating Bank’s vision for growth
Generating new sources of revenue
Capturing new customers
Differentiating Bank through customer service & marketing
Operations Management
Protecting capital & asset quality
Managing the regulatory process
Restructuring operating costs
Generating capital for investment
Organization Leadership
Developing an effective executive team
Attracting & developing leadership talent
Maintaining a healthy culture
External Relations
Engaging in the political process
Connecting with the local community
Retaining the confidence of the Board
38 Using Surveys to Support OD Projects Mick Verran
Hospital Housekeeping1 2 3 4
N/CommentI Strongly Disagree I Somewhat Disagree I Somewhat Agree I Strongly Agree
Housekeeping staff adhere to all health and safety standards.
5 11 10 1 1
17% 39% 36% 4% 4%
Housekeeping staff act appropriately toward patients and visitors.
4 1 17 4 2
14% 4% 61% 14% 7%
Housekeeping staff interact collaboratively with medical staff.
5 5 16 0 2
18% 18% 57% 0% 7%
Housekeeping staff respond flexibly to changing needs.
8 14 5 0 1
28% 50% 18% 0% 4%
Housekeeping Supervisors are responsive to Unit needs.
6 12 10 0 0
21% 43% 36% 0% 0%
Housekeeping staff respond in a timely manner to urgent needs.
9 14 5 0 0
32% 50% 18% 0% 0%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
... adhere to allhealth and safety
standards.
... act appropriatelytoward patients
and visitors.
... interactcollaboratively with
medical staff.
... respond flexiblyto changing needs.
... respond in atimely manner to
urgent needs.
No OpinionI Strongly AgreeI Somewhat AgreeI Somewhat DisagreeI Strongly Disagree
39 Using Surveys to Support OD Projects Mick Verran
Team surveys
40 Using Surveys to Support OD Projects Mick Verran
Team Effectiveness
Initiative Implementation
Team survey with leader’s response
41 Using Surveys to Support OD Projects Mick Verran
360 feedback
42 Using Surveys to Support OD Projects Mick Verran
Repeat 360
43 Using Surveys to Support OD Projects Mick Verran
January April July
Other
s
4
Other
s
4
Others
4
3 3 3
2 2 2
1 1 1
N R O F N R O F N R O F
January April July
Other
s
4
Other
s
4
Others
4
3 3 3
2 2 2
1 1 1
N R O F N R O F N R O F
1 Making condescending or dismissive statements N = Never R = Rarely O = Often F = Frequently
2 Talking over and/or interrupting people N = Never R = Rarely O = Often F = Frequently
‘How to summarize open-ended data’ 1
Read through once
List the themes
Cut ‘n’ paste sentences to themes
Cluster overlapping themes
Editing decisions; • Every theme? • Every sentence, or representative quotes?• Edit easily-identified sources?• Omit obvious grudges?• Edit same point by same person over and over?• Correct spelling, grammar?
44 Using Surveys to Support OD Projects Mick Verran
Strengths• Leadership Team members step up & take a leadership role when required.• Leadership Team meetings & conference calls are effective. • Leadership Team members collaborate with & support each other.• Leadership Team members give & receive feedback without becoming
defensive or aggressive.• XX is willing to listen & be influenced, & offers significant support.• Leadership Team members are optimistic about the team's future & ability
to accomplish it's mission.
Noteworthy Limitations• Team goals & responsibilities are not as clear as individual assignments.• Team capabilities and resources are a concern.
‘How to summarize open-ended data’ 2
45 Using Surveys to Support OD Projects Mick Verran
www.Wordle.net 46 Using Surveys to Support OD Projects Mick Verran
‘How to summarize open-ended data’ 3
Event reviewsGoalsAssess achievement of objectivesIsolate impact of content – process – presenterAssess commitment to ActionsIdentify need for further support & ‘next agenda’ItemsPre-work relevance & use Content RelevancePresenters: Relevance of examples – Responses to questions –Interactions Clarity of action planOutstanding issuesPersonal commitment to changeOngoing support requirements
47 Using Surveys to Support OD Projects Mick Verran
Handout www.MickVerran.com Password BFR
978-505-5275
Thank you.
You are free to copy and distribute the work under the following conditions: you must attribute the work to “Mick Verran [email protected]” ; you may not use the work for commercial purposes; if you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.