Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
Dave Ward Puget Sound Partnership
RESEARCH 101 The Basics
Agenda
• Purposes of research
• Research methods
• Cost
• Process
• Lessons learned
• Reading and interpreting research
Purposes of Research
PERCEPTION RESEARCH to understand public perception of issues
FORMATIVE RESEARCH to inform program development
EVALUATIVE RESEARCH to measure program effectiveness
BENCHMARK RESEARCH to measure long term change
PERCEPTION RESEARCH to understand public perception of issues
• What do people think?
• What do people know?
• What do people think they know?
• What are people doing?
• What do people think?
– If a biosolids soil mix was available in bags, how likely would you be to buy and use it?
– How important is it to provide financial incentives to property owners for shoreline protection?
PERCEPTION RESEARCH to understand public perception of issues
23%
61%
• What do people know?
– For each of the following, please tell me whether you think King County provides that service (water quality, stormwater, groundwater)?
– Is your home served by a septic system?
PERCEPTION RESEARCH to understand public perception of issues
• What do people think they know?
– On a scale of 1 to 5, how at risk do you think salmon populations in our region are?
– Of the following list of problems facing the environment, which would you say are the most serious?
PERCEPTION RESEARCH to understand public perception of issues
1 in 7 people
47% answered correctly on average 30% answered incorrectly on average 22% don’t know on average
• What are people doing?
– Do you fertilize your lawn?
– How many times has your septic tank been pumped in the past ten years
– How frequently do you pick up your dog’s waste in your yard?
PERCEPTION RESEARCH to understand public perception of issues
FORMATIVE RESEARCH to inform program development
• What is the scope and scale of the problem?
• What is the appropriate BMP?
• How can we motivate people to implement the BMP?
• What are the barriers and motivators to implement the BMP?
• What is the best message?
• What is the best communication method?
FORMATIVE RESEARCH to inform program development
1. Review existing programs
2. Review academic research
3. Inventory relevant businesses
4. Inventory relevant products
5. Quantify the scope and scale of the problem
6. Conduct focus groups
7. Conduct a telephone survey
8. Analyze relevant demographic data
9. Identify problem areas using GIS and water quality data
EVALUATIVE RESEARCH to measure program effectiveness
• Did the program do what we expected?
• Did people implement BMPs as a result of the program?
• Did people change their behavior/practices as a result of the program?
EVALUATIVE RESEARCH to measure program effectiveness
• X% of people started picking up pet waste
• Y% of homeowners replaced their lawns with trees
• Y% of people stopped washing their cars on pavement
• Z% of homeowners installed a rain garden
BENCHMARK RESEARCH to measure long term change
• What are the long term trends?
• Did people implement BMPs?
• Did people change their behavior/practices?
• Are we on an upward or downward trend?
• Did water get cleaner?
SOUND BEHAVIOR INDEX
KING
BENCHMARK RESEARCH to measure long term change
• X% decrease in pesticide use since 1990
• Y% increase in aluminum recycling over the past ten years
• Z% decrease in coliform bacteria level since 2001
RESEARCH METHODS
Research Techniques (Conventional Methods)
• Telephone surveys
• Focus groups
• Interactive polling sessions
• Informant interviews
• Intercept surveys
• Mail surveys / questionnaires
• Internet surveys
Research Techniques (Formative Methods)
All Conventional Methods plus: • Staff / audience communication • Consumer research data • Demographic research data • Key informant interviews / ethnographic assessments • Inventory/analysis of existing programs • Review existing research • Message testing (neighbors, friends, focus groups,
surveys) • Evaluate BMPs – life cycle cost, targeting,
effectiveness
Research Techniques (Program Measures)
All Conventional Methods plus: • Staff / audience communication • Audience observation • Windshield surveys • Program/event attendance • Program/event follow-up contact • Reply cards • Promotional item distribution (quantity/location) • Responses to invitations/solicitations • Calls for technical assistance • Return on investment
Telephone Surveys
Quantitative information
Statistically valid
Results can be extrapolated
Large numbers of respondents
Tightly scripted
Questionnaire
Fixed questions
Costly ($00,000)
Asks “How many?”
Objective (in method only)
Focus Groups
Qualitative information
Not statistically valid
Results cannot be extrapolated
Low numbers of respondents
Loosely scripted
Discussion guide
Can change on-the-fly
Not as costly ($0,000)
Asks “Why?”
Subjective
Telephone Surveys and Focus Groups
Cou
rt D
ecis
ion
Source: The Gallup Poll
Surveys and Focus Groups are a Snapshot in Time
1. Determine goals You
2. Select a consultant You
3. Identify audience You and consultant
4. Develop screener You and consultant
5. Develop discussion guide You and consultant
6. Select participants Consultant
7. Arrange for and set-up room Consultant
8. Order and pay for food Consultant
9. Facilitate group sessions Consultant
10. Observe sessions You
11. Record sessions – video/audio Consultant
12. Pay participants Consultant
13. Produce report Consultant
14. Interpret results You and consultant
Focus Group Process
• Screener • Discussion guide • Transcript and/or digital video • Sound recording • Report/analysis • Participant list
Focus Group Work Product
• About $7,000 - $9,000 per focus group
Cost
• Recruiting from small audiences
• Recruiting from unusual audiences
• Recruiting from a list (sometimes)
• Looking up phone numbers
What adds cost?
1. Determine goals You
2. Select a consultant You
3. Identify audience You and consultant
4. Develop questionnaire You and consultant
5. Pretest questionnaire You and consultant
6. Field questionnaire (make calls) Consultant
7. Produce topline results Consultant
8. Produce cross-tabulated data Consultant
9. Conduct statistical analysis Consultant
10. Produce report Consultant
11. Interpret results You and consultant
Telephone Survey Process
• Questionnaire • Topline results • Cross-tabulated results • Report/analysis
Telephone Survey Work Product
• 15-20 minute survey of 400: About $20,000 - 40,000
Cost
• Polling small audiences
• Polling unusual audiences
• Polling from a list (sometimes)
• Looking up phone numbers
What adds cost?
1. Meet the focus group facilitator before you select the contractor
2. First decide who you want to hear from; then let the consultant develop the screener
3. First decide what you want to know; then let the consultant develop the questionnaire/discussion guide
4. Do everything you can to ensure the contractor understands what you want to know
5. Stay close to the contractor
Lessons Learned
Lessons Learned 6. Wordsmith
7. Pay close attention to sequence, flow, and how information is revealed
8. Meet with the facilitator immediately before every focus group
9. Schedule focus groups in pairs using the same discussion guide
10. Don’t allow the consultant to start calling without your approval of the questionnaire / screener
11. Don’t allow the focus group to begin without your approval of the discussion guide
READING & INTERPRETING
RESEARCH
Three things to question about ALL research
• Method Understand the methodology section
• Data Is the data sound?
• Interpretation Do the data support the conclusions?
Three things to remember about ALL research
• Nothing is isolated Everything is connected
• Every study reflects a moment in time
• One data point is not a trend
Do you approve or disapprove of the job John Boehner is doing?
Approve 33% Disapprove 37% Not sure 30%
Public Policy Polling, July 15-17 2011, N=928
If God exists, do you approve or disapprove of its performance?
Approve 52% Disapprove 9% Not sure 40%
Public Policy Polling, July 15-17 2011, N=928
Pay Attention to the Margin of Error!
Approximate Sample Size Margin of Error 200 6.9% 300 5.7% 400 4.9% 800 4.0% 1,300 2.7% Split samples increase the margin of error!
Would you support rebuilding salmon runs if it meant an increase in your taxes?
Would you support a tax increase to rebuild salmon runs for future generations?
Do you support rebuilding salmon runs?
Many projects that benefit salmon also reduce erosion, water pollution, and flooding of downstream properties. Knowing that, would you support rebuilding salmon runs if it meant an increase in your local taxes?
Question Phrasing
3. Do you support or oppose laws that prohibit killing unborn babies?
2. Do you support or oppose laws that prohibit abortions?
1. Do you support or oppose laws that protect abortion rights?
Question Bias
Triangulation
Question 10
Question 24 Question 3
The real answer
HAVE FUN
It’s an exploration!