Upload
molly-hawkins
View
216
Download
2
Tags:
Embed Size (px)
Citation preview
How to Focus and Measure Outcomes
Katherine Webb-MartinezCentral Coast and South Region
All Staff Conference April 23, 2008
Workshop Desired Outcomes
An understanding of how basic logic model techniques help define outcomes and focus program evaluation
An understanding of options for data collection methods to measure outcomes
Workshop Agenda
Introductions
Part 1: Logic Model
• Presentation – Logic Model helps with outcomes
• Activity – “Using a logic model to focus outcomes evaluation”
Part 2: Measuring Outcomes
• Presentation - Review of Data Collection Options
• Activity – Methods Café
Close
Part 1: Logic Model as Road Map
Where are you going?
How will you get there?
What will show that you’ve arrived?
“If you don’t know where you are going, how are you gonna’ know when you get there?”
Yogi Berra
University of Wisconsin-Extension, Program Development and Evaluation
HUNGRY
Feel betterFeel betterGet foodGet food Eat food Eat food
Everyday example
University of Wisconsin-Extension, Program Development and Evaluation
Logical chain of connections showing what the program is to accomplish
What we do
Who we reach
INPUTS OUTPUTS OUTCOMES
Program investments
Activities Participation Learning (Short-term)
Action (Medium
-term)
Impacts (Long-term)
What we invest
What results
University of Wisconsin-Extension, Program Development and Evaluation
Outputs vs. outcomes
Example: Number of patients discharged from state mental hospital is an output. Percentage of discharged who are capable of living independently is an outcome
Not how many worms the bird feeds its young,
but how well the fledgling flies (United Way
of America, 1999)
University of Wisconsin-Extension, Program Development and Evaluation
Social-economic-environmental impacts
What do you want to know?
Source: Bennett and Rockwell, 1995, Targeting Outcomes of Programs
Reactions
Learning
Actions
Number and characteristics of people reached; frequency and intensity of contact
Degree of satisfaction with program; level of interest; feelings toward activities, educational methods
Changes in knowledge, attitudes, skills, aspirations
Changes in behaviors and practices
Participation
University of Wisconsin-Extension, Program Development and Evaluation
Needs assessment:
What are the characteristics, needs, priorities of target population?
What are potential barriers/facilitators?
What is most appropriate to do?
Process evaluation:
How is program implemented?
Are activities delivered as intended?
Are participants being reached as intended?
What are participant reactions?
Outcome evaluation:
To what extent are desired changes occurring?
Who is benefiting / not benefiting? How?
What are unintended outcomes?
Impact evaluation:
What are the net effects?
What are final consequences?
To what extent can changes be attributed to the program?
University of Wisconsin-Extension, Program Development and Evaluation
ACTIVITY: PARENT EDUCATION PROGRAM
What do you (and others) want to know about the program?
Staff
Money
Partners
Assess parent ed programs
Design- deliver evidence-based program of 8 sessions
Parents increase knowledge of child dev
Parents better understanding their own parenting style
Parents use effective parenting practices
Improved child-parent relations
Research
INPUTS OUTPUTS OUTCOMES
Facilitate support groups
Parents gain skills in new ways to parent
Parents identify appropriate actions to takeParents
of 3-10 year olds
attend
Reduced stress
Parents gain confidence in their abilities
Strong families
Logic Model Reflection
How does a logic model help focus on what outcomes to measure and when to measure them?
How might you use a logic model in your own work?
Part 2: Measuring Outcomes
How will you answer your questions?
1. Decide on what evidence answers the questions
2. Determine sources of information
3. Choose data collection methods
University of Wisconsin-Extension, Program Development and Evaluation
Identify evidence
How will you know it when you see it?
What are the specific indicators that will be measured?
- Often expressed as # or %
- Outputs and outcomes indicators
- Quantitative or qualitative
University of Wisconsin-Extension, Program Development and Evaluation
Logic model with Indicators for Outputs and Outcomes
Program implemented
Targeted farmers
Farmers learn
Farmers practice new techniques
Farm profitability increases
Number of workshops held
Quality of workshops
Number and percent of farmers attending
Number and percent who increase knowledge
Number and percent who practice new techniques
Number and percent reporting increased profits; amount of increase
Outputs Outcomes
Remember:
“Not everything that counts can be counted.”
9 (Quantity) Happy (Quality) Kids
Quantitative: numbers, breadth, generalizability Qualitative: words, depth, specific
University of Wisconsin-Extension, Program Development and Evaluation
Possible Evaluation questions, indicators
Staff
Money
Partners
Parents increase knowledge of child dev
Parents better understand their own parenting style
Parents use effective parenting practices
Improved child-parent relations
Research Facilitate support groups
Parents gain skills in new ways to parent
Parents identify appropriate actions to take
To what extent is stress reduced?To what extent are relations improved?
To what extent did behaviorschange? For whom? Why? What else happened?
To what extent did knowledge and skills increase? For whom? Why? What else happened?
Who/how many attended/did not attend? Did they attend all sessions?Supports groups? Were they satisfied – why/why not?
How many sessions were held? How effectively?#, quality of support groups?
What amount of $ and time were invested?
Parents of 3-10 year olds
Deliver series of 8 interactive sessions
EVALUATION QUESTIONS
# Staff$ used# partners
# Sessions held
Quality criteria
INDICATORS
#,% attended per session
Certificate of completion
#,% demonstrating increased knowledge/skill
Additional outcomes
#,% demonstrating changes
Types of changes
#,% demonstrating improvements
Types of improvements
Develop parent ed curriculum
Reduced stress
Parents gain confidence in their abilities
Strong families
Determine Sources of Information
Program participants
Existing data• Program records, attendance logs, etc• Pictures, charts, maps, pictorial records
Others/Non-participants• Key informants • Funders • Collaborators• Etc.
University of Wisconsin-Extension, Program Development and Evaluation
Decide on Data Collection Methods
Survey Interview Test Observation Group techniques Portfolio review
Diaries, journals Case study Photography,
video Document review Expert or peer
review
University of Wisconsin-Extension, Program Development and Evaluation
Data collection plan template
Questions Indicators Data collection
Sources Methods Sample Timing
When choosing your methods consider:
Purpose
Participants
Resources Available
Some things to remember…
There is no one right method of collecting data
Each has a purpose, advantages and challenges
The goal is to obtain trustworthy, authentic and credible evidence
Often a mix of methods is preferred
Are the data reliable and valid?
Validity: Are you measuring what you think you’re measuring?
Reliability: If something was measured again using the same instrument, would it produce the same (or near the same) results?
University of Wisconsin-Extension, Program Development and Evaluation
Logic model and reporting
Methods Cafe
What approaches to measure outcomes work well for CE programs and why?
Is there anything you’ve tried that you would not recommend?
Methods Reflection
Given what we have discussed, what might you change or do differently with the methods that work for you?
Are there any other, new methods that might work for you and your clientele?