Embed Size (px)
Citation preview
Data Analysis for Evaluation
Eric Graig, Ph.D.
Slide 2Innovation Network, Inc.
Purpose of this Training
To increase your skills in analysis and interpretation of the information you collect
To increase your ability to report your findings to a variety of audiences
To learn how to make evaluation results actionable
Slide 3Innovation Network, Inc.
Three Steps to Data Analysis
Analyze Results Communicate Findings Use Findings for Program Improvement
Slide 4Innovation Network, Inc.
Keep it Simple
Aim for a systematic effort Keep your audience in mind Pay attention to the usability of your
evaluation report
Slide 5Innovation Network, Inc.
Two Kinds of Data
Quantitative Qualitative
Slide 6Innovation Network, Inc.
Quantitative Data
Data that is numerical, counted, or compared on a scale
Demographic data Answers to closed-ended survey items Attendance data Scores on standardized instruments
Slide 7Innovation Network, Inc.
Qualitative Data
Narratives, logs, experience Focus groups Interviews Open-ended survey items Diaries and journals Notes from observations
Slide 9Innovation Network, Inc.
Summary Statistics
Measures of Central Tendency Mean Median Range
Frequency distributions Percentage distributions
Slide 10Innovation Network, Inc.
Frequency Distributions
To what extent did you increase your skills in putting together a household budget?
A lot Some A little Not at all
Women (N=30) 14 9 5 2
Slide 11Innovation Network, Inc.
Percentage Distributions
To what extent did you increase your skills in putting together a household budget?
A lot Some A little Not at all
Women (N=30) 46% 30% 17% 7%
Slide 12Innovation Network, Inc.
Graphing Frequency Data
How did you first hear about the web-site?
Court ReferralSocial WorkerFriend or Acquaintance
Web Search EngineLibrarian
Newspaper StoryOther
Slide 13Innovation Network, Inc.
Means and Medians
History 95
English 96
Biology 93
Latin 92
Math 98
Music 94
Gym 40
Mean = 87
Median = 94
Math 98
English 96
History 95
Music 94
Biology 93
Latin 92
Gym 40
Slide 14Innovation Network, Inc.
Some Subtleties
40 50 55 94 100 100 100
40 92 93 94 95 96 98
Mean = 81
Mean = 87
Slide 15Innovation Network, Inc.
Histograms
0
1
2
3
4
5
6
60 70 80 90 100F
req
uen
cy0
1
2
3
4
5
6
7
60 70 80 90 100
Fre
qu
ency
Slide 16Innovation Network, Inc.
Cross Tabulations
Program Type Area of Inquiry OutcomeWeb site Employment law SatisfiedI & R Line Family law Not satisfiedLaw clinic Immigration PendingWeb site Immigration SatisfiedI & R Line Immigration SatisfiedI & R Line Family law Not satisfiedWeb site Employment law Not satisfiedLaw clinic Other SatisfiedI & R Line Other Not satisfiedI & R Line Other SatisfiedLaw clinic Employment law SatisfiedWeb site Family law SatisfiedLaw clinic Family law SatisfiedWeb site Immigration Not satisfiedLaw clinic Immigration Not satisfiedI & R Line Family law SatisfiedI & R Line Immigration Not satisfiedI & R Line Employment law Not satisfiedLaw clinic Other Pending
Count of Outcome OutcomeProgram Type Not satisfied Pending Satisfied Grand TotalI & R Line 7 5 12Law clinic 1 3 7 11Web site 6 5 11Grand Total 14 3 17 34
Count of Outcome OutcomeProgram Type Not satisfied Pending Satisfied Grand TotalI & R Line 58% 0% 42% 100%Law clinic 9% 27% 64% 100%Web site 55% 0% 45% 100%Grand Total 41.18% 8.82% 50.00% 100.00%
Slide 17Innovation Network, Inc.
Graphing comparisons
Satisfaction with Services
0
5
10
15
20
25
30
35
40
A B C D E
Clinic Name
Sat
isfa
ctio
n S
core
Slide 18Innovation Network, Inc.
Some subtleties
Satisfaction with Services
0
2
4
6
8
10
12
14
16
A B C D E
Clinic
Sat
isfa
ctio
n S
core
Staff
Advice
Facility
Slide 19Innovation Network, Inc.
Some more subtleties
Satisfaction with Services
0
2
4
6
8
10
12
14
16
Staff Advice Facility
Satisfaction Component
Sat
isfa
ctio
n S
core
A
B
C
D
E
Slide 20Innovation Network, Inc.
A Note on Quantitative Data Entry
Several approaches Paper and pencil tally Word processing table Spreadsheet Custom database
Slide 21Innovation Network, Inc.
Qualitative Data
Textual data Interview transcripts Case notes/ clinical notes Open-ended survey questions
Photographs Video recordings
Slide 22Innovation Network, Inc.
Qualitative Analysis
Identify themes or patterns Pre-set themes Determine as you go
Tag text segments with themes
Slide 23Innovation Network, Inc.
Sample data sheet
Interview Write Up Sheet Naming Standard: Siteinterviewtype# Cluster ND Futures Site Oakville Interviewee Diane J anssen, Elementary School Principal Date of Interview December 2004 Tape # 02-1234-PCM Length 31 Interview Types: Focus Com Coach Sup Plan Other
Site
Name
Intervie
w Type/ #
Questio
n ?#
Text
Ellendal
e
Plan PC.1 She states that she came from to the town to attend
the local Bible College, stayed to become a teacher in
the local school and then took a job teaching at the
college before becoming the principal of the
elementary school. She states that she first got
involved with the program after the large community
meeting that took place in April of 2004. The school
leadership (she doesn't indicate precisely who) felt it
was important that someone from the school get
involved in the planning committee so she volunteered
since she was interested in getting involved anyway.
She mentions both a desire to bring to the committee
the perspective of the school while at the same time to
take back to the school whatever knowledge she
acquired through participation.
PC.2 She talks a little bit about the school. It is the K-12
school with about 350 children. She oversees the K-6
program while another principal supervises grades 7
through 12. There are 157 students in the elementary
school with a bit of a budget at the lower grades
indicating some new families moving to town. Overall
Site
Interviewee
Date
Tape number
Length
Question number
Slide 24Innovation Network, Inc.
Exercise
Break into small groups Using the sheet provided search for snippets
of text that relate to the codes provided Identify any other themes that may be
important that are not provided on the sheet
Slide 25Innovation Network, Inc.
Communicating Findings
“Information that is not effectively shared with others will not be effectively used.”
Source: Building a Successful EvaluationCenter for Substance Abuse Prevention
Slide 26Innovation Network, Inc.
Audience
Staff Board Funders
Partners Other agencies Public
Slide 27Innovation Network, Inc.
Reporting Formats
Written report Data book Flyer Formal oral presentation Round table with partners/ stakeholders Poster Web site
Slide 28Innovation Network, Inc.
Whatever strategy you choose:
Link findings to the program’s outputs and desired outcomes
Include successes and challenges Support your claims with data Acknowledge knowledge gaps
Slide 29Innovation Network, Inc.
Organizing Findings
Overview Describe program/ describe evaluation
Program Implementation What did you do, and how well did you do it?
Program Outcomes: What were the changes that occurred?
Lessons learned Sub-group differences
Slide 30Innovation Network, Inc.
Overview Section
Your organization Your program Purpose of the evaluation Evaluation design Structure of the report
Slide 31Innovation Network, Inc.
Program Implementation
Characteristics of program participants Demographics
Outputs Number of sessions held Number of participants served Partnerships formed Satisfaction with program
Slide 32Innovation Network, Inc.
Program Outcomes
What has changed? What do participants now know? What are they now able to do? What do they actually do? How are there lives different?
Slide 33Innovation Network, Inc.
Lessons Learned
Which of the program’s assumptions were validated, which need to be reconsidered?
Does the program affect different kinds of people differently?
Are results better with some groups? How will findings be addressed? What new hypotheses did the evaluation generate? How did the evaluation work?
Slide 34Innovation Network, Inc.
Additional Resources
One-on-one technical assistance Synchronous course Asynchronous material InnoNet website www.innonet.org
