Hows it Working? Evaluating Your Program MAAPS Conference, 7
May 2010 Debra Smith & Judah Leblang Program Evaluation &
Research Group School of Education, Lesley University
Slide 2
PERG Founded 1976 Over 600 program evaluation and research
studies in various educational settings Also offers professional
development and consultation
Slide 3
Session participants will: Be introduced to the basics of
program evaluation through an example Define a question or
questions about their own program Identify methods for collecting
data that would help to answer their question/s Discuss next
steps
Slide 4
What is program evaluation? A type of applied research focused
on systematically collecting and analyzing data to help answer
questions about a program, or some aspect of a program, in order to
make decisions about it.
Slide 5
Purposes Accountability Program development Generating
knowledge
Slide 6
Formative vs Summative Formative evaluation offers feedback
along the way to improve programs Summative evaluations sum up the
results of a program at the end of a period of development or
implementation.
Slide 7
Audiences Funders Program leaders Program participants
Organizational partners Others
Slide 8
Evaluation process 1. Goals/ logic model 2. Questions 3.
Evaluation plan 4. Data collection 5. Data analysis 6. Reporting
PROGRAM
Slide 9
An example: Evolutions After school program begun in 2005,
connected with Peabody Museum of Natural History at Yale
Universityinitially involved approximately 40 low SES/ minority
students
Slide 10
Slide 11
Evolutions program goals To provide opportunities for students
to: Prepare for post-secondary (college) education; Learn about
scientificand other careers; Expand their knowledge of and interest
in science (science literacy); Develop transferable skills for the
future; and learn about the Peabody Museum/museum careers.
Slide 12
Slide 13
Slide 14
Slide 15
Slide 16
Slide 17
Logic models Map a coherent chain of connections between goals,
resources, activities and what you expect (short term), want (over
an intermediate period) and hope (in the long term) to happen. They
also reflect your assumptions and theory of action or change.
Slide 18
Logic Model Key Concepts CategoryResources or Inputs Activities
OutputsShort-term outcomes Long-term outcomes General information
Staff, funds, materials, space, etc What we plan to do/who we will
do it for The results of our program direct outputs Outcomes
(changes) at completion of the project year or soon after Outcomes
(changes) several years beyond completion of the project
Slide 19
And EVO example CategoryResources or Inputs
ActivitiesOutputsShort-term outcomes Long-term outcomes General
information Staff, funds, materials, space, etc What we plan to
do/who we will do it for The results of our program direct outputs
Outcomes (changes) at completion of the project year or soon after
Outcomes (changes) several years beyond completion of the project
EVO examples Full time project director funds from Peabody Museum
and other funders, classroom space, etc. In-depth exploration of:
science topics tours of Peabody collections Yale scientist labs
Students will meet: at least 6 scientists students will visit no
less than 2 natural history museums Students will: learn skills
associated with producing a museum exhibition Understand key
science themes Students will: understand different types of careers
within disciplines understand the college application process be
inspired to pursue a career in the sciences
Develop a logic model for your own program/ project
Slide 22
Evaluation process 1. Goals/ logic model 2. Questions 3.
Evaluation plan 4. Data collection 5. Data analysis 6. Reporting
PROGRAM
Slide 23
Questions: Think Goldilocks Specific but not too detailed
Important but not too broad in scope
Slide 24
Key Questions: Part One How does EVO prepare students for
college or high school? How are EVO students involved in developing
an exhibit at the museum? Do students develop increased science
literacy, as defined by EVO staff?
Slide 25
Key Questions: Part Two How (if at all) do students express
more confidence about and interest in doing science? Are students
more aware of careers in science? How (if at all) do students
demonstrate increased knowledge of the college application process,
and develop criteria for choosing a college that meets their
needs?
Slide 26
What questions do you want to answer about your program?
Slide 27
Evaluation process 1. Goals/ logic model 2. Questions 3.
Evaluation plan 4. Data collection 5. Data analysis 6. Reporting
PROGRAM
Slide 28
Data collection methods Observation Interviews/ focus groups
Surveys Document/artifact review
Slide 29
PERG Evaluation Matrix Evolutions 2005-06 Data collection
activities>> EVALUATION QUESTIONS: Observe Evo students
Student focus groups Interview project director Review project docs
and artifacts Examine pre-post survey Student prep for
college/academic planning Student involvement in museum exhibit
Students' development of science literacy Student learning
Students' interest in science/environment Students' confidence in
doing science Students' interest in/knowledge of science
careers
Slide 30
Technical considerations: Validity Will the data answer the
questions? Are we asking the right questions?
Slide 31
Triangulation Is there adequate triangulation (use of multiple
methods and/or data sources) to ensure validity?
Slide 32
Drafting your own matrix: What data will help you answer your
questions?
Slide 33
Evaluation process 1. Goals/ logic model 2. Questions 3.
Evaluation plan 4. Data collection 5. Data analysis 6. Reporting
PROGRAM
Slide 34
Collecting data Make sure your plan is doable given time and
resources available. Design instruments to focus your data
collection, ensure consistency and avoid bias. Be organized: take
notes, develop a system for tracking/ filing your data.
Slide 35
Collecting data Communicate clearly about what you are doing,
why and how the findings will be shared and used. Be mindful of
human subjects protections. Does your organization have an
institutional review board (IRB)?
Slide 36
The First Year: site visit On-site data collection Focus groups
with students Interviews with director, project staff Observation
of end of year event Parent interviews
Slide 37
Evaluation process 1. Goals/ logic model 2. Questions 3.
Evaluation plan 4. Data collection 5. Data analysis 6. Reporting
PROGRAM
Slide 38
Analyzing data What stands out? What are the patterns? What are
the similarities? What are the differences? Is more information
needed?
Slide 39
Reliability Are the patterns in the data, or judgments about
the data, consistent?
Slide 40
Validity, again Is the data helping you answer the questions?
Is the data credible?
Slide 41
Evaluation process 1. Goals/ logic model 2. Questions 3.
Evaluation plan 4. Data collection 5. Data analysis 6. Reporting
PROGRAM
Slide 42
Reporting Consider purpose and audience/s Reporting relevant
findings, questions/ recommendations Engaging stakeholders in
discussion Using findings to inform next steps
Slide 43
Results of the first-year evaluation The impact of the
evaluation on EVO more focused program, clearer objectives,
suggestions for sustainability. Evidence of program success:
Retention, student engagement, positive changes in students view of
doing science and scientists.
Slide 44
The Ongoing Evaluation-- shaping the program: Implementation of
evaluator suggestionsexamples: informational interviewing,
developing a smaller exhibit, refining requirements for
students
Slide 45
EVO: 2006-Today Continued development and expansion of EVO2006
until today: Expansion of the program from approximately 40 to more
than 80 students, introduction of internships and Sci Corps.
Different areas of science focus environmental awareness,
geoscience, depending on funding sources.
Slide 46
Evaluation resources W.K. Kellogg Foundation Evaluation
Handbook www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf
www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf Kellogg Logic Model
Development Guide www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf
www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf Basic Guide to
Program Evaluation
www.managementhelp.org/evaluatn/fnl_eval.htm
Slide 47
Evaluation resources Program Evaluation & Research Group
Lesley University 29 Everett St. Cambridge, MA 02138
www.lesley.edu/perg.htm 617-349-8172 [email protected]