How’s it Working? Evaluating Your Program

How’s it Working? Evaluating Your Program

MAAPS Conference, 7 May 2010Debra Smith & Judah Leblang

Program Evaluation & Research GroupSchool of Education, Lesley University

PERG Founded 1976

Over 600 program evaluation and research studies in various educational settings

Also offers professional development and consultation

Session participants will:

Be introduced to the basics of program evaluation through an example

Define a question or questions about their own program

Identify methods for collecting data that would help to answer their question/s

Discuss next steps

What is program evaluation?

A type of applied research focused on systematically collecting and analyzing data to help answer questions about a program, or some aspect of a program, in order to make decisions about it.

Purposes

Accountability

Program development

Generating knowledge

Formative vs Summative

Formative evaluation offers feedback along the way to improve programs

Summative evaluations “sum up” the results of a program at the end of a period of development or implementation.

Audiences

Funders Program leaders Program participants Organizational partners Others

Evaluation process

1. Goals/ logic model

2. Questions

3. Evaluation plan4. Data

collection

5. Data analysis

6. Reporting

PROGRAM

An example: Evolutions

• After school program begun in 2005, connected with Peabody Museum of Natural History at Yale University—initially involved approximately 40 low SES/ minority students

Evolutions program goalsTo provide opportunities for students to:• Prepare for post-secondary (college)

education; • Learn about scientific—and other

careers; • Expand their knowledge of and interest

in science (science literacy); • Develop transferable skills for the

future; and • learn about the Peabody

Museum/museum careers.

Logic models

Map a coherent chain of connections between goals, resources, activities and what you expect (short term), want (over an intermediate period) and hope (in the long term) to happen.

They also reflect your assumptions and theory of action or change.

Logic Model

Key Concepts

Category Resources or Inputs

Activities—

Outputs Short-term outcomes

Long-term outcomes

General information

Staff, funds, materials, space, etc

What we plan to

do/who we will do it

for

The results of

our program—direct outputs

Outcomes (changes) at completion

of the project year or soon after

Outcomes (changes)

several years

beyond completion

of the project

And EVO exampleCategory Resources

or InputsActivities— Outputs Short-term

outcomesLong-term outcomes

General information

Staff, funds, materials, space, etc

What we plan to do/who we will do it for

The results of our program

—direct outputs

Outcomes (changes) at

completion of the project

year or soon after

Outcomes (changes)

several years beyond

completion of the project

EVO examples

Full time project

director funds from

Peabody Museum and other funders, classroom space, etc.

In-depth exploration of:

science topics

tours of Peabody

collections Yale scientist

labs

Students will meet:

at least 6 scientists students

will visit no less than 2

natural history museums

Students will: learn skills

associated with producing

a museum exhibition

Understand key science

themes

Students will: understand

different types of careers

within disciplines

understand the college application

process

be inspired to pursue a

career in the sciences

Goal

Rationale

Assumptions

Resources Activities Outputs

Short-term outcomes

Mid-term outcomes

Long-term outcomes

Logic models may look different..

Develop a logic model for your own program/ project

Evaluation process

1. Goals/ logic model

2. Questions

3. Evaluation plan4. Data

collection

5. Data analysis

6. Reporting

PROGRAM

Questions: Think Goldilocks

Specific but not too detailed

Important but not too broad in scope

Key Questions: Part One

How does EVO prepare students for college or high school?

How are EVO students involved in developing an exhibit at the museum?

Do students develop increased “science literacy,” as defined by EVO staff?

Key Questions: Part Two

How (if at all) do students express more confidence about and interest in doing science?

Are students more aware of careers in science?

How (if at all) do students demonstrate increased knowledge of the college application process, and develop criteria for choosing a college that meets their needs?

What questions do you want to answer about your program?

Evaluation process

1. Goals/ logic model

2. Questions

3. Evaluation plan4. Data

collection

5. Data analysis

6. Reporting

PROGRAM

Data collection methods

Observation

Interviews/ focus groups

Surveys

Document/artifact review

PERG Evaluation MatrixEvolutions 2005-06 Data collection activities>>

EVALUATION QUESTIONS:

Observe Evo students

Student focus groups

Interview project director

Review project docs and artifacts

Examine pre-post survey

Student prep for college/academic planning

√ √ √ √ √

Student involvement in museum exhibit

√ √ √

Students' development of science literacy

√ √ √ √

Student learning √ √ √ √ √

Students' interest in science/environment

√ √ √ √ √

Students' confidence in doing science

√ √ √

Students' interest in/knowledge of science careers

√ √ √ √

Technical considerations: Validity

Will the data answer the questions?

Are we asking the right questions?

Triangulation

Is there adequate triangulation (use of multiple methods and/or data sources) to ensure validity?

Drafting your own matrix: What data will help you answer your questions?

Evaluation process

1. Goals/ logic model

2. Questions

3. Evaluation plan4. Data

collection

5. Data analysis

6. Reporting

PROGRAM

Collecting data

Make sure your plan is doable given time and resources available.

Design instruments to focus your data collection, ensure consistency and avoid bias.

Be organized: take notes, develop a system for tracking/ filing your data.

Collecting data

Communicate clearly about what you are doing, why and how the findings will be shared and used.

Be mindful of human subjects protections. Does your organization have an institutional review board (IRB)?

The First Year: site visit

On-site data collection• Focus groups with students• Interviews with director, project staff• Observation of end of year event• Parent interviews

Evaluation process

1. Goals/ logic model

2. Questions

3. Evaluation plan4. Data

collection

5. Data analysis

6. Reporting

PROGRAM

Analyzing data

What stands out?

What are the patterns?

What are the similarities?

What are the differences?

Is more information needed?

Reliability

Are the patterns in the data, or judgments about the data, consistent?

Validity, again

Is the data helping you answer the questions?

Is the data credible?

Evaluation process

1. Goals/ logic model

2. Questions

3. Evaluation plan4. Data

collection

5. Data analysis

6. Reporting

PROGRAM

Reporting

Consider purpose and audience/s

Reporting relevant findings, questions/ recommendations

Engaging stakeholders in discussion

Using findings to inform next steps

Results of the first-year evaluation

• The impact of the evaluation on EVO—more focused program, clearer objectives, suggestions for sustainability.

• Evidence of program success: Retention, student engagement, positive changes in students’ view of doing science and scientists.

The Ongoing Evaluation--shaping the program:

• Implementation of evaluator suggestions—examples: informational interviewing, developing a smaller exhibit, refining requirements for students

EVO: 2006-Today

• Continued development and expansion of EVO—2006 until today: Expansion of the program from approximately 40 to more than 80 students, introduction of internships and Sci Corps.

– Different areas of science focus—environmental awareness, geoscience, depending on funding sources.

Evaluation resources

W.K. Kellogg Foundation Evaluation Handbook www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf

Kellogg Logic Model Development Guide www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf

Basic Guide to Program Evaluationwww.managementhelp.org/evaluatn/fnl_eval.htm

Evaluation resources

Program Evaluation & Research GroupLesley University29 Everett St.Cambridge, MA 02138www.lesley.edu/perg.htm617-349-8172perg@lesley.edu