1

Putting Evaluation to Work For You

Chesapeake Bay Funders Network

Program Evaluation

Training Workshop

OMG Center for Collaborative Learning

January 9-10, 2008

2

Building an Evaluation Plan

I. Theory of Change/Pathway Map

II. Outcomes

III. Indicators

IV. Methods

V. Putting Evaluation to Work for You

3

Putting Evaluation to Work for You

I. Data Collection Management and Work Plan

II. Working With Evaluation Stakeholders

III. Integrating Evaluation into Your Work

IV. Reporting and Sharing Results

4

1. Designate roles and responsibilities for all elements of evaluation• Management, design, data collection, relationships, data

entry, analysis, reporting and use

2. Establish a timeline for collecting the data (pre-post measures, implementation, short term and long term results)• Build in extra time for delays (implementation and data

collection)

3. Establish a calendar for reporting, sharing results

Let’s Construct a…Data Collection Management and Workplan

5

4. Design, find and/or adapt appropriate data collection instruments and tools to track indicators of your specified outcomes. Train staff and volunteers responsible for data collection

5. Build necessary relationships to obtain data from other sources

6. Develop protocol for data management and storage • Electronic and hard copy, timelines for entry

Let’s Construct a…Data Collection Management and Workplan

6

7. Develop a data analysis plan for qualitative and quantitative data – driven by intended use

8. Develop an evaluation budget – what are you going to spend on this

Let’s Construct a…Data Collection Management and Workplan

7

To Project the Cost Involved, Consider: Labor costs, consultants Equipment costs – water testing; cameras; computer hardware

and software Direct costs (mailing, copying, telephone use) Incentives for participation Costs for conducting focus groups or forums (food, space,

babysitting, transportation) Travel costs

Presentation materials

Let’s Construct a…Data Collection Management and Workplan

8

Working With Evaluation Stakeholders

Who are the individuals or groups that have a stake in this evaluation?

How might they use or be affected by the evaluation’s results?

9

Working With Evaluation Stakeholders

How do we involve stakeholders in the evaluation process?

– Developing the TOC– As key informants– To pre-test instruments– For feedback on preliminary findings– To disseminate findings to other audiences

10

How to Integrate Evaluation Into Your Work

Key Evaluation Questions (See “Evaluation Framework”)

Are programs/activities being implemented as planned?

To what extent is the program/activity meeting projected outcomes?

What factors are associated with successful vs. unsuccessful outcomes?

Use TOC as Guide: How do you look at your activities over time?Your outcomes? Develop a plan to address these questions on anongoing basis.

11

How to Integrate Evaluation

Develop agreement as to what you need to collect toanswer your key evaluation questions.

• What of this are you already collecting?Start with an Inventory: • Forums held? Activities Conducted? Volunteers Participating?

Number of Mailings, Emails, Phone Calls to Decision Makers?• Press mentions? Membership increases?• Square footage restored? Lbs/Tons of trash removed?• Enforcement actions taken?• Photographs of the area

12

How to Integrate Evaluation

• What else can you collect? What would measure your desired outcomes? What methods can use to collect it?

• What are you going to do with it once collected?• Process for entering, tracking, and analyzing the data• Process for using it in decision-making (sharing with the board,

at staff meetings, when considering project changes, expansions)

• Process for reporting the data to other stakeholders (internal and external)

13

Reporting and Sharing Results

• You’ve developed a theory of change, identified outcomes for your program and measured the results.

• Now it’s time to report your findings and share them with others.

• But who should you be talking to? And how can you present the data so it has meaning?

14

Reporting and Sharing Results

Audience and their Needs? Report Structure?

Outline….• Background and Purpose

• What are the questions that this data seeks to answer?

• Context for the work • Where were you starting from? What was the environment

(social, political, organizational, partnerships, etc.)?

• Include Your Theory of Change (can be simple)• Share goals and what you did to try to achieve them.

15

Describe the lessons learned….

• What have you learned from the evaluation findings about how to do your work more effectively?

• How will your program change as a result of these lessons?

• What program activities will you continue to do?

• What program activities will you do differently?

Reporting and Sharing Results

16

• Were there unforeseen issues that affected your program’s ability to meet your goals and expectations? How will your program address these issues in the future?

• Were there any unintended outcomes or successes that happened as a result of your work?

• Are there results or outcomes that you will look for in the future that you hadn’t looked for before?

Reporting and Sharing Results

17

Make Recommendations

• What programming or program delivery changes ought to be considered?

• What internal or structural changes should be considered? Do you need a plan that includes who is responsible, and/or a list of actions and deadlines?

• Are your recommendations for change explicitly linked to your findings?

18

Make Recommendations

• Prioritize your recommendations – distinguish those of primary importance.

• Consider multiple options for recommendations (menu).

• To the extent you can, anticipate potential obstacles and discuss how you plan to address them.

• Are recommendations concrete and feasible?

19

Frequently Asked Questions

Q. What if our data is incomplete, or messy?

A. Be honest. Start off by saying that this data is limited and presents only part of the picture. Outline the specific reasons that the data is not complete (e.g., we just started collecting it, we lost three staff this year, our computer crashed). Describe what you do know and flesh it out with what you can make an educated guess about. If appropriate, explain what you’ll do differently in the future to make sure data is of higher quality.

20

Frequently Asked Questions

Q. How do we report negative findings?

A. Emphasize what you’ve learned. Don’t hide the disappointment, but highlight any good outcomes of the program (including unexpected ones), and then analyze why the program or strategy as a whole didn’t turn out as hoped. Finally, explain how you’re going to adjust for the future.

21

Get Feedback

Before you put your report in front of a board of directors or a funder, find another pair of eyes to review it.

• Choose someone who is not heavily invested in the success of your program, to provide a skeptical eye.

Share findings with staff, for internal feedback. Probe for their understanding of the data’s implications and for their suggestions for report recommendations.

22

Sharing Results

 

In addition to your primary audience, consider sharing with other stakeholders (community members; collaborative partners; other environmental groups – regional and elsewhere).

Invite responses by holding a community forum or smaller focus group meeting to discuss findings, recommendations, and next steps.

23

Other Forums for Sharing Results

 

•Use poster displays to share results – post in schools, community centers, environmental centers, by river walks, etc.

•Use postcards or flyers that community members can pick up at the local grocery store, laundromat, etc.

•If appropriate, include options for getting involved: action steps and who to contact.