1

Outcomes and Indicators

OMG Center for Collaborative Learning

January 9-10, 2008

Chesapeake Bay Funders Network

Program Evaluation

Training Workshop

2

Building an Evaluation Plan

I. Theory of Change/Pathway Map

II. Outcomes

III. Indicators

IV. Methods

V. Putting Evaluation to Work for You

3

What are outcomes?

Outcomes are changes in behavior, skills, knowledge, attitudes, condition or status.

Outcomes are related to the core business of the project, they are realistic and attainable, they are within the project’s sphere of influence, and they are appropriate.

Outcomes are what a project is held accountable for.

4

What makes an outcome logical and reasonable?

There is a logical connection between our activities and the outcome.

It is reasonable to believe that the outcome can be accomplished within the timeframe that we have established. Based on:a. Our previous experienceb. Context (e.g. current or upcoming political changes)c. Resources

Start with Logical and Reasonable Outcomes

5

What makes an outcome logical and reasonable?

The information we are collecting to track this outcome will help us to know more about how we might improve this project or our organization.

Each outcome specifies:

a. The group or condition or process targeted for change and

b. What we hope to change about it

Start with Logical and Reasonable Outcomes

6

The group or condition or process targeted for changeExample 1: FarmersExample 2: Water Quality

and

What we hope to change about itExample 1: Use of Fencing and Buffers to Restrict Cattle Access

to Creek Example 2: Improved (e.g. reduced sediment concentrations).

Start with Logical and Reasonable Outcomes

7

Examples of Outcomes

Short Term Farmers learn benefits of fencing and buffers

Farmers discuss options with agricultural agents, other colleagues

8

Examples of Outcomes

Long Term

Farmers install buffers and fences to restrict cattle access to the creek

Sediment and nutrients from cattle in creek decrease allowing creek to meet designated uses

9

Indicators and Targets

10

What are Indicators?

Indicators are specific characteristics or changes that represent achievement of an outcome.

Indicators are directly related to the outcome and help define it.

11

What are Indicators?

Indicators are measurable, can be seen, heard or read, and make sense in relation to the outcome whose achievement they signal.

Indicators are what you measure to determine whether participants have achieved an outcome, or to determine the presence or absence of a condition or event.

12

Targets

Targets specify the amount or level of outcome attainment that is expected, hoped for, or required.

Targets include the level of change in the indicator and the time by which the desired change must be completed.

Example: The percentage of farmers in x county using buffers and fences to keep cattle from the creek will increase from 30% to 60% over the next three years.

13

Targets: Some words of caution

Set targets that are realistic so they are not over or under ambitious.

If the target statement indicates change in magnitude (i.e., increases or decreases), be sure to specify the initial levels and what is considered a successful change.

Be sure target statements are in sync with meaningful project time frames.

14

Targets: Some words of caution

Performance targets should be specified in advance (i.e., when deciding to measure outcomes).

Be sure there is buy-in regarding what constitutes a positive outcome – when the project has achieved the target and when it has missed the mark.

When lacking data on past performance, it may be advisable to wait to set targets.

15

What is the difference between outcomes, indicators, and targets?

Outcome

Indicator

Increase participation in activism of local constituents through e-network

Number of new e-activists added to e-network during grant period equals or exceeds 2000

Target

16

Identifying Indicators: Questions to ask yourself

1. What does this outcome look like when it occurs?

2. What would tell us it has happened?

3. What could we count or track?

4. Can you observe it?

5. Does it tell you whether the outcome has been achieved?

17

The “I’ll know it when I see it” rule in action -- some examples:

I’ll knowI’ll know that I have increased the engagement of volunteer river watchers

when I see when I see an increase in the number of calls we receive from river watchers notifying us of concerns warranting attention

and when I seeand when I see an increase in letters sent to local politicians by river watchers raising concerns regarding the river.

Identifying Indicators: Use the ‘I’ll Know it When I See it’ Rule

18

Outcomes Indicators

Short Term (one to two years):

Decrease in the volume of trash and debris thrown into the river

Fewer tons of trash are collected through the trash netting system over a 6 month period than in prior periods

Short Term (one to two years):

Stream is restored to improve protection of the water

X linear feet of stream channel restored

X square feet of riparian buffers restored

Examples of Outcomes and Indicators

19

Outcomes Indicators

Long-term (3 years plus):

Improved stream water quality X % decrease in average pollutant concentration (as measured in an in-stream grab bag)

Examples of Outcomes and Indicators

20

Final Thoughts to Consider When Identifying Outcomes and Indicators

• There is no correct number of outcomes and no correct number of indicators for each outcome

• Be sure to think about when to realistically expect outcomes and when to measure them• Shorter term, Intermediate, Longer Term

• Outcomes, especially long-term, can go beyond the project’s formal conclusion

21

TIMEFRAME

Are changes likely in the timeframe that you’ve defined?

RESOURCES

Do you have staff, money, relationships to monitor over time?

PAST EXPERIENCE

Do lessons learned by similar programs or efforts indicate this change is likely?

PROGRAMMING

Does your initiative provide services directly related to the indicator?

C0NTEXT

Is the indicator appropriate in this context & environment?

IndicatorsLITMUS TEST

So how do we know if our indicators pass the test?

22

Exercise: Outcomes and IndicatorsWhich is an outcome? Which an indicator?

• Reduced amount of impervious surface on properties

• Redirection of downspouts to flow to lawn or garden

• Private Industry Absorbs Some Responsibility for Operations and Maintenance (O&M) for AMD Treatment

• Properties were modified for greater storm-water runoff sustainability

• Company X agrees to construct an AMD treatment system and to have designated 0&M responsibilities for the system which are developed and agreed to in partnership with Y

23

Exercise (Small Groups)

Refer to the partially completed Theory of Change for the Fullojunk Watershed Alliance and the description of the Alliance’s work. – In addition to the example outcome, develop at least

two short term outcomes and one long term outcome for the Alliance.

– Develop indicators for those outcomes. (How will you know the outcome has happened - when you see…?)