How to plan and manage an

environmental education program –

AND evaluate its success

Philip Cox, Plan:Net Limited (www.plannet.ca)

Alberta Council for Environmental Education (ACEE)

October 18th Webinar

A craft that comes by different names…

Outcomes MeasurementPerformance Measurement and EvaluationOutcomes MappingManaging by ResultsResults Based Management

A craft that comes by different names…

Outcomes MeasurementPerformance Measurement and EvaluationOutcomes MappingManaging by ResultsResults Based Management

Introducing Three Outcomes Measurement Tools…

Logic Model

Risk Analysis and Management Table

Monitoring and Evaluation Plan

Project Initiation

Proposal Development

Donor Appraisal

Start Up

Implementation

Monitoring / Evaluation

Phase Out

1.The basis for an introductory conversation with stakeholders

2. The centrepiece of your proposal or plan

3.The basis for workplans and job descriptions

4. A management reference during team meetings

5.A guide for monitoring and reporting on progress & adjusting strategies

6.An aid in designing an evaluation

Logic Model, Risk Analysis and Management Table, and Monitoring and Evaluation Plan are relevant to all parts of the program cycle…

What is Outcomes Measurement?

It is a way of thinking and a set of tools to help us:1. Convert a project idea into a set of activities and

anticipated changes in a way that is:• Mindful of context• In keeping with the time and resources available

2. Keep track of progress against the plan and use this information to:• Make management decisions, • Service accountability relationships and• To be a learning organization.

Outcome Measurement is Part of a Global Trend

• A response to new realities...

– Need to demonstrate value to citizens and taxpayers…

– A response by funding bodies around the globe who are under pressure to provide evidence that their programs achieve results

– EEC, World Bank, USAID, UN, WFP and others need to provide credible evidence of results they achieve

– By mid 90’s, almost all OECD governments had adopted RBM approaches

– In Canada - federal government and all provinces

– Increasingly philanthropic organizations such as Alberta Ecotrust Foundation, EcoAction, etc.

–

What does this mean in practical terms?

• When making a plan, or reporting on progress, we are challenged like never before to:

• Describe the connection between the work we do and the difference we make

If you are planning…• A new web-based information portal for educators on

climate change• You might focus attention on…

– The quantity and quality of information loaded on to the portal– The ‘user-friendliness’ of the web site– The number of users and how they are accessing the portal– And also…– What teachers are looking for on the portal– How they are using the information to inform their teaching– How the teaching-learning experience in the classroom is

altered

Exercise

• Think about a project that you know well

• Craft two outcome statements that reflect the change you want to see

• Share your statement with your neighbour and then in plenary

Where can Outcomes Measurement be used in your

organization?

• Organization as a whole –Donor funded programs

• Partner organizations– Individual Partner Initiatives/Projects

Three Outcomes Measurement Tools

Logic Model

- “Measuring Success…” pages 19 - 23

Risk Analysis and Management Table

Monitoring and Evaluation Plan

The Logic Model• Outcomes Measurement practice revolves

around the logic model

Splash and Ripple!

Splash and Ripple !Another way of thinking about

Activities and Outcomes

Situation Assessment

Before beginning a results based plan..

1. Looking Inward

- Mandate- Competencies- Experience

2. Looking Outward- Issues & Opportunities

3. Clarifying Priorities

Situation Assessment - Stakeholders

Influence

Inte

rest

or

Impo

rtan

ce High Interest/ Importance, High Influence

Low Interest/ Importance, High Influence

Low Interest/ Importance, Low Influence

High Interest/ Importance, Low Influence

Build coalition of support

Spend more time on the other Stakeholders

Consider ways to strengthen or protect interests

Caution - be aware of risks or obstacles

HIGH

LOW

HIG

H

LOW

Discussion

• Generate a list of groups with a stake in the success of your project

• Place them on the stakeholder matrix according to their current– Interest/importance– Influence

Logic Model Format - One VersionInputs

Inputs

Inputs

Inputs

Inputs

Activities

Activities

Activities

Activities

Activities

Impact

Outcomes

Outcomes

Outcomes

Outputs

Outputs

Outputs

Outputs

Outputs

Logic Model Format - Another VersionUltimate

Outcomes

How

What we want

Why

Inputs

ImmediateOutcomes

ImmediateOutcomes

ImmediateOutcomes

ImmediateOutcomes

Activities Activities Activities Activities Activities

Outputs Outputs Outputs OutputsOutputs

Intermediate Outcomes

Intermediate Outcomes

Intermediate Outcomes

Inputs…

• Are the human and physical resources that make a program or project possible.

• Links outcomes logic to budgeting

Example: Train and Mentor Environmental Educators in Outcomes Measurement

Ultimate Outcomes

Intermediate Outcomes

Immediate Outcomes

Outputs

Activities

Inputs Trainers, Materials, Equipment, Communications…

Activities

• Describe the essential work of the program or project – what you DO

• Should be summarized in no more than 5 statements.

Example: Train and Mentor Environmental Educators in Outcomes Measurement

Ultimate Outcomes

Intermediate Outcomes

Immediate Outcomes

Outputs

Activities Assess training needs

Design training materials

Deliver training

Provide follow-up coaching

Inputs Trainers, Materials, Equipment, Communications…

Outputs…

• Short term…a change that is observable as you

complete the activity.

• Each output relates to one activity.• You have substantial control.• Involves

– a specific group of people - those directly involved in the activity, or

– the creation of a product

• The change helps make outcomes possible.

Example: Train and Mentor Environmental Educators in Outcomes Measurement

Ultimate Outcomes

Intermediate Outcomes

Immediate Outcomes

Outputs Defensible report issued

Relevant, user friendly training materials in place

Participants complete training

On-the-job queries addressed

Activities Assess training needs

Design training materials

Deliver training Provide follow-up coaching

Inputs Trainers, Materials, Equipment, Communications…

Immediate Outcomes…

• Observable within the project cycle, following activities

• Describes the potential created by the outputs

• You still have considerable control

• Centres on a very specific group of people - those directly involved in the activity or using the product.

Example: Train and Mentor Environmental Educators in Outcomes Measurement

Ultimate Outcomes

Intermediate Outcomes

Immediate Outcomes

Stakeholder commitment to a training strategy

Trained managers apply Outcomes Measurement approaches and tools in the design and implementation of the projects to which they are assigned

Outputs Defensible report issued

Relevant, user friendly training materials in place

Participants complete training

On-the-job queries addressed

Activities Assess training needs

Design training materials

Deliver training Provide follow-up coaching

Inputs Trainers, Materials, Equipment, Communications…

Intermediate Outcomes…

• Describe the key changes you want to make in the program/project

• Observable at or shortly after the completion of the program/project

• You have influence, but not control

• Usually involves a larger group of people than those directly involved in the program/project

• Are fewer in number; each flows naturally from several Immediate Outcomes

Example: Train and Mentor Environmental Educators in Outcomes Measurement

Ultimate Outcome

Intermediate Outcomes

Agency leaders make evidence- based, results- focused planning and management decisions

Donors and community stakeholders recognize the ‘value-added’ by the agency’s work

Immediate Outcomes

Stakeholder commitment to a training strategy

Trained managers apply Outcomes Measurement approaches and tools in the design and implementation of the projects to which they are assigned

Outputs Defensible report issued

Relevant, user friendly training materials in place

Participants complete training

On-the-job queries addressed

Activities Assess training needs

Design training materials

Deliver training Provide follow-up coaching

Inputs Trainers, Materials, Equipment, Communications…

Ultimate Outcome• Long term

…a picture of a preferred future; the reason why the program or project is important.

• You can only make a contribution and have an indirect influence.

• The change usually involves an even wider group of people

• Is observable well after the program or project is complete.

Example: Train and Mentor Environmental Educators in Outcomes Measurement

Ultimate Outcomes

People/groups identified in environmental education projects engage with the subject matter in a way that helps them become active stewards of the environment

Intermediate Outcomes

Group leaders make evidence- based, results- focused planning and management decisions

Donors and community stakeholders recognize the ‘value-added’ by the group’s work

Immediate Outcomes

Stakeholder commitment to a training strategy

Trained educators apply Outcomes Measurement approaches and tools in the design and implementation of the projects to which they are assigned

Outputs Defensible report issued

Relevant, user friendly training materials in place

Participants complete training

On-the-job queries addressed

Activities Assess training needs

Design training materials

Deliver training Provide follow-up coaching

Inputs Trainers, Materials, Equipment, Communications…

A Progression of Results - Typical Terms

What we Need - Staff Equipment Supplies Office Space Services

Inputs

What we Do - Research Build Train/Mentor Promote Procure Convene

Activities

What we Produce - Networks Courses Trained participants Assessments Publications Infrastructure

Outputs

Capacities Developed - Institutional exchanges/collaborations

Applied knowledge Functioning spaces/infrastructure

System improvements/adaptations New/Improved leadership

Aligned Decision-making

Immediate Outcomes

Performance Improved - Lifestyle shifts

Institutional behaviour changes - policy & programming

Social mobilization

Intermediate Outcomes

Conditions - Social Economic

Civic Cultural Environmental

Ultimate Outcomes

Logic Model Quiz

“Organize a Provincial Climate Change Summit”

Making a Logic Model - Some Tips

1. Vantage Point• Logic Models can be written from many

different points of view - • it is important to be clear who is doing the

activities - spending implementation funds • there is normally only one vantage point in a

framework, it could be• a single entity, or • shared across two or more entities that are

working in partnership

2. When building an logic model consider three variables…

a. Time• Outputs - Immediate

• Immediate Outcomes - within time-frame following related activities

• Intermediate Outcomes - by the end of the project or shortly afterward

• Ultimate Outcomes - beyond the project

b. Control• Outputs - substantial control

• Immediate Outcomes - still considerable control

• Intermediate Outcomes - direct influence

• Ultimate Outcome - indirect influence

• Diffusion• Outputs - persons directly involved in the activity

• Immediate Outcomes - same, possibly early interaction with additional people closely connected (e.g. users, colleagues, family members)

• Intermediate Outcomes - larger teams, organizations, population segments

• Ultimate Outcome - systems (communities, networks, societies)

3. Proportion…• Outputs, outcomes and impacts vary in their

magnitude according to how big (in money terms) and how long the project is.

• If you have a three week, $5,000 project, your outcomes should reflect what is reasonable to expect at three weeks and after $5,000 has been spent

• If you have a three year, $500,000 project, your outcomes should reflect what is reasonable to expect at three years and after $500,000 has been spent

4. Scale - macro to micro

Outcome Measurement can be used at any scale - but which is best for you?

Writing Activity and Results Statements - Tips

5. Separating the Action from the Change– Activities - where you describe what you, as the implementing

group, are DOING

– Results (output and outcome - immediate, intermediate and ultimate) - where you describe what is DIFFERENT

– For example… • Activity: Train 25 participants in Outcomes Measurement• Result: Participants apply Outcomes Measurement knowledge in

when making project plans

6. Avoid ‘run-on’ statements - for example:“Rural population retained in rural areas THROUGH

improving their living conditions BY MEANS OF enhancing rural income, better access to health, social services and

infrastructure.”-An impact statement for a rural development project

“Improved political freedoms of local communities TO access, participate and CONTRIBUTE TOWARDS sustainable

regional development.”- An impact statement for a governance related project

Difficult to locate the actual change you seek

7. Do not lose sight of the people in your outcomes statements.

– Sometimes we write statements without naming the people or groups who are experiencing the change. For example…

“increased awareness”

- Try to include the subject in the statement at all times

8. Avoid tentative wording like: “able to”, “have increased capacity to…”, “contribute toward”. These are vague in meaning and very hard to measure.

Now your turn…..• Describe your group project in

a results logic model?• 1 Ultimate Outcome • 2 Intermediate Outcomes• 3-4 Immediate Outcomes• 4-6 Activities and Outputs

Sample Logic Model WorksheetSITUATION (I.e. context, problem, identified needs, mandate, objectives)

Organization :

Name of project :

Duration :

How ? What do we want ? Why ?

Inputs Activities Outputs Immediate outcomes

Intermediate outcomes

Ultimate outcome

Three Outcomes Measurement Tools

Results Logic Model

Risk Analysis and Management Table Monitoring and Evaluation Plan

Enablers and Constraints…

• Are forces affecting any part of your project both positively and negatively

Risk Analysis MatrixEffects

Significant (3)

Moderate (2)

Minor (1)

Low (1) Medium (2) High (3)

Likelihood

Monitor Risks

Acceptable Risks

Acceptable Risks

Monitor Risks

Monitor Risks

Risk Mitigation Required

Risk Mitigation Required

Risk Mitigation Required

Acceptable Risks

Project Example…

Climate Change Conference

Immediate Outcome:Conference participants deliberate across stakeholder lines on conference topics; they identify climate change strategies around which there is common agreement, as well as strategy areas requiring further discussion

1. What could hamper progress?

2. What is the likelihood of a problem?

3. What effect would the problem have on the Outcome?

• Risk mitigation strategies should reduce the likelihood of an undesirable event, or minimize the effect the event could have on the project.

• Risk areas that remain above the level of acceptable risk should be monitored using risk indicators.

Risk Mitigation & Monitoring

A Typical Risk Analysis and Management Table

Description of Risk

Likelihood of Occurrence

Effect on Planned Results (outputs/

outcomes/ impact)

Rank in Importance

1 = highest

Owner of the Risk

Risk Mitigation Strategy

Risk Indicators

Intermediate outcome level

Immediate Outcome level

Output level

Three Outcomes Measurement Tools

Logic Model

Risk Analysis and Management Table

Monitoring and Evaluation Plan - “Measuring Success…” Appendices, pages 44 - 66

Monitoring and Evaluation - The Distinction

• MonitoringA continuous (or regular) self-assessment of progress -collecting data and comparing current performance with planned activities and outcomes.

• EvaluationIndependent (often external), periodic, strategically focused assessment of a program/project’s continuing relevance, management, governance, results, coverage, external relationships, sustainability, etc.

Conventional vs Participatory Monitoring & Evaluation - Contrasts

Empowerment - to help people initiate, navigate, control

Accountability - to address donor requirements

More frequent, small scale evaluations

Usually Mid and End of Project

Self-evaluation, simple methods adapted to local conditions; open, immediate sharing of results through local involvement

Focus on scientific objectivity, evaluator distanced from other participants; delayed and limited access to results

People identify their own indicators of success

Predetermined indicators of success

Community members, project staff, facilitator(s)

External experts

ParticipatoryConventional

Why

When

How

What

Who

Adapted from Narayan-Parker, 1993: 12 (taken from Estrella and Gaventa, 1998)

Different kinds of Evaluation

• Formative - mid stream, where inquiry focused on areas where the project can improve

• Summative - at the end, where inquiry focused on end results and lessons for future practice

• What is your experience with these two kinds of evaluation?

Indicators…• Evidence that the

program is producing expected results at the output and outcome levels.

• They can be expressed quantitatively and qualitatively.

• Information collected should help managers make wise decisions.

What indicators tell us about the wind…

Who has seen the wind?

Neither you nor I.

But where the trees bow down their heads,

The wind is passing by.-The Wind, Christina Rosetti, 1830-1894

Quantitative IndicatorsExpressed as a percentage, ratio, proportion, or number that can be analyzed statisticallyExamples?

– Number of conferences/public engagement activities held

– Ratio of males to females attending public meetings

– Number and type of research grants distributed

– Proportion of targeted schools making requests for assistance with educational resources

Quantitative Indicators -Your Turn

?• Climate Change

Conference Project

• Immediate Outcome

• Major media outlets cover the Climate Change Summit with ample in-depth news analysis pieces

Qualitative IndicatorsExpressed as a change or comparison between two states of understanding or experience that can be analyzed for patternsRelies on people’s judgment or perceptionsExamples:

– Trends in the way community newspapers cover citizen actions to reduce environmental impacts

– Before-after comparison in the way the largest five land developers are handling wetland areas zoned for development

Qualitative Indicators -Your Turn

?

• Climate Change Conference Project

• Immediate Outcome

• Major media outlets cover the Climate Change Summit with ample in-depth news analysis pieces

Qualitative Indicators - Setting out Markers of Progress

• Sometimes you cannot meaningfully track progress by counting “units” at the beginning (baseline), middle and end – Indicators are not always as easy as “number of people

trained”…

• You may need to set out…– Stages of completion, or– Markers of progress

Qualitative Indicators - Setting out Markers of Progress

• Consider “policy formulation”…– Indicator is ‘Progress in updating ‘x’ policy’

• You could say…– Baseline - ‘0’ updated policy – Target - ‘1’ updated policy

• But how useful is that from a management point of view?

Qualitative Indicators - Setting out Markers of Progress

• Would be more useful to identify progress markers. For example:

a) Agreed need for updated policyb) Policy research underwayc) Policy options formulatedd) Discussion and approvals underwaye) Updated policy approvedf) Updated policy enacted

• Baseline might be ‘a’• Yr 1 target might be ‘c’• Yr 3 target might ‘e’

Indicator ChecklistClimate Change Conference -

Key stakeholder groups commit financial and human resources to implement the Action Plan

Possible Indicators - How well do they score?

a. Trends in the level of financial commitment, by action plan strategy, by source

b. Number and type of organizations publicly endorsing the Climate Change Action Plan

c. Number and type of initiatives underway in the Province, by action plan strategy

Validity - Does it measure the result? Yes No

Is it cost-effective to collect the information?

Yes No

Does it help understand how the program or project is affecting men and women, and/or specific sub-groups of people differently?

Yes No

Does it provide useful information for management decisions?

Yes No

Does it communicate well to stakeholders (public, policy makers, donors)?

Yes No

Indicators, Baseline, and Target

Indicator

Baseline

Target

Meaning

The information to be collected

What the indicator would tell you at the beginning

What you want the indicator to tell you at a specific point in the future

Example

The number of visits to the organization’s new website

720

2,000

Sources of Information

• Data Sources are the people or documents that provide data, not the method of collecting data

• Sources of performance information include: people, individual beneficiaries, groups of beneficiaries, organizations, partners, and documents, etc.

• To ensure reliability, try not to change data sources over time.

Methods of Gathering Data

See “Measuring Success…” Appendix I, Pros and Cons, Pages 49 - 51; Appendix IV, Samples ,Pages 55 - 66

Collecting the Data

• Conventional Methods - surveys, focus groups, observation, document reviews, counts, etc.

Collecting the Data

• Participatory Reflection and Action Methods - modeling/mapping, ranking exercises, calendars, walks, historical profiles

Frequency of Collecting Data

• For each indicator, determine how often data will be collected.

• Indicators for outputs and immediate outcomes are more likely to be collected on a quarterly or six-monthly time period, for intermediate outcomes - less frequently

Responsibility - who does it?

• Spell out clearly who will do the collection and analysis of the data for each indicator

• Ensure local capacity is built and is responsible to ensure sustainability.

• Decentralize responsibility as much as possible.

A Typical Monitoring and Evaluation Plan- Completed example on page 30 “Measuring Success…”

Results Levels

Measurement

IndicatorsBase-Line

Target Data Source/ Method

Report Frequency

& Resp.

Intermediate Outcome

Immediate

Outcome

Output

Now your turn…..

1. Take one outcome statement, brainstorm indicators

2. Use the indicator checklist to refine your best indicators

3. Identify the source, means of data gathering (a.k.a. ‘instrument’), and frequency

4. Repeat, if time permits

5. Share with a neighbour