Result Based Monitoring and Evaluation for Agriculture and Rural

Development

Stella Clara Massawe

ReSAKSS-ECA, ILRI

Safari Park Hotel, Nairobi on 25th- 26th June 2012

Session One

Learning Objectives Session 1

o Discuss the concept of Result Based M&E

o To discuss the role of M&E in agriculture and rural

development interventions

o Developing a theory of change in M&E context

o Practical Assignment

Monitoring

• Tracks what is happening within an intervention.

• Continuous process of observation, interpretation and institutional learning

• Is an internal management tool whose purpose is to:

• to take timely corrective action in the day-to-day

management decisions

• Track progress, or lack thereof, in the achievement of

results.

• Facilitate subsequent evaluation

Evaluation 1

• Periodic

• Systematic assessment of ongoing or completed interventions

• Looks at their design, implementation and results

• Deal with strategic issues such as: Performance, quality, relevance, effectiveness, efficiency, and sustainability.

• Addresses various questions:

Evaluation 2

• Addresses various questions:

– Why: What caused the changes

– How: What was the sequence or processes that led to successful (or not) outcomes

– Compliance/ Accountability: Did the promised activities actually take place and as they were planned?

– Implementation process: Was the implementation process followed as anticipated, and with what consequences?

Why M&E System for Agriculture?

• Accountability & Transparency: development effectiveness….national SHs, donors, beneficiaries …..

• Learning: Decision making, learning……

– Key SHs: accountability & development effectiveness

– Management: evidence decision making, right strategy?

– Implementers: delivering on time? Budget? and target?

• Adaptive Management/ Managing for impact: Respond to changing circumstances, increased understanding from knowledge generated by M&E

• If done well contributes ++ to team morale and foster an atmosphere of transparency and professionalism.

What to Monitor and Evaluate

• Best to develop at the beginning of the design process

• Theory of change indicates intervention-to-impact pathways

• Need to define key indicators along the pathways, including their measurements, methods, and data requirements

• Laying out the theory of change for an intervention is important when deciding what to monitor and evaluate

Underlaying Logic or Theory of change

• Theory of how an initiative leads to the desired results,

• Describes the causal logic of how and why a particular intervention will reach its intended outcomes

• Often developed during project conceptualization.

• Involvement of stakeholders to put together a TOC improve programme design

Thinking about impact pathways

• Example from the regional framework document

• Channels of impact on poverty and food security

• A key step in reaching the goals (reducing poverty and increasing food security) is by improving the productivity of existing assets and/or by accumulating more assets

Channels of impact: Examples

Market Access and

Trade Facilitation

Early Warning and Disaster Preparedness

NRM & Agricultural Production

•Outcomes: (i) lower transactions cost, (ii) higher

marketing margins, (iii) lower postharvest losses, (iv)

greater volume/value of trade

•IP: (i) direct via higher income, and (ii) indirect via higher

output-input price ratios and/or greater access to production

technologies

•Outcomes: (i) greater adoption of profitable technologies

and enterprises, (ii) higher productivity, output, and

marketable surplus, and (iii) more sustainable production

activities

•IP: (i) direct via higher food supply and income, and (ii)

indirect via higher real income (lower food prices)

•Outcomes: greater protected productive assets and

investments in times of disaster

•IP: (i) direct via reduced negative effect of disaster on

production, consumption, marketable surplus, and trade

OUTCOMES / IMPACT CHANNELS INTERVENTION AREA

Components of a Theory of Change Model

Should include the basic elements of theory of change:

causal chain, outside conditions and influences

key assumptions

A theory of change must

• Depict a sequence of inputs, the project or policy will use, activities, outputs, outcomes and impacts expected

• Identify events or conditions that may affect obtaining the outcomes

• Identify the assumptions the programme is making about cause and effects

• Identify critical assumptions that, based on the policy and environmental context and review of literature, the evaluation needs to examine

Components of TOC

Your Planned work (Implementation) • Describes what resources you think you need to implement

your program and what you intend to do.

– Resources/ Inputs: include the human, financial, organizational, and community resources a program has available to direct toward doing the work.

– Activities: What we do. Actions taken to convert inputs to outputs. Stated with a verb (“market”, “train”, “deliver”).

– Outputs: What we produce. Are the direct products (tangible goods and services) resulting from program activities. These are usually expressed as nouns.

Components of TOC Your Intended Results

• All of the program’s desired results

– Outcomes: Why we do it. Are the specific changes in program participants’ behavior, knowledge, skills, status and level of functioning.

– Results likely to be achieved once the beneficiaries use project outputs (usually achieved in short-to medium term)

– Final outcomes/ Impact: Long-term changes that result from accumulation of outcomes.

– Is the fundamental intended or unintended change occurring in organizations, communities or systems as a result of program activities. Can be similar to strategic objectives.

Modeling Theory of change

• Gertler et al., 2010 indicates that theory of change can be modeled in various ways e.g.

– Using theoretical models,

– Logic models,

– Logical frameworks and outcome models,

– Outcome models,

– Result chain

A result Chain

Outcomes • Intermediate effects of outputs on

clients

Outputs • Products and services produced

Activities

• Tasks personnel undertake to transform inputs to outputs

Inputs • Financial, human, and material

resources

Goal (Impacts)

• Long-term, widespread improvement in society

Imp

lem

en

tati

on

R

esu

lts

Supply Side

Supply +

Supply

Results-Based Monitoring: Agriculture Extension Programme

Outcomes • Increased better farming skills; increased

adoption of new technologies

Outputs • Number of farmers visited, number trained in

field days

Activities • Farm visits, farmer field days, distribute

handouts

Inputs • Extension workers, materials

Goal (Impacts)

• Higher yields, higher income levels; increase access to food

Theory of Change Template

Strategies (5) Assumptions (6)

Influential factors (4)

• Foreseen budget for fertilizer subsidies is available in time

• Climatic conditions

• Government policies

• Macro-economic conditions

• Programme implementation capacity

Problem Issue (1) Low productivity, poor maize

harvest, inadequate food, need for food aid

Community Needs/ Assets (2) • Almost 38% of the population

needed emergency food aid. • Fertilizer is not affordable

Desired results (3) (Outputs, outcomes and

impacts)

• Fertilizer is more accessible

• Fertilizer is more affordable

• use of fertilizer increases

• soil able to support increase food availability

• Reduce hunger

• Increased income (from sales)

Session two

Session 2: M&E framework

• To discuss the meaning of M&E framework

• To discuss the relationship between TOC and M&E framework

• Discuss examples of M&E frameworks

• Developing an overarching FW

Participatory Discussion

• What is the M&E system?

• Using the case of Agriculture M&E framework in your country

What is a M &E Framework?

• Builds on the Theory of change

• Lists each indicator from the program logic model

• Presents how indicators are defined and calculated

• Defines Who, What, When, How by:

• Identifying who is responsible for data collection

• What the data source is ( the methods we use to collect data, report and disseminate)

• When and how often an indicator is measured

M&E Frameworks (1)

• M&E Framework outlines the plan for monitoring in concrete steps providing the who, what, where and when

• Define relationships among inputs, activities, outputs, outcomes, impacts and external factors.

• Assist with thinking through programmatic strategies, objectives and planned activities, and whether they are indeed the most appropriate ones to implement.

• Plan about implementation of M&E activities

M&E Frameworks (2)

• Developing a corresponding M&E plan that acts as a monitoring tool by defining how information from the programme will be tracked.

• Develop the framework and plan before activities are implemented.

• Different kinds of interventions will need different kinds of frameworks, tools and indicators.

• If M&E activities and tools are considered and built into programmatic work from the start, the resource and time burden is minimized.

Designing, Building and Sustaining agriculture M&E system

•Borrow from the model: Ten steps to Designing, Building & Sustaining a Result-Based M&E System –Kustel and Rist, 2004

Step 1: Readiness assessment should be conducted to understand:

• Willingness of the stakeholders

• Incentives and Demand for M&E, what disincentives may hinder

• Organizational Capacity for M&E (technical, financial, managerial)

• key players that will be responsible for program implementation.

• Roles and responsibilities

• Barriers

M&E technical staff, project managers and senior managers

involved in strategy development should be involved

Step 2: Agreeing on Specific Outcomes to Monitor and Evaluate

• Generate interest in assessing the outcomes and impacts in addition to the implementation issues

• Need to be derived from strategic priorities of the country

• Involves all relevant stakeholders to arrive at a mutually agreed set of goals and objectives for the program.

• Linkages: Programme strategy-- theory of change—translate into key set of indicators

• Clearly setting outcomes-and deriving indicators based on them-essential to designing and building result-based M&E system M&E technical staff, project managers and senior managers

involved in strategy development should be involved

Step 3: Select Key indicators to monitor outcomes

• Numerical measurement that tracks progress (or not) toward achieving an outcome

• SMART---Specific, Measurable, Achievable, Relevant and Time-bound

• Decide how trends in agreed outcomes will be measured.

– For example, if children’s schooling were an important outcome for a program: use the proportion of children enrolled in school, test scores, school attendance, or another metric?

• Indicator development drives: data collection, analysis and reporting

• The costs of measurement need to be considered

• Where possible reduce costs by using already available indicators

Minimum set of indicators (2)

• Selecting indicators

• All indicators should be neutrally stated, not as “increase in” or “decrease in”

• Feasibility of deploying each indicators: Completing each cell in the matrix below

Matrix for indicators

indicator Data source

Data collection method

Who will collect?

Frequency of collection

Cost to collect data

Difficulty to collect data

Who will analyze and report data

Who will use the data

Step 4: Baseline data, where are we today • Measurement of initial conditions, preprogram data

• Very helpful in assessing the program’s impact

• Provided qualitative and quantitative information

• Sources of baseline data can be primary or secondary data

• Decide who is going to collect data and how, develop data collection instruments

Step 5: Planning for Improvements Establish Realistic Result Targets:

• These targets can also be used to monitor results.

• This effort includes setting periodic targets over time (for example, annually or every two years).

• Consider the duration of the likely effects of the program

• Important to consider other factors (political..)

Step 6: Monitoring results

• Involves the collection of good-quality data.

• Data sources: Primary,& Secondary data sources, Project data, Spatial data

• Establish data collection system ---------Appropriate forms for record & data management systems …Staffing: enumerators, data entry

• Leverage Partnerships in data collection management, analysis and dissemination

Step 7: Using Evaluations

• Evaluation important in supplementing information require to gauge whether there are achievements with regard to impacts

• Evaluation helps in in indicating whether:

– We are doing right things (strategy)

– We are doing things right (operation)

– There are better ways of doing it (learning)

Data Collection Methods

Conversation with concerned individuals

Community Interviews

Informal/Less Structured Methods

Field visits

Reviews of official records (MIS and admin data)

Participant Observation

Key informant interviews

Focus Group Interviews

One-Time Survey

Questionnaires

Panel Surveys

Field experiments

Census

More Structured/Formal Methods

Direct observation

Steps 8-10

• Step 8: Reporting: Feedback sessions & strategic communication activities, appropriately targeted to different audiences--------- M&E, COM, KM, PRO, MA Reports,

newsletters, issue briefs, website, CDs

• Step 9 & 10: Using findings and sustaining the system

Session Three

Case Study: CAADP M&E

Framework

Rationale for common M&E framework:

The need to assess impact more broadly, adding up to more than of progress and performance associated with individual pillars:

(i) to comprehensively assess whether and how investments and policies surrounding the CAADP implementation are having their desired impact on targeted growth and poverty reduction goals

(i) individual pillar M&E frameworks focus on specific targets without considering how they interact with those of other pillars (e.g. through price effects) to affect the overall CAADP goals and objectives

Idea behind M&E framework

Rationale for common M&E framework

• A mechanism by which processes put in place, commitments etc can be regularly and transparently measured against stated targets and if necessary can lead to the revision of the processes – Are countries achieving the targeted growth rates? (6%) – Are countries investing at the targeted level? (10%) – Are these investments having their intended impacts on poverty and hunger?

• To bring cohesion across the different systems being developed to track specific components of CAADP, including the individual CAADP pillar M&E systems, APRM and MAF

• To inform the review processes established by CAADP-PP (mutual, peer and progress reviews)

• To further inform policy-making and dialogue

http://www.resakss.org/

THANK YOU

References

• World Bank (2004). Designing and Building a Results-Based Monitoring and Evaluation System: A Tool for Public Sector Management, A Workshop for Government Officials and Their Development Partners, Presentation made on 20th October, 2004

• Gertler,PJ., Martinez, S.,Premand,P., Rawlings,LB., Vermeersch, CMJ. (2011) ImpactEvaluation in Practice, The International Bank for Reconstruction and Development / The World Bank, Washington DC., USA

• Kusek JZ. and Rist RC (2004).Ten steps to a results-based monitoring and evaluation system : a handbook for development practitioners, The International Bank for Reconstruction and Development / The World Bank, Washington DC., USA

• Benin, S.; Johnson, M.; Omilola, B.; Beintema, N.; Bekele, H.; Chilonda, P.; Davis, K.; Edeme, J.;Elmekass, A.; Govereh, J.; Kakuba, T.; Karugia, J.; Makunike, R.; Massawe, S.; Mpyisi, E.; Nwafor, M.;Olubode-Awosola, F.; Sanyang, S.; Taye, B.; Wanzala, M.; Yade, M.; Zewdie, Y. Monitoring and Evaluation (M&E) System for the Comprehensive Africa Agriculture Development Programme (CAADP). ReSAKSS Working Paper No. 6. International Food Policy Research Institute (IFPRI). Washington, D.C.:

Exercise 1

Constructing and Using a Theory of Change Task objective: to enable the participants practice how to develop a theory of

change for their programme Estimated time: 30 minutes

Task 1 • Consider a programme you are currently working

with or one which you are familiar but not directly involved. Develop a graphic showing of the theory of change for this programme or project.

• Be sure to identify the assumptions underlying the programme or project

• Use simple theory of change template (please only include a few outputs and activities)

• Reference pages 150-171