UNDP M&E framework for adaptation, VRA and IAS

C. O. Nyandiga and A.F Wittmann

Presented at the UNDP CBA- UNV Workshop, August 2-10, Dakar-Senegal

Monitoring = A continuous data collection and analysis process implemented to assess a project (or program or policy) and compare it with the expected performance

Evaluation = A systematic and objective measurement of the results achieved by a project, a program or a policy, in order to assess its relevance, its coherence, the efficiency of its implementation, its effectiveness and its impact, as well as its sustainability

(source : World Bank)

Monitoring and Evaluation = a systematic and on-going process, that is part of project inception, planning and implementation

M&E is necessary to ensure that the OBJECTIVES of the project are met

M&E Provides project management and stakeholders with on-going

indications of progress or lack of progress in achievements of objectives

Helps identifying problems and successes of projects, during implementation

Provides a basis for corrective actions Helps reinforcement of initial positive results Helps in determining if a project is still relevant

ADAPTATION : What do we want to measure ?

We want to make sure our objectives are achieved : improving the adaptive capacity / reducing the vulnerability of the communities and the ecosystems on which they rely.

What can we measure ?- Coverage : extent to which the project reaches vulnerable

stakeholders (individuals, households, businesses, government agencies, policymakers…)

- Impact : extent to which the project reduces the vulnerability, through policymaking, capacity building…)

- Sustainability : ability of stakeholders to continue the adaptation process beyond project lifetime

- Replicability : extent to which projects generate and disseminate results and lessons of value for replication

Coverage i. Number of households, businesses engaged in vulnerability reduction or

adaptive capacity development activities, as a proportion of households in the community or region targeted by the project.

ii. Number of policies introduced or adjusted to incorporate climate change risks.

iii. Number of investment decisions revised or made to incorporate climate change risks.

iv. Number of stakeholders (individuals, households, communities, etc.) served by new or expanded climate information management systems (e.g. early warning systems, forecasting,etc.).

Impact i. Percent change in stakeholders’ behaviours utilizing adjusted practices or

resources for managing climate change risks, assessed via QBS. ii. Percent improvement in stakeholders’ capacities to manage climate

change (e.g. communicate climate change risks, disseminate information, or make decisions based on high quality information), as relevant, assessed via QBS.

iii. Percent reduction in perceived vulnerability: a. Percent improvement in stakeholder perceptions of vulnerability to a

recurrence of primary climate change-related threat(s), assessed via QBS.

Sustainability i. Number of beneficiaries of project receiving training in

implementation of specific adaptation measures or decision-support tools.

ii. Local (or spatially appropriate) availability of skills and resources necessary to continue adaptation after conclusion of project, assessed via QBS.

iii. Support for project activities among participating communities as assessed by QBS.

iv. Number of outside programmes, policies or projects incorporating project results into their processes.

Replicability i. Number of “lessons learned” from the project

CBA falls within the UNDP Framework for M&E of Adaptation to Climate Change

The specificity of CBAdaptation = it is a challenge to monitor and evaluate it, because :

“Adaptation is not generally an outcome, but rather consists of a diverse suite of ongoing processes that enable the achievement of development objectives under climate change”

“Adaptation cuts across many development objectives”

The results of adaptation can be measured in the long-term, whereas projects have a short duration

It is difficult to decouple climate risks from other drivers (UNDP Framework for Monitoring and

Evaluation of ACC)

M&E aims to ensure and show that the CBA objectives are met :◦ Building adaptive capacity◦ Reducing community vulnerability◦ Increasing the resilience of global environmental

benefits to climate change (including variability) in the GEF focal areas

M&E applies to the 3 CBA outcomes◦ Outcome 1 – Local level (projects)◦ Outcome 2 – National level◦ Outcome 3 – Global level

4 sets of indicators are monitored◦ The UNDP Adaptation Indicators

Adaptive capacities Sustainable community management of natural

resources

◦ The SGP’s Impact Assessment System (IAS) GEB Livelihood / Empowerment

◦ The Vulnerability Reduction Assessment

◦ Volunteerism and Inclusive Participation indicator

4 sets of indicators are monitored◦At the project level◦Aggregated at the national level◦Then again aggregated at the global level

Adaptive capacity fostered among natural resource dependent communities through awareness building, enhanced access to climate change and scenario information, and improved access to alternative resources◦ 2.1 Number of stakeholders at community level (e.g. businesses, community

representatives, CBOs, NGOs) engaged by project and provided with training in climate change risk management and scenario planning.

◦ 2.2 Population covered by awareness building programmes to increase understanding of risks associated with climate change among general public and key stakeholder groups.

◦ 2.3 Increase in awareness of climate change related risks to natural resources (QBS)◦ 2.4 Percentage change in natural resource dependent population with access to

alternative or supplementary livelihood options (QBS)

Sustainable community management of natural resources in the face of climate change promoted ◦ 4.1 Percentage of population in relevant areas engaged in sustainable community

management activities◦ 4.2 Number of measures deployed as part of sustainable resource management

activities.◦ 4.3 Percentage of area of concern in which sustainable resource management activities

are implemented.◦ 4.4 Number of stakeholders (e.g. families/households) benefiting from sustainable

resource management activities (e.g. in terms of increased income or food security).◦ 4.5 Success of sustainable resource management interventions in securing livelihoods

and protecting resources (QBS).

Why ? Measure the Global Environmental Benefits and the Livelihood / Empowerment benefits generated by the project

What ?◦ GEB /

o Biodiversity-(# of species, innovations/new technologies, local/national policies)

o Land Degradation- (ha. of land restored, land sustainably managed, tons of soil erosion prevented, # of innovations/new technologies, local/national policies)

o Livelihood / Empowerment /o Poverty Reduction –(# of households or individuals

who benefited from the project, income generation achieved through the project…)

o Capacity Building –(# of NGOs, community groups whose capacities were increased, #of women participating, support ensured from local / governmental institutions…)

Who, How ? And When ? At project conception, chose one GEB indicator and one LIVELIHOOD / EMPOWERMENT indicatorDuring project development, measure the baseline values of the indicators and prepare your monitoring plan (in project proposal) – During project implementation, measure the evolutions of the indicators (project reports) – RESPONSIBILITY OF THE GRANTEEAt the end of the project, measure the final indicator (final participatory evaluation / final report) – RESPONSIBILITY OF THE GRANTEE

IAS Indicator

to be measured

How it will be measured

When it will be measured

Target value to be achieved by project end

Nb of ha of degraded land restored / combat land degradation

Addition of the surface of all the pilot sites that have been planted, and that have benefitted from resilient farming practices

Regular monitoring after completion of each activity + At the end of the project

4 ha of farmland + 10 ha of forest land

Nb of NGOs and Community Groups participating / capacity building

For all activities, a list of participants is made, desegregating the different community groups (farmers, NGO members, women, elders)

Regular monitoring after completion of each activity + At the end of the project

-One NGO-3 community groups (farmers, women, youth).

Monitoring Plan –

-What you will measure (indicator)-What is the target value of your indicator-How you will measure it-When you will measure it-Who will measure it

Don’t forget to report the indicators in the SGP

database

Arid & semi-Arid zone ecosystems Local population estimates of arid and semi-arid zone species Changes in extent of undegraded arid and semi-arid habitat.Coastal, Marine and Freshwaters: Local population estimates of target aquatic species Changes in extent of healthy coral reef Changes in extent and quality of mangroves (patch size, maximum tree

size).Forest: Local population estimates of forest species Changes in natural forest extent Number of concessionaries in the formulation of timber extractionMountain: Local population estimates of montane species Changes in extent of undegraded montane habitat Rate of deforestation on slopes >20% in project areasAgrobiodiversity: Changes in number of local land races & domestic livestock breeds. Changes in extent of agricultural area using low-input high diversity

production methods Incorporation of local land races and indigenous breeds in national

breeding programmes Certification standards for agricultural products

WHAT Systematically capture and document off-site,

indirect and longer-term impacts- “ex-post evaluation”. Generates GEB Measurements

HOW Using proxy and development indicators while

project is ongoing Using structured QBS during and after project

closure External independent consultantsWHEN Minimum 5 years after project completion-Done

by UNDP EO

What ?The objective of a CBA project is to REDUCE the vulnerability of the community. VRA is a tool that contributes to measure achievement of this objective.

VRA is a form of Participatory Impact Assessment, which focuses on the community’s own perception of its vulnerability and adaptation capacities.

Why ?◦ Measure the community’s perceptions of current and

future climate change risks and adaptive capacities (now and in the long-run)

◦ Measure the obstacles to adaptation, and the assets◦ Evaluate the sustainability of the project◦ Capture qualitative information / knowledge from the

community, essential for project development/management (to make sure the project responds to community priorities) and for knowledge-management

◦ Capture quantitative (thus comparable) information (=scores) that will allow to verify the achievement of CBA objectives (at project / national / global levels)

How ? When ?VRA is composed of 4 indicators, that are transformed into 4 questions (tailored to the local context).These questions form the core of a “VRA participatory workshop”, that is organized at least 3 times in the course of a project, in the community (at the beginning, at mid-course of project implementation, at the end).

UNDP Adaptation Policy Framework Step

VRA IndicatorVRA QuestionIn these examples, we consider the case of a community facing increasing drought risks

Assessing current vulnerability

1. Vulnerability of livelihood/welfare to existing climate change and/or climate variability.

Example: What happens when there is drought? How does this affect you and your community?

Assessing future climate risks

2. Vulnerability of livelihood/welfare to developing climate change risks.

Example: What would happen if drought was twice as frequent? How would this affect you and your community?

Formulating an adaptation strategy

3. Magnitude of barriers (institutional, policy, technological, financial, etc) barriers to adaptation.

Example: What stands in the way of adapting to increasing drought? What means do you or your community have to manage events occurring more frequently?

Continuing the adaptation process

4. Ability and willingness of the community to sustain the project intervention

Example: Rate your confidence that the (project activity) will continue after the project period.

The H-Form = the tool that is used for VRA facilitation and data collection

Where ?Choose an appropriate location in the village, according to what you already

know about the community.The location has to be accepted by / accessible to everyone

When ?Accommodate the community schedule, including women’s schedule

Who ?Facilitating team : 1 or 2 lead facilitators (with leverage and experience ; who know the local

language and culture)1 person to take notes on the H-form1 person to take extensive notes on the side1 person to take pictures and coordinate the logistics

MOBILIZE the grantee as well as community-members.BE GENDER-SENSITIVE

QUICK TIPS-Make sure all the members of the community are invited and informed about the meeting-Create a friendly atmosphere, where everyone is at ease to speak-Promote open discussion (the participants are the “teachers”, because we want to learn from them and capture their perceptions)-Use a language that is appropriate-Prepare your H-forms in advance (as well as all the material you might need)-Inform the authorities in advance

1-Introduction

Present the facilitating team

Explain the context of the workshop (CBA project) and the objectives of the workshop

Explain the “rules” : promote open participation (there is no wrong / right answer; everyone is allowed to speak…)

2-Launch a general discussion about the local environment and climate

Use simple questions and tackle climate aspects one by onewhat is the “usual” climate here ? when does it rain ? when is it hot ? For how long…Are there any storms, droughts ? When ? you can draft a season calendar for example

 3-How does the community experience climate change?

Have you noticed any changes in the past years ? Has it rained more ? less ? When did it rain last year ? for how long ? Is it hotter than in the past ? is the dry season longer ? Are there more storms etc.

 

QUICK TIPS

-Adapt to relevant climatic matters

-Be very specific : referring to “climate change” is too vague in most contexts

-Ask simple questions one by one, so people have time to discuss them

 4-What are the impacts of these changes on your livelihood = VRA QUESTION 1 / Fill out H-form

 5-What will be the impacts if these changes (more storms / more droughts) double in the

future ? = VRA QUESTION 2 / Fill out H-form

The impacts are noted in 2 columns (positive impacts / negative impacts)After the discussion, assessment : how serious are these impacts, in the view of the communityWhat are the possible solutions to limit these impacts ?

 6-What are the obstacles / assets for adaptation ? = VRA QUESTION 3 / Fill out H-form

What prevents you from implementing the solutions that you mention ?After the discussion, assessment : are the obstacles stronger than the assetsWhat are the possible solutions to overcome the obstacles and build on the assets ?

 7-Do you think that with the project, you will be able to adapt further in the future ? = VRA QUESTION 4 / Fill out H-form

(Assess the sustainability of the project, and how people will contribute)After the discussion, assessment : are the participants optimistic about the outcomes of the projects ? are they willing to actively participate ?

 8-Conclusion

Thank people for their participationPresent the next steps of the project

What are the results of a VRA session ◦ QUANTITATIVE DATA = SCORING

Generate a score for each question (noted on 5; 1 shows that the vulnerability is high / 5 shows that the vulnerability is low

Calculate the average from each questionMultiply the total average by 2 to obtain a score

noted on 10

◦ QUALITATIVE DATA = all the information captured during the session, and classified in the different parts of the H-form

Negative impacts Positive impacts Ideas, Solutions

How to use the results of a VRA session◦ The initial VRA results

Are used for project development an incorporated in the project proposal (provides data for baseline assessment / local knowledge / and ideas for project outcomes/outputs)

Provide the baseline vulnerability measurement, reference for project monitoring

◦ The 2nd and 3rd VRA results are to be compared with baseline, in order to monitor the project.

Men Women

Q1 1 1

Q2 1 1

Q3 1 1

Q4 2 2

Total 5 5

Average (noted /5) 1,25 1,25

Average (noted /10) 2,5 2,5

VRA SCORE (/10) 2,5

QUICK TIPS

Always make sure you thoroughly document the VRA sessions

Take detailed notes on your own H-form, Note your personal side-comments –how many people ? Where ? When ? Who facilitated ? What went right / wrong ? What can be improved ? …Take pictures

Don’t forget to report the indicators in the SGP database

Lessons learned from implementation VRA is a powerful way to capture a lot of information from the community It gives an opportunity for mobilization and discussion, and contributes to

fostering community ownership of the project Language and communication are critical Gender appropriateness is essential (in the choice of the location, scheduling,

language, facilitators) : in Morocco, for example, we do specific women’s workshops VRA is a flexible tool : its organization can be tailored to the specific context (for

ex. In Morocco, we don’t always use the big H-form, and we added some questions/discussions to prepare for the VRA questions) – PROVIDED ALL THE INFO IS COLLECTED FROM THE DISCUSSION

VRA can be incorporated in a larger workshop (information about CC, training, rural assessment…) so that the communities are prepared and so that the discussions are to the point.

The main challenge is the QUANTITATIVE DATA (for discussion)How can they be better understood (scale of value is not necessarily translated in numbers) ? How can they be better captured (voting/scoring might block participants; people don’t want to

stand out)? How can they be more reliable for monitoring (composition of the group changes from one VRA

to the next ; participants might over/underestimate the scores in order to ensure the funding… ?

There is a potential for improvement, in order to be able to prove the impacts of our projects.

BASELINE indicators (project proposal)- What are the mechanisms for volunteerism that

already exist in the community before the CBA project (for example, traditional mechanisms for mutual assistance, associations, etc.)?

- Number of volunteers in the community already engaged in climate change adaptation activities before the CBA project

- What are the opportunities or obstacles that could facilitate or impede people from engaging in voluntary activities?

The CBA project aims at reinforcing Volunteerism : these indicators should be monitored over the course of the project

Be sure to capture quantitative / qualitative data on community volunteers and participation during the project : in the activity reports / follow-up on the Volunteerism chart

Inclusive participation : monitor the vulnerable group’s participation (esp. women, youth, people with a handicap, elders, property-less farmers…). In the activity reports / follow-up on the Volunteerism chart

Project Conception Project development / Planning grant Project implementation

Project closing

SGP IAS Chose the relevant indicators (in concept note)

Collect baseline dataIncorporate data in project proposalReport data on SGP database

Evolution of the indicators in project reports (before each disbursement)

Final participatory project evaluation;Include final measurements in final report and K&M document

UNDP adaptation Chose the relevant indicators (in concept note)

Collect baseline dataIncorporate data in project proposal

Evolution of the indicators in project reports (before each disbursement)

Final participatory project evaluation;Include final measurements in final report and K&M document

VRA Prepare 4 tentative questions, based on the main climate risk for the community

Organize VRA session and collect quantitative and qualitative dataIncorporate data in project proposal

Organize mid-term VRA session and collect quantitative and qualitative data (include in Mid-Term report)

Organize final VRA session and collect quantitative and qualitative data;Include data in final report and K&M document

Volunteerism Collect baseline info for project note

Collect baseline dataIncorporate data in project proposal

Evolution of the indicators in project reports (before each disbursement)Participation of volunteers is highlighted in activity reports

Final participatory project evaluation;Include final measurements in final report and K&M document

APR / PIR : Annual project report / Project Implementation reportGlobal Portfolio

Write case studies / share with PMU and ALMFill out SGP databaseQuarterly portfolio report (incorporate all the added indicators / measurement from the projects)Yearly report

National Portfolio

Aggregate indicators

More and more, we are asked to prove our results.

M&E is ESSENTIAL to show that our projects achieve their objectives.

This is why we need to assess the challenges of our system, and try to improve it.

How do you implement our M&E ? How do you explain it to the proponents /

communities, and support them in implementing ?

What are the challenges you are facing ? What do you need to overcome these

challenges ?

Quantitative aspect of VRA ? Consolidate our 4 set of indicators into 1,

without losing any of the information ? Incorporate project outcome indicators ?