Climate change adaptation indicators - Sida's Helpdesk …sidaenvironmenthelpdesk.se/wordpress3/wp-content/uploads/2013/05/... · 1 Climate change adaptation indicators . A Logic

1

Climate change adaptation indicators

A Logic framework assessment and indicator analysis of Sida’s bilateral and regional contributions under the framework of the Climate Change Initiative

2013-04-10

Emelie César

Gunilla Ölund Wingqvist

Susanne von Walter

2

Table of Contents 1. Introduction ........................................................................................................................ 3

2. Methodology ...................................................................................................................... 4

2.1 Questions to be answered ................................................................................................. 4

3. Categories and indicators ................................................................................................... 5

3.1 Indicators for Category 1 - Enhancing resilience to climate change ................................ 6

3.1.1 The results matrix ...................................................................................................... 7

3.1.2 The indicators ............................................................................................................ 8

3.2 Indicators for Category 2 - Policy and administrative management for climate change . 9

3.2.1 Results matrix ............................................................................................................ 9

3.2.2 Indicators ................................................................................................................. 12

3.3 Indicators for Category 3 - Education, training and awareness on climate change ....... 13

3.3.1 Results matrix .......................................................................................................... 13

3.3.2 The indicators .......................................................................................................... 14

3.4 Indicators for Category 4 - Climate studies, scenarios and impact research .................. 14

3.4.1 The results matrix .................................................................................................... 15

3.4 2 The indicators .......................................................................................................... 15

3.5 Indicators for Category 5 - Coordination on climate change measures and activities across relevant actors ........................................................................................................... 16

3.5.1 Results matrix .......................................................................................................... 16

3.5.2 Indicators ................................................................................................................. 18

4. Summary and conclusion ................................................................................................. 19

4.1 Results matrices .............................................................................................................. 19

4.2 Indicators ........................................................................................................................ 20

5. Issues for Sida to consider ................................................................................................ 22

References: ............................................................................................................................... 24

Annex 1: Category 1 indicators ................................................................................................ 26





Annex 6: Analysis of objectives and indicators (separate file) ................................................ 35

3

1. Introduction

In 2013 the Swedish Government’s Climate Change Initiative (CCI), including the bilateral and regional part channelled through Sida, is to be evaluated. The purpose of the evaluation is to assess the attainment of the overall objective of the special climate change funding, to draw lessons that can be utilised in future climate change initiatives, as well as to investigate effective ways of channelling funds.

During spring 2012 the Sida Helpdesk for Environment and Climate Change was requested by Sida to develop a preparation study prior to the CCI evaluation. The study defined suitable categories within the field of climate change funding and it also categorised the contributions within the CCI.

This study is a continuation of the previous categorisation-study. It deepens the earlier categorisation of contributions by focusing on indicators to measure results in order to be able to assess whether Sida’s CCI contributions have been relevant, efficient and effective.

The purpose of the current assignment is to identify and, if possible, group indicators for climate change adaptation and to deepen the earlier study on categorisation. Another aim of the assignment is to learn about the use of indicators from the CCI-experience and propose good (or SMART1) climate change indicators for Sida.

The assignment shall attempt to answer the following questions:

• What indicators are used for the different categories? • What indicators are used to measure climate change adaptation, adaptive capacity,

vulnerability and resilience? • Are there similarities between the indicators in the same category? • Are the indicators possible to group? • Are the indicators SMART/measurable? • Do the indicators measure climate change (CC) adaptation? • Are the indicators in line with the purpose of the contributions? • Are the objectives and the indicators appropriate?

The assessment is based on the logic frameworks or results-based management matrices and aims to both be an underlying document for the coming CCI evaluation and to support Sida’s preparations related to contributions with focus on climate change. Hence, the main target group for this study is the evaluators of the CCI and also staff at Sida that work with issues related to climate change.

In Section 1 a brief introduction is given. Section 2 presents the methodology used. Section 3 provides information on the indicators utilised in the CCI contributions, and the indicators are grouped and assessed. Section 4 provides brief conclusions of the assessment, and Section 5 presents some issues for Sida to consider in the future when developing new climate change adaptation projects and assessing if the indicators are SMART.

1 Specific, Measurable, Appropriate, Relevant, and Time bound (SMART)

4

2. Methodology

The 44 contributions, which are included in the assignment, are more or less the same as in the previous assignment2 and all have been selected by Sida. The Climate Change Initiative also includes other, often smaller, types of contributions such as mapping exercises, etc. These smaller contributions were neither included in the previous nor in the current assignment.

The current assignment builds on the categories identified during the previous assignment, which were based on OECD suggestions3, with slight modifications. The categories are briefly described in Table 1.

To identify indicators within each category the Helpdesk reviewed the logic framework assessment (LFA) matrices (or similar) attached to the project documents, and summarised the information in tables that include information related to development objectives, activities, indicators and targets at impact, outcome and output level (when this information is available). See Annex 6 for all tables. After compiling information related to the indicators, the Helpdesk summarised initial perceptions of the indicators for each contribution in a comment to facilitate analysis.

In the previous study each contribution was categorised into one or more of the six selected categories. The categories were given different weight depending on the perceived focus of the contribution. The assessment in the current assignment has focused on each contribution’s primary category. Hence, the Helpdesk analysed each LFA with a focus on the contributions’ primary category/categories and selected activities, indicators and targets that fall within this specific category.4 For each category two or more subcategories were identified to group adaptation indicators.

As many of the LFAs lack indicators the Helpdesk in some cases made suggestions of indicators that could be used. These suggested indicators are not necessarily recommended by the Helpdesk. Rather, they should be seen as suggestions on how indicators could be formulated based on the targets, activities or milestones already identified in the LFA matrices.

2.1 Questions to be answered During the review of the LFA/results matrices and indicators, the Helpdesk has attempted to answer the following questions:

• The results matrix - Are the log frames/results matrices easy to understand and well thought-through? - Are the objectives and the indicators appropriate? - Are the indicators measuring the stated objectives? - Are the indicators SMART/measurable?

• The indicators

- Are there similarities between the indicators in the same category? - Are the indicators possible to group?

2 All contributions from the previous categorisation-study except five were included in this study. The selection was made by Sida, Elisabeth Folkunger. 3 Lamhauge et al.( 2011) 4 Note: For category 6 – Funds the Helpdesk did a revision of the categories and included these contributions within their secondary category.

5

- What indicators are used for the different categories? - Do the indicators measure climate change adaptation? - What indicators are used to measure climate change adaptation, adaptive capacity,

vulnerability and resilience?

3. Categories and indicators

In the previous categorisation-study, six categories where identified based on an OECD report prepared for the Task Team on Climate Change and Development5. The categories were slightly modified to be more suitable for the assignment and included: 1) Enhancing resilience to climate change; 2) Policy and administrative management for climate change; 3) Education, training and awareness on climate change; 4) Climate studies, scenarios and impact research; 5) Coordination on climate change measures and activities across relevant actors; and 6) Climate Change funds. The same categorisation and definitions have been utilised in this study. See Table 1 below for further description of the categories.

During the current assignment Category 6 (Climate change funds) has been excluded, and the programmes included under Category 6 are assessed under Category 1 (Enhancing climate change resilience) and Category 2 (Policy and administrative management for climate change). Thus, in the current assignment only categories 1 to 5 are included.

5 Lamhauge et al. (2011)

6

Table 1. Categories of climate change interventions

Category Description

1 Enhancing resilience to climate change

Implementation of initiatives that enhance the resilience to climate change through general livelihood improvement measures. Also including risk reduction measures or sectoral measures such as water conservation, irrigation, infrastructure, and flood prevention.

2 Policy and administrative management for climate change

Implementation or improvement, monitoring and enforcement of legislation, policies, strategies for climate change mainstreaming, and taking into consideration all stakeholders. Improvement of administrative structures, institutions and systems for climate change integration including institutional capacity development.

3 Education, training and awareness on climate change

Dissemination of information on climate change risks, improving adaptive capacity, and training activities aimed at changing behaviour, or increasing disaster preparedness.

4 Climate studies, scenarios and impact research

Development of climate change studies, scenarios and climate impact studies, collection of data and information, analysis, development of tools and equipment necessary to better understand climate change and associated vulnerabilities.

5 Coordination on climate change measures and activities across relevant actors

Creation of linkages between institutions, participation of stakeholders in dialogues and decision making, strengthened community of practice on climate change, sharing and managing knowledge, lessons learned and good practices, and use of research for dissemination and policy making.

6 Climate Change Funds Establishment of, and capacity to manage, climate change trust funds for implementing regional, national or local climate change strategies and actions plans.

Based on Lemhauge et al. (2011), modified by Authors.

3.1 Indicators for Category 1 - Enhancing resilience to climate change There are 12 projects included under Category 1:

- Mali – IUCN REDDIN - Mali – Norwegian Church Aid - Burkina Faso – MAHAR - Burkina Faso – IUCN adaptation fund - Cambodia - UNDP SGP Support CCBAP - Regional Asia - MFF phase 2 - Cambodia - Forum Syd - Bangladesh – CC multi donor trust fund (BCC MDTF)

7

- Mali – RESO - Bolivia – Agua Tuya - Bolivia – Proagro I - Bolivia – Proagro II

3.1.1 The results matrix

Similar to the other categories the results matrices for category 1 are often difficult to read and follow. Large programmes with many objectives, sub-components, etc. are often more difficult to understand than those with relatively few and well formulated objectives. Quantity appears to dominate over quality. All of the LFA matrices are developed differently. The LFA matrices that are based on the RBM approach (with impact, outcome and output objective) are generally easier to understand than others. In some cases the LFA matrices are too short and do not seem to grasp the contents described, or the results stated, in the programme document.

The development objectives are stated either in the program document or in the results matrix, or both. Most objectives are the same in the text and the LFA but in some cases the text contains objectives that are not mentioned in the LFA.

Often impact level or overall objectives are well formulated and in line with enhancing resilience to climate change e.g. adaptive capacities, ecosystem resilience, livelihood opportunities and living conditions are areas which are included in many objectives.

Appropriateness of the objectives and the indicators Indicators are not really indicators:

In many cases the indicators are targets (mål) rather than indicators (mått). Examples of these types on “indicators” include

- 25% of degraded land recovered: This statement is the actual goal or target which the project/programme aims for. It is, however, easily translated into an indicator, for instance ‘share of degraded land recovered’.

- Livelihood of population improved by at least 15% after 3 years: This statement is also a target rather than an indicator and the Helpdesk suggests that the statement is reformulated into – ‘Share of population with improved livelihood’. Furthermore, the term ‘livelihood’ could be specified (e.g. Variability in agricultural yields or income over a multi-year period; or similar)

Indicators which are difficult to interpret

Some of the stated indicators and objectives include words or phrases that need to be clearly defined. This is particularly the case for livelihood indicators. One example of a phrase that needs to be qualified is “quality of life”. Without a definition it will be difficult to measure quality of life, and it will be difficult to compare results over time.

Quantity indicators vs quality indicators

To attain some goals it is sometimes better to use quantitative indicators rather than qualitative indicators and vice-versa. It is often easier and less time-consuming to measure quantity than quality, which is probably why more quantitative than qualitative indicators are to be found within category 1. Examples of quantitative indicators within category 1 include:

- ‘Proportion of land covered by forest’. This indicator measures an increase or decrease of forested area. However, it does not provide any information regarding the type, or the

8

quality, of the forest. It is a large difference between a new forest plantation and a rain forest in terms of biodiversity, biomass, ecosystem services, etc.

3.1.2 The indicators

The indicators for Category 1 are possible to group. The Helpdesk suggests the following groups:

1. Ecological indicators 2. Livelihood indicators 3. Technical indicators

Most indicators within category 1 are state indicators (describing the current state/situation). The indicators are usually more practical and have a relatively strong link to climate change adaptation and resilience compared to the other categories. Many category 1 contributions include very clear adaptation objectives and indicators e.g. "rehabilitated water reservoirs effectively adapted to climate change" and “number of new adaptation initiatives implemented”. There are also often other activities and indicators presented that do not have a direct link to climate change adaptation but has a strong connection to resilience e.g. "improved agricultural yields" and “early warning system in place (yes/no)”. In some cases, there are no indicators stated in the LFA matrix. As stated above several of the indicators articulated are targets rather than indicators. Often these “indicators” are well in line with the stated objectives. However, they need to be formulated differently, so that they are to measure. (See Annex 6 for more detailed information on indicators for each contribution). In these cases the helpdesk has made a suggestion of indicators based on stated objectives, activities, and or target.

Objectives are often stated for impact, outcome and output level, but indicators or targets are often only stated for one or two levels, often output or outcome and rarely for impact level.

Examples of category 1 indicators are listed in Annex 1. Below, more information related to the use of i) ecological indicators, ii) livelihood indicators, iii) technical indicators are provided.

Group 1: Ecological indicators: There are relatively few indicators that fit under the description ecological indicators. Most have been suggested by the helpdesk out of interpretation of objectives and targets. Ecological indicators are often distinct and clear but demand a lot of capacity and time to be possible to be measured properly. Ecological indicators seldom directly measure adaptation to climate change but are often an appropriate measure to indicate increased or decreased adaptive capacity, vulnerability and resilience to climate change.

Group 2: Livelihood indicators: As for ecological indicator, livelihood indicators are often easy to understand and are often highly relevant to the stated objective. However, some of these indicators are relatively difficult to measure and need clearer definitions, e.g. how is “quality of life” and “reinforced livelihoods” defined.

Group 3: Technical indicators: Most category 1 indicators falls within the sub-group technical indicators. These indicators are relatively easy to measure and give a clear result, for example “number of dams built”.

9

However, these types of indicators do not say much about quality or state, they rather give information on quantity.

3.2 Indicators for Category 2 - Policy and administrative management for climate change A large share of Sida-supported projects include aspects related to Category 2 and a high number of indicators are identified in the CCI portfolio.

Thirteen projects under the climate change initiative are included under category 2:

- Bangladesh – CDMP UNDP - BCC MDTF - Bolivia, PASAP - Bolivia National Forestry Action Program - Cambodia – Climate Change Alliance - Cambodia – Local Governments and Climate Change - Mali – RESO Climate - Regional Africa –CC IUCN PREMI - Regional Africa – EAC CC Master Plan - Regional Asia – SEAFDEC - Regional Asia – Mekong, Climate Change Adaptation Initiative - Regional Asia – Strengthening Governance of climate change finance to benefit the poor

and vulnerable - Regional Asia - Core Env Program and Biodiversity Conservation Corridors initiative

3.2.1 Results matrix

In general, for Category 2-projects, the results matrices are not well developed, which sometimes make them difficult to understand and follow. Especially when the programmes are large and complicated, including different projects or components, the programmes are difficult to comprehend just by looking at the results matrices. The format of the results matrixes/log frames could be developed to be easier to understand (although results matrices based on RBM generally are easier to understand than those using the LFA structure).

In some cases there is not one matrix but several different; each containing information related to different things, e.g. one matrix related to objectives, one for activities or one for indicators. In these cases, the different matrices are not always easy to match together (for instance, which activity and which indicator belongs to which objective). In some cases there is no results matrix available at all, but the indicators must be found in the text.

Often, the stated indicators are de facto not indicators but activities, targets or milestones. In some cases, only objectives and expected results are stated, not measurable indicators. In these cases, the helpdesk has provided suggestions on how to formulate indicators related to the objectives/ activities/targets/milestones (see Annex 2 and Annex 6)6.

Appropriateness of objectives and indicators The development objectives are stated either in the program document or in the results matrix, or both. Most of the times, the different sources provide consequent information, i.e. the development objective is the same in the text as well as in the results matrix. Based on the

6 The Helpdesk has attempted to reformulate the actions, targets or milestones into measurable indicators. This does not imply that the Helpdesk recommends the use of these indicators, it is rather suggestions on how to formulate indicators.

10

development objectives of the projects in Category 2, the (impact level) indicators should measure the following:

- Vulnerability or resilience to adverse events, or preparedness for CC (institutional level) - Institutional capacity ( including ability to implement strategies and action plan) - Quality of planning and implementation of CCA - Quality of strategies - Sustainable development/economic growth - Ecosystem quality/quantity or resilience, or the sustainability of resources

The objectives appear to be relevant and well in line with needs in the country or region, and relate to climate change, resilience, vulnerability, adaptive capacity, etc. Furthermore, the specific (or outcome) objectives often link to the development objective, so the logical chain is adhered to in most cases.7 The impact indicators are mostly more complex, vague and unspecified than the outcome or output objectives (in the cases when these levels are separated and specified).

It is difficult to measure administrative or institutional capacity, or the implementation of certain strategies. Almost by necessity, there are shortcomings in the indicators attempting to measure these aspects. Below, difficulties with Category 2-indicators are identified and systemised: indicators not providing the requested answer; indicators that are too vague, or indicators that are too output-oriented.

Indicators not providing the requested answers

The indicators often attempt to measure the stated objectives, or at least most parts of them. However, the indicators usually fail to provide a full picture of the level of attainment towards the goal. The indicators are mainly process-related indicators, not measuring “vulnerability” or “institutional capacity” or “ability to implement” as ought to be done considering the development objectives. Rather, the indicators relate to actions and measures to improve policy, planning, etc. Examples of indicators that measure actions or measures, that may or may not lead to the attainment of the goals, are:

- “Number of policies designed and adopted/ratified”: This type of indicator provides information on stated priorities by the parliament (or the authority with the mandate to adopt/ratify agreements or policies) and ability to influence the agenda, but does neither say anything about resilience, adaptation capacity or responsiveness, nor about resources de facto allocated or the level of implementation

- “Annual financial audits performed”: The indicator provides information on administrative capacity in the sense that the institution is able to manage its financial resources according to certain standards, keep books, and go through financial audits and thereby indicates that the financial resources are used for agreed purposes. However, it does not provide information on institutional capacity or level of preparedness for climatic events.

7 Only in a few cases does the Helpdesk find that the objectives are not well elaborated. In one case, the objective is to “develop a CC master plan”. The proposed indicator does indeed measure the fulfilment of this objective: “A CC master plan developed (yes/no)”. However, it would be more interesting to receive information on whether or not the master plan is adopted and implemented by the countries.

11

- “Strategies and plans reflect the influence of the climate change adaptation initiative”: This indicator provides information on if the funded initiative is well-known rather than the quality of the strategies and plans or the level of climate change integration, i.e. the wanted results.

There are also some examples of indicators where there are attempts to measure ‘implementation’. It could be related to developing policy instruments to incentivise certain behaviour; or measuring financial and human resources allocated to a certain area. The Helpdesk believes that these types of indicators are valuable and should be complementing the more administrative indicators.

Indicators that are too vague

In other cases, the indicators are too vague or unspecified to be measurable. This is often relating to abstract concepts and areas that are difficult to measure, such as “strengthened institutional capacity” or “awareness amongst decision-makers”. These objectives are indeed difficult to measure, which is exemplified below:

- “Effective policy (established/endorsed) to streamline CC initiatives across sectors”: This indicators will tell us if there is a policy for mainstreaming or not, but it may be difficult to measure the effectiveness of it. Effectiveness could be related to whether or not it is fit for purpose and delivering results. However, either the indicator could be rephrased to include information on for instance: (i) does such policy exist?; (ii) are sector policies reviewed and revised to take climate change considerations into account?; (iii) is climate change mainstreamed into sector policies?; and (iv) impact of implemented policies. Due to the vagueness of this indicator, it is not certain exactly what will be monitored or what type of information that will be provided, as it could include either of i), ii), iii), iv) or all of them or, indeed, something different.

- “Percentage of people with enforced livelihood”: Similarly with the above, this indicator would provide valuable information although it appears to be difficult to measure as it is vague and not enough specified. What is meant by “enforced livelihood”? Is it relating to household income, job opportunities, food security, or other? These types of indicators are important, in order not to have only output-related indicators, but they should be qualified.

- “Legislation enforced (e.g. related to supporting use of low-carbon equipment and behaviours)”: It is very positive that not only the availability of the law but also the implementation and enforcement of it is monitored. But how will it be done? For instance related to the above example on low-carbon development, what will be measured? Overall electricity use/capita, or total of green-house gas emissions, or number of projects financed, or number of complaints filed/sanctions issued?

It appears that for Category 2 projects, these types of indicators, although important, are often abstract, vague and qualitative, with sweeping statements (such as “enhanced awareness ” or “level of integration”). Important as they are, it would be good to improve measurability by being more precise and qualify the indicators. If the indicators are too vague and qualitative, accountability may be negatively affected. Who is assessing the goal achievement: is it the project implementer? Are there perhaps incentives to perform good assessments as it is your own results that are being under scrutiny? It would be more valuable to include at least some objectively verifiable indicators, to complement the more “assessment”-based indicators.

Too output oriented indicators

12

Another typical challenge in identifying indicators reflecting the objectives in Category 2 projects is that the indicators are too output oriented and specific to give a comprehensive picture of attainment of the goals. Examples include:

- “The office fully equipped/operational/number of meetings held/number of reports issued”: These types of indicators indeed provide an initial measure of institutional capacity, because without a functioning office, or ability to arrange meetings, etc., it is hard to perform any substantial work. However, these types of indicators should be matched with other, more outcome-related indicators related also to the contents and the quality of the work performed.

- “Adaptation and mitigation strategies developed /Number of new adaptation projects

planned by members of the network”: Again, these types of indicators do provide some information on the ability of the institution to develop strategies/projects, but it would be interesting to complement these types of indicators with more implementation-related information. For instance, share of deviation between approved budgets and expenditures, or share of activities rated ‘delayed’ would provide good and complementary information.

If the project or program only includes these types of output-oriented indicators, it does not provide any information at all on outcome or impact level. They should be complemented by other indicators.

Taking the above challenges into consideration, some projects are managing this dilemma (between too vague indicators that are difficult to measure – and too narrow, output-focused indicators that only provide a piecemeal picture of the whole) by developing a very high number of indicators. There are examples where one program has identified, and thereby intends to monitor, as many as 50 indicators. There is a risk that either the indictors will not be measured as they are too many, or that a significant amount of capacity and resources will be put on monitoring and reporting.

To sum up: Based on this brief assessment, it appears that often the indicators are not SMART (Specific, Measurable, Accepted, Relevant or Time bound). The most difficult part appears to the ‘S’ (Specific) and the ‘M’ (Measurable).

3.2.2 Indicators

The Helpdesk proposes to divide the Category 2 indicators in the following groups:

1. Institutional capacity (administrative structures, institutions and systems) 2. Policy and legal framework (including development of strategies and action plans) 3. Implementation (and enforcement of legislation, policies, strategies, projects, etc.)

There is a high number and a wide variety of indicators used for Category 2. Many of these indicators are relatively difficult to measure. They are process- rather than state indicators.

The many Category 2 indicators are collected, grouped (and in some cases re-phrased) and listed in Annex 2. These indicators are SMART, and all have benefits and could be utilised as sources of inspiration for different interventions, depending on the objectives of the projects and programmes. However, most of them would benefit from being more specific, reflecting the particularities and objectives of the intervention.

As can be seen from the indicators listed in Annex 2, there are few indicators in this category that provides information specifically related to CCA, adaptive capacity, vulnerability or

13

resilience. This is not surprising as the category deals with institutional capacity or the policy and legal framework.

3.3 Indicators for Category 3 - Education, training and awareness on climate change Eleven projects under the climate change initiative are included under category 1:

- Mali – Gedefor - Mali – ITP CC West Africa - Mali – ITP Forest - Bolivia – Environmental Network LIDEMA - Bolivia – Proagro II - Cambodia, UNDP SGP Community Based Adaptation Programme - Cambodia, Forum Syd - Regional Asia, IUCN MFF - Regional Asia, EEPSEA - Regional Asia, WA proposal - Regional Africa, ECOWAS - Regional Africa, IUCN PREMI - Regional Africa – CC UNEP Marine


In several cases the LFA matrices are a bit difficult to understand. There are often clear objectives but seldom stated activities, indicators or targets. Indicators, activities and targets are often mixed up. For example the stated indicators are often in fact targets and not measurable indicators. As in the cases of the other categories the authors have created suggestions for indicators where indicators are lacking. Indicators or targets are frequently stated in long sentences depicting goal formulations rather than measurable and objective indicators, which make it more difficult to understand what the intentions are and how the targets and indicators are linked to the objectives.

There are many LFAs that include few indicators or objectives within category 3. For some, the LFA do not include objectives and indicators within category 3 at all. Sometimes it seems as if the text in the project document differs somewhat from the objectives and indicators in the matrices.

Education, training or awareness are seldom included as the overall objective but are often the means and activities to reach the overall objective.


As mentioned above, category 3 indicators are rather activities or targets or means of verification. The activities/targets are often stated with long sentences and are not possible to measure. Examples of indicators that are rather activities or targets or means of verification than indicators include:

- Workshop reports: This statement is supposed to measure supporting environmentally sustainable livelihoods among coastal communities. One of the activities stated is workshops. However the indicator is only stated as workshop reports. This statement is impossible to measure and is rather a mean to be able to find results information. The statement could be replaced by an indicator measuring for example: Number of

14

workshops regarding environmentally sustainable livelihoods among coastal communities held.

- Enhanced capabilities: 70% of the 260 specialists from partner institutions and 300 users and watershed coordinators who have been trained in MIC and the multiple risks associated with climate change, confirm the improvement of their skills and apply them in the context of their work. This statement is a target rather than an indicator. To become measureable this long sentence should be divided into several different areas e,g, number of specialists and users trained, number of specialists and users confirm improvement of their skills etc.

More quantitative than qualitative indicators:

Most indicators within category 3 are quantitative indicators that measures number of workshops/training events/participants etc. These indicators are specific, measurable, accepted and time bound but not always relevant. The indicators do not say anything about the content of the training course, or participants’ level of satisfaction. Quantity is often easier to measure but sometimes it is also relevant to include one or two indicators that measures quality by including for example what kind of information that is included in a training event or level of satisfaction or share of participants that use attained knowledge in their current work situation etc.

3.3.2 The indicators

The Category 3 indicators intent to measure events of education and training to disseminate information regarding climate change risks, improving adaptive capacity, and raise awareness to change behaviour, or increasing disaster preparedness. These aspects are more often a part of a project and not the main activities or objectives. Hence, the indicators related to category 3 more likely to be found at the output or outcome levels, not impact level.

The analysis of contributions with category 3 as primary category demonstrates the possibility to categorise indicators in two groups, namely: i) Education and training, ii) Awareness

Group 1: Education and training: There are relatively few indicators that fall within education and training. Most are very similar and regards “number of training events”; “number of persons trained”. The indicators within education and training often include information regarding the persons taking the course but none of the indicators include any information regarding the course/training event. As stated above there are no indicators that measures quality of the training event. The indicators stated in the LFAs are mostly about quantity; how many people that attended a certain training course or education event or how many courses that have been conducted. Examples of education and training indicators are presented in Annex 3

Group 2: Awareness As for education and training there are relatively few indicators within awareness as well. Most awareness indicators regard awareness raising events e.g. workshops. The indicators are mostly focused on measuring the quantity of events and number persons attaining the events. Examples of awareness indicators are presented in Annex 3.

3.4 Indicators for Category 4 - Climate studies, scenarios and impact research Eleven projects under the climate change initiative are included under category 4:

- Regional Africa, UNEP CC adapt

15

- Regional Africa, UNEP Marine - Regional Africa, Bioinnovate - Regional Asia, COBSEA - Regional Africa, CC RFGI - Regional Asia, EEPSEA - Regional Africa, NEPAD - Regional Asia, ICIMOD A - Regional Asia, ICIMOD B - Regional Africa, AFDB ClimDev

3.4.1 The results matrix

Several of the LFA matrices are difficult to understand. For example in one matrix there are no clear indicators and there are no clear objectives on output level stated. Instead all information on output-level seems to be activities. Indicators, activities and/or targets are often long sentences and this makes it more difficult to see the links to the objectives and indicators becomes more difficult to measure.


Several indicators are stated as means of verification. Examples of indicators that are rather means of verification include:

- Landscape and national level studies: This statement is not possible to measure. It is vague and not clear how many landscape and national studies should be carried out. The helpdesk has made the following suggestion for an indicator - Landscape and national level studies carried out (yes/no year) (study).

- Comprehensive impact assessment: This is difficult to measure, not only because it is not stated as an indicator but also because the choice of the word comprehensive. What is the definition of a comprehensive impact assessment? Based on the information given the helpdesk has provided the following suggestion: comprehensive impact assessment conducted (yes/no: year).

3.4 2 The indicators

As stated above the indicators in category 4 are primarily process-oriented indicators. They are not directly related to adaptation, but rather a measure of how to proceed in order to reach adaptation activities, such as baseline studies, etc. Hence, the indicators express the capacity and tools to prepare for and to deal with climate adaptation.

From analysis indicators within category 4 are categorised into two groups: i) Methods and tools and ii) Study and research.

Group 1: Methods and tools There are more indicators measuring tools than methods. Mostly the indicators measures of the tools have been conducted or are in use. Often the indicator and objective only include the word “tool” which makes it relatively difficult to understand exactly how comprehensive the activity is. A tool can be anything from a small measuring device to a large database. Some indicators could be more appropriate and measure the “right” thing. For example it is not as valuable to measure if a tool is tested or not. It is more valuable to see whether it fulfil its purpose or not. Examples of methods and tools indicators are presented in Annex 4.

16

Group 2: Study and research The indicators stated within study and research mostly include if a report/baseline/analysis or similar has been conducted, number of report/baselines/analysis or similar developed, or number of peer reviewed articles. Indicators are often stated as targets or as activities e.g. at least 1 study conducted. The Helpdesk has in these cases changed the indicator to number of study conducted. Examples of study and research indicators are presented in Annex 4.

3.5 Indicators for Category 5 - Coordination on climate change measures and activities across relevant actors Five projects are included under Category 5:

- Bolivia, Sumaj Huasi - Regional Africa – CC IUCN PREMI - Regional Asia, Regional Climate Change Adaptation Knowledge Platform - Regional Asia, RECOFTC - Regional Asia – SEAFDEC


Category 5 programs are often quite complex and the results matrices difficult to understand and follow. It appears to be difficult to phrase objectives and indicators for Category 5, which includes networking, coordination, participation, sharing and utilising lessons learned in order to improve the community of practice. The area is quite abstract in itself and is often relating to not just numbers and figures (quantitative measures) but very much to qualitative aspects (quality of the dialogue). This does not facilitate the ability to find suitable indicators.

Indeed, the formulation of the indicators suggests that the project implementer often is aware of the importance of the qualitative aspects. The indicators are very seldom measurable indicators, but rather long sentences describing a desired state (i.e. goals), activity, or in other ways formulated to include the complexity of the Category. However, there are few measurable indicators. Quite commonly, the statements replacing the indicators are so long and complicated that they are difficult to understand. Some of them are likely possible to respond to only after careful assessments, interviews and qualifications.

In a few cases, no indicators are included at all; only goals and activities are stated. In this case, the activities include information on the process (inform the member states, provide recommendations, etc.) and not to contents (that it relates to sustainable fishery and livelihood improvements). In one case, the indicators are replaced by “means of verification” such as minutes of meetings or list of participants.

Appropriateness of objectives and indicators The Category 5 indicators intent to measure aspects related to ‘capacity of stakeholders’, ‘level of coordination/participation/knowledge’ or ‘knowledge sharing’. Although important ends in themselves, these aspects are often means to other ends, such as enhanced integration of CC in development processes, improved adaptation capacity, or livelihood improvements. Often, therefore, the indicators related to Category 5 are found at the output or outcome levels, not impact level.

Indicators are not really indicators

As mentioned above, the Category 5 areas in themselves are quite vague and therefore difficult to measure. Often the indicators are not really indicators but long sentences including different relationships and desired outcomes. Examples of indicators that are rather goals (or

17

targets) than indicators and thereby too complex, containing too much information and hard to measure include:

- “The different levels of stakeholders are able to use the knowledge and products provided by the Adaptation Knowledge Platform to change and improve their planning and decision-making”: This statement wants to measure if the planning and decision-making processes are improved due to the availability of knowledge in the platform. The information is interesting and is likely to be answered in a general way in narrative reports, but it will likely be difficult to say much about how the platform contributed to improved planning. The indicator could well be divided into several different areas, for instance: is the planning and decision-making improving? Thereafter the implementer could reason about the contribution from the platform.

- “By 2013, national diagnostic reviews /…/ are completed and shared within and among countries so that they can be replicated”: This target could be replaced by indicators measuring, for example: (i) No. of national diagnostic reviews performed; (ii) % of diagnostic reviews are shared with the government (or other policy-area owner); (iii) No. of new projects/programmes where the knowledge is utilised.

- “Increased participation/trends/knowledge”: These types of statement would be

classified as (output) goals rather than indicators. It would be valuable to describe in what way participation will be increased, how the results of the participation will be included and used, etc. Importantly, in order to see a change, a baseline is required.

Indicators are too vague

There are other examples where the indicators are just too non-specific to be measurable. Some examples include:

- “The level of implication of members and partners in the economic development strategic processes”: This indicator is likely referring to how stakeholders are able to influence strategic processes. It is important, but so unspecified that it is hard to measure. It would be good to elaborate on what “level of implication” means and how it will be measured.

- “Level of implementation of improved regional environmental policies”: This is important to monitor, but the indicator is so unspecified that it is hard to measure. Either it should be clear how ‘implementation’ will be assessed and measured, or the indicator could be divided: (i) policies are reviewed and revised to include aspects of sustainable natural resource management; (ii) revised policies are adopted, etc.

- “No. of decision makers sensitised”: The indicator attempts to measure not only the number of contacts and dialogue opportunities but also provide a more qualitative aspect to it. But, again; the indicator is vague and needs qualification: what is meant by sensitised? What type of evidence is needed for someone being sensitised? Would it be possible to measure a changed agenda, a new position, or changes in policies or legislation?

- “Analytical outcomes inform decision-makers about the social impacts and potential mitigation measures”: Again, too much information in the indicator which makes it confusing: what will be measured? And, in what way are the analytic outcomes informing decision makers? Are there any targeted efforts or communication strategy developed? How is the information shared? Is it enough to inform decision makers, or should there be

18

any type of reaction? It would be good to be more specific as to how the information is presented: broadly, as website information or in policy briefs, or will there be pro-active, targeted communication (meetings, face-to-face communication).

Indicators not measuring the goal attainment

The indicator is not always measuring the goal attainment but something else. Examples include:

- “Increased trend of exchange of information, knowledge at the national level/ Increased trend of E-newsletter accessed and used by community partners”: These indicators (which really are more of output goals than indicators) does not provide information related to the goal ‘generation of new knowledge’. Rather, it provides information to the goal ‘communication of new knowledge’. It may be more appropriate to either change the goal, or use an indicator which is related to research, or knowledge generation. If the latter is chosen, valuable information would be related to how the knowledge will be generated – through field research, monitoring and evaluating of adaptation projects or other?

3.5.2 Indicators

It is possible to group the Category 5 indicators into the following:

1. Co-ordination and integration (creation of linkages between institutions and coordinating activities)

2. Participation (stakeholder participation in dialogue, decision making, formulation and design of programmes, etc.)

3. Research-to-policy (use of research or knowledge for dissemination to influence policy- or decision makers)

4. Community of practice (sharing of lessons learned and good practices, strengthening community of practice)

There are not as many Category 5 indicators, and a majority of them relate to group 4. The indicators for each subgroup are listed in Annex 5. The indicators provide examples and could be utilised by other actors in different interventions, depending on the objectives of the program/projects.

The indicators often communicate that the intentions are to have an impact on policy making, project design, poor people’s voice, etc. The coordination, participation, and communication could relate to any area, not only to improving resilience or climate change adaptation capacities.

It would be useful if the indicators would be a bit more ‘fit for purpose’, more specified and easier to measure than the often very broad statements found in this category. Indicators will, almost by definition, only provide small pieces of information related to attainment of the goals. They cannot and should not measure everything. It is better to have a narrow indicator that is possible to measure, than a broad statement that will require a narrative report to be responded to.

Furthermore, indicators should be specific and to the point, and linked to the wanted results. For instance, statements such as “sharing information” or “dissemination” could be more specified. Just disseminating information broadly, for instance posting project evaluations on the website, may not have any impact at all. It could be worthwhile to be more strategic, perhaps defining a target group, elaborating a strategy for how the information will be

19

presented, and striving towards a two-way, perhaps even face-to-face communication. If this is the case, this should be specified.

What is the purpose of the coordination or participation, and would that purpose be possible to measure? A good example is taken from RECOFTC that at actually do attempt to include a category 5-indicator at impact level: “No of countries that integrates the well-being of local people and environmental sustainability in their national forest programs”. Even if also this indicator will be difficult to measure, and would benefit from being more precise, they do try to measure the impact.

As a complement to one impact level indicator, or attempts to measure the purpose of the coordination, participation, etc., simple output indicators may be useful. The broader issues could be saved for the narrative reporting. As stated previously; indicators should be used with care and they must reflect the objectives, or the wanted results.

4. Summary and conclusion

4.1 Results matrices In general there are opportunities to improve the quality of the results matrices. Commonly they are not well developed, which sometimes make them difficult to understand and follow. Especially when the programmes are large and complicated, including different projects or components, the programmes are difficult to comprehend just by looking at the results matrices. The format of the results matrices/log frames could be developed to be easier to understand. The results matrices that are based on the RBM approach (with impact, outcome and output objective) are easier to understand than those based on the LFA method.

In some cases the results matrices are too short and do not seem to grasp the contents and results stated in the programme document, while, in other cases, the results matrices are too long with too many stated objectives, activities, targets and indicators. Furthermore, in some cases there is not one matrix but several different; each containing information related to different things, e.g. one matrix related to objectives, one for activities or one for indicators. In these cases, the different matrices are not always easy to match together (for instance, which activity and which indicator belongs to which objective). Moreover, in some cases there is no results matrix available at all, but the indicators must be found in the text.

Appropriateness of the objectives and the indicators In general, most objectives appear to be relevant and well in line with the needs in the country or region, and relate directly or indirectly to climate change, resilience, vulnerability, adaptive capacity, etc. The specific (or outcome) objectives often link to the development objective, so the logical chain is adhered to in most cases.

Indicators are seldom stated at an impact level; if they are, they are sometimes complex, vague and unspecified, and therefore quite difficult to measure. Also outcome-level indicators are commonly vague and unspecified, and difficult to measure. The output indicators are often more specific and measurable, although not providing relevant information.

From the analysis (Section 3) it can be concluded that formulating SMART indicators is not an easy task. The indicators often attempt to measure the stated objectives, or at least most parts of them but, although formulating the indicators comes with various challenges, including:

- Indicators are too vague and unspecified - Indicators are not really indicators but rather targets, activities, milestones or goals

20

- Indicators are not measuring the goal attainment - Indicators are too output oriented - The number of indicators is too large or too small

For all categories there are good examples of indicators that provide enough information on results achievement without being too vague. There are also examples of very well elaborated indicators. For instance, the Cambodia Climate Change Alliance shows a well-elaborated results matrix with various category 2 indicators, and even if they have identified too many indicators they cover a wide range of information that is important.

Based on this analysis, it appears that often, the indicators are not SMART (Specific, Measurable, Accepted, Relevant and Time bound). The most difficult part to cover seems to be the “S” – Specific, and the “M” – Measurable. Furthermore, most indicators ought to more carefully consider the purpose of the activity.

It can thus be concluded that once the objectives are formulated (not before), the indicators can be identified. The indicators should measure the level of goal attainment, or a certain aspect of goal attainment, it should be objectively verifiable, defined and specific enough to enable a comparison over time. Last but not least, there should be an adequate number (i.e. neither too many nor too few).

4.2 Indicators The analysis of all contributions demonstrates that for all categories, it is possible to divide the indicators in sub groups. The Helpdesk proposes the following subgroups for each category:

Category 1: i) Ecological indicators, ii) Livelihood indicators, iii) Technical indicators

Category 2: i) Institutional capacity, ii) Policy and legal framework, iii) Implementation

Category 3: i) Education and training, ii) Awareness

Category 4: i) Methods and tools, ii) Study and research

Category 5: i) Coordination, ii) Participation, iii) Research-to-policy, iv) Community of practice

As mentioned above, often the identified indicators are de facto not indicators but should rather be classified as goals, activities, targets or milestones. In some cases, only objectives and expected results are stated, not measurable indicators. In these cases, the helpdesk has provided suggestions on how to formulate indicators related to the objectives/ activities/targets/milestones (listed in Annex 1-5).

Not surprising, most indicators that provides information specifically related to climate change adaptation, adaptive capacity, vulnerability and resilience are found within category 1. These indicators are mainly state indicators, describing the current state (e.g. of the environment) /situation. The indicators are usually more practical and have a relatively strong direct link to climate change adaptation compared to the other four categories.

Indicators within categories 2 and 5 are commonly process indicators, often abstract, vague and qualitative with sweeping statements such as “enhanced awareness” and “level of integration”. In comparison category 3 and 4 are almost the opposite with almost nothing but quantitative indicators with few indicators related to quality.

Examples of indicators that are utilised in the Climate Change Initiative are listed in Annexes 1-5, (slightly modified by the authors). More or less all indicators are specific and measurable and possible to compare over time (timebound). That is S, M and T in SMART. However, the

21

A and the R (appropriate and relevant) is only possible to assess in relation to the objectives. To develop SMART indicators, the objectives should be stated.

Measuring climate change adaptation capacity, vulnerability and resilience Indicators related specifically and directly to climate change adaptation ought to measure the ability of individuals, households, communities, infrastructure, etc. to cope with climate variations (e.g. floods, droughts), extreme weather events (e.g. storms), and climate change (e.g. changes in precipitation or raising sea levels). This ability is difficult to measure, why indicators instead attempt to measure the resilience of the ecosystems (e.g. related to forest cover or land degradation); or the socio-economic resilience of households to external shocks (e.g. income, agricultural yields, health, etc.). There are also some indicators at technical level that include aspects related to the climate change resilience of infrastructure. In these projects and programmes there are few indicators measuring vulnerability, such as “number of people affected by natural/climate related disasters/events”, or “number of people living in vulnerable areas” or similar. As mentioned above, most indicators directly linked to climate change adaptation belong to category 1.

The indicators reviewed for the categories 2-5 are often less directly linked to climate change adaptation or resilience. Example areas include: are administrative structures developed to manage climate change impacts; are the policies including aspects of climate variability and change; are the institutions prepared and able to manage climate-related issues; are people aware of climate change impacts; are there capacity to develop scenarios, etc. The indicators are, in these cases, often quite general as they relate to awareness, level of education, institutional structures, intra-institutional coordination, participation, the impacts of participation on decisions, ability to manage public funds, etc. In other words, these indicators could measure capacity for climate change adaptation, or other types of capacities, depending on how they are formulated. The great challenge lies with the ability to be exact and precise on what should be measured in order to monitor goal achievement.

4.3 Conclusion In many cases there are too many and/or too vague indicators. It is important that the choice of indicators depends on the objectives of a specific intervention. Furthermore, in general, it is beneficial to utilise a few but well balanced indicators measuring output, outcome and impact-level objectives. At the output-level it is good if the indicators are not only quantitative indicators, milestones or check-lists but that they could include an element of quality and vice-versa. An example from Category 2: we should measure not only that a policy has been adopted, but include qualitative aspects such as: what is the contents of the revised policy; is the quality adequate, has the process been inclusive, etc. A mix of indicators is good in order to get a broad sense of the level of goal attainment.

Complement a few output-indicators with outcome-indicators, for instance related to implementation (e.g. resource allocation or enforcement). Although the outcome-level is very important to include, it is often difficult to find suitable indicators and measure goal attainment. Check that the logical chain between activities, output and outcomes are be clear.

We believe that it is also good to include at least one impact indicator, perhaps a state-indicator. The impact indicator will help the implementers to keep their minds on the goal, and not only the process.

For instance if a project deals with water resource management, the institutional arrangements, legal or policy changes, mainstreaming activities and stakeholder participation

22

are aspects that could be measured. This could be complemented with information related to the quality/quantity of the water resource. Another example: a project intends to support livelihood improvements in a vulnerable community. At least one indicator should try to measure some kind of livelihood aspect, such as income, diversification, health, food security, etc.

The indicators must reflect the objectives, or the wanted results. If the objective is to have many new policies in place, the indicator ‘number of policies/strategies’ would be appropriate. But if the objective rather is one good policy in place and implemented, it is probably better to check if it has been approved/endorsed; if resources are allocated from the budget; if compliance is being monitored, etc.

In short: utilise a few but well balanced indicators, measuring achievement of goals at output, outcome and impact levels. Utilise both process and state indicators and think broad (not only output, administrative but quality and implementation). Be as specific as possible as vagueness obstructs measurability. Make sure that the indicators are de facto indicators and not goals, targets, or general statements.

5. Issues for Sida to consider

The analysis of results matrices within climate change initiative gives an indication both of difficulties to develop a results matrix and difficulties to interpret and understand the contents of the matrix. It is important that results matrices objectives are linked between all levels, impact, outcome and output. Chosen indicators are entirely dependent on the objectives. Hence, the formulation of objectives is crucial for monitoring of results. Some of the results matrices analysed in this study are not in line with the information in the project document. This is confusing and indicates of difficulties to include all aspects of a project in a results matrix. It is clear that the results framework needs to be improved. There is a need for project partners or the developer of the results matrix to understand the purpose of the objectives and indicators, and the differences between goals, activities, targets and indicators. Objectives, whether overall or sector specific, should be measureable so that suitable indicators can be identified and results measured. The person developing the results matrix need to start with the end in mind: What are the desired results and how will progress towards achieving them be measured? What indicators could be appropriate? Will the monitoring of these indicators provide information on results and the level of goal attainment? Development of measurable objectives (relevant for overall objectives as well as for sector objectives) facilitates the process of identifying adequate indicators. In general it is beneficial to utilise a few well balanced indicators rather than too many. Indicators are often not indicators but rather targets or activities. Many indicators need to be more precise. If indicators are too vague and qualitative, accountability may be negatively affected. It is important to carefully review each indicator and assess if it really measures the stated objective and if it is measureable (e.g. not an activity or target or too vague). Questions that might be useful to think of and try to answer when developing or reviewing a results matrix include: - Are the objectives and identified indicators in line with the information stated in the

project/program document?

23

- Are there objectives for each level (impact, outcome and output) and indicators linked to each level? Is there a logical chain between the output-, outcome- and impact levels?

- Is there a balance between process indicators and state indicators, between quantitative and qualitative indicators?

- Are the indicators really indicators (e.g. “number of studies conducted until 2015”) or are they stated as targets (e.g. “at least one study conducted by 2015”) or activities (e.g. “conduct a study”)

- Are the indicators SMART (specific, measurable, appropriate, relevant and time bound)? - Will it be possible to measure the indicators? How will the indicators be measured, and by

whom? - Will the indicators provide some kind of information that is relevant to the objectives and

the attainment of the goals? - How will the goal attainment be reported? What information should be included? All

aspects do not need to be measured by indicators; the information obtained from the indicators should be complemented with an analysis and a narrative reporting.

24

References:

Lamhauge Nicolina, Elisa Lanzi, and Shardul Agrawala, 2011. “Monitoring and Evaluation for Adaptation: lessons from development cooperation agencies”. Environment Working Paper No. 38, OECD, Environment Directorate

Sida, 2009. “Guidance Note – Climate change Initiative 2009-2011”. Sida Memo 2009-06-30.

Projects documents (or similar) for 44 Sida-funded contributions

Bangladesh (2)

51060017 Bangladesh Climate Change Resilience Fund - Multi Donor Trust Fund

51060020 Comprehensive Disaster Management Programme Phase 2

Bolivia (8) 51070018 Community forestry program - Baba Carapa 51070019 National Forestry Action Plan

51070022 LIDEMA

51070023 PROAGRO

51070038 PROAGRO Phase 2 51070039 PASAP Periurban water and sanitation 51070041 Sumaj Huasi, Periurban water and sanitation 51070042 Agua Tuya, Periurban water and sanitation Burkina Faso (2) 51130034 Water reservoirs, MAHRH Ministry of Agriculture 51130030 Support to IUCN CCI Cambodia (4) 51100019 Cambodia Climate Change Alliance 51100026 Forum Syd Joint Climate Change Initiative

51100030 Small Grants Programme Cambodia Community Based Adaptation Programme

51100050 Local Governments and Climate Change

Mali (6) 51130018 CCI REDDIN/IUCN wetlands 51130029 CCI RESO Climat Mali 51130027 CCI GEDEFOR adaptation 51130053 ITP, West Africa Climate change 54050063 ITP Cert. Forest 51130067 Norwegian Church Aid

25

Reg Africa (REED) (8)

51000062 NEPAD Conservation Agriculture 73001014 Regional Climate Change Program

21300003 UNEP, adapting to CC induced water stress in the Nile River Basin

51050048 UNEP marine 51050025 Clim-BioInnovate 51050044 LVRLAC Reg Africa (REPS) (5)

51030015 ClimDev Africa – African Climate Policy Centre and ClimDev Trust Fund 29000388 UNECA Klimat ACPC JFA (same as ClimDev above) 51000063 RFGI (Regional Forest Governance Initiative) 51050019 IUCN-PREMI 51000035 ECOWAS Strategic Programme for CCA Reg Asia (14) 51020035 MRC 2009-2010 51000071 ICIMOD 51000058 SEAFDEC 51000061 COBSEA UNEP 73000971 IUCN - MFF 73001007 Wetlands Alliance

51020037 RECOFTC (Regional Community Forestry Training Centre for Asia and the Pacific)

51020036 EEPSEA 49300035 Klimatplattform/UNEP/SEI/SENSA

(Also contrubution no: 4930057 SEI, 4930058 UNEP) 51020096 Core Env Program and Biodiversity Conservation Corridors initiative

51020060 Strengthening Governance of climate change finance to benefit the poor and vulnerable – A multidisciplinary programme in Asia and the Pacific to support a country led response to climate change

26

Annex 1: Category 1 indicators

Group 1: Ecological indicators:

- Deforestation rate (% and trend) - Share of land degradation (% and trend) - Changes in threat status of species (% and trend) - Number of hectares of coastal ecosystems restored or rehabilitated - Size of area of marine and coastal protection area (increased/decreased) - Area affected by bushfires (m2 per year) - Share of forest degradation (% of total in the geographic area) - Forest biomass rate and trend (%) - No. of landscaped forests and pastures - Share grassland degradation (%) - Share of forest degradation (%) - Proportion of land covered by forests;

Group 2: Livelihood indicators: - No. of hectares allocated to the poorest and women - Amount of income and quality of life among project beneficiaries - No. of communities or regions and their populations with increased preparedness for CC

impact - No. of individuals with new resilient employment opportunities - No. of protected agricultural areas - Percentage of people with livelihoods reinforced - Share of rural population with access to credit (men and women; %) - Agricultural productivity (yield/year) - Variability in yields or income over a multi-year period - Share of population in targeted areas suffering from starvation (%) - Share of targeted families and their members that survive (%) - Share of targeted women and children that survive (%) - No. of adult starvation and large scale male migration reported in the project area during

the four months covered by the project - No. of child deaths due to malnutrition reported in the projected area - Share of targeted women reported having been able to deliver or breastfeed without

complication - Share of population from targeted area that have migrated due to lack of access to food

(%) - Rate of increase in income derived from activities - Share of population with improved livelihood (e.g. Variability in yields or income over a

multi-year period; or similar) - No of interest groups that have received support from project. (No per year, and trend) - No. of (identified) vulnerable communities that have increased adaptive capacity to cope

with climate change impacts - No. of people with knowledge to use and maintain systems - No. of families (number of people) with access to a sewer connection (at home) or

ecological toilet - % of how usage of sanitation facilities - Average income yield (%) for 30.000 agricultural producers in the program region

27

- Share of women using services (%) - Share of men using services (%) - Gender issues are included in management models and services (yes/no) - Cultural interests are included in management models and services to prevent conflict

(yes/no) Group 3: Technical indicators: - No. of dams built - No. of dams with an alert system installed - No. of local water committees established - No. of hectares of irrigated schemes constructed and rehabilitated - Functional maintenance system in place (yes/no) - Share of fishermen that use proper techniques of fishing and fish processing (%) - No. of hectares of coastal ecosystem under some type of sustainable management system - No. of disaster and risk reduction plans/ exercises with ecosystem based approaches

completed at community level. - Implementation of management plans in existing marine and coastal protection areas - No. of basic health facilities established in safe areas - No. of food security plans put in place - No. of safe dwelling built - Storm early warning systems in place (yes/no) - No. of embarked roads, canals, berm, and other engineering works to manage flood waters - No. of immediate response emergency relief operations successfully carried out - No. of adaptation projects realized by the members of the network, funded by Sida - No. of new adaptation initiatives implemented by the network members. - No. of homes with functioning water supply - A functioning tariff system for water in place and operating (yes/no) - No. of composting/recycling centres implemented and operating - No. of decentralized wastewater treatment plants implemented and operating - No. of agricultural producers applying models to improve their resilience (eg increase the

efficiency of water use, ensure the availability of on river water for different users, introduction of resistant crops and culturally accepted and - if possible - tradable, damping mechanisms for production risk and marketing)

- No. of institutions that provide services based in proven management models - Services offered include local needs and demand as well as traditional knowledge for

adaptation to climate risks (yes/no) - No. of irrigation project prepared by experts - Share of irrigation projects recorded in the system of quality control (%) (group3) - No. of small scale producers that have implemented management models that improve

access to water and efficient use of water - No. of Smallholder agricultural producers using services offered to increase resilience of

agricultural production.

28


The Category 2 indicators divided into the following three groups: (i) institutional capacity development; (ii) policies and legal framework; and (iii) implementation. As there are many and a large variety of Category 2-indicators, subgroups are also identified. Below, examples are presented of the indicators used for each group and subgroup under Category 2.

Group 1: institutional capacity

• Administrative (institution/secretariat/ inter-ministerial teams, etc. fully operational) o Staff appointed (yes/no; according to plan) o Office equipped (yes/no; year) o Meetings held regularly (yes/no) o Operational plan developed (yes/no; year) o Monitoring and reporting routines established/requirements fulfilled (yes/no; year) o Responsibilities for monitoring and reporting at regional, national, and local levels

are clearly defined (yes/no; year) o Financing arrangements in place/Continued financing (yes/no) o Budget allocations for gender responsiveness (yes/no) o Annual audits performed on time (yes/no) o Administrative and technical responsibilities (e.g. forest management)

decentralised • Negotiation

o No. of official negotiators with more than 3 years’ experience of CC o No. of inter-ministerial technical team involved in at least one key international

UNFCCC meeting o No. of regular CC negotiation related events held nationally

• Project management o No. of projects supported that are in line with national development priorities o Share of deviation between approved budget and expenditure (%) o No. of fiduciary issues raised, with resulted in disciplinary action or compensation o Grant proposal appraisal mechanism established and functional (yes/no; year)

• Capacity o Capacity gaps identified (in ministry/parliament/organisations, etc.) (yes/no; year) o Share of capacity gaps addressed (%) o No. of government staff trained on CC/tools/impacts/adaptation measures o Structure/governance mechanism for coordinating CC in place (vertical and/or

horizontal coordination) (yes/no; year) o Share of political decision makers and planners who have participated in (state

what) declare they have improved their knowledge (%) o No. of countries with budgets, public financial management and M&E systems

that enables assessment of quantity and quality of climate expenditures o No. of times countries report on climate change related expenditures disaggregated

in relation to poverty and gender targets o No. of staff (women and men) able to implement safeguard assessment o No. of prepared investments proposals on integrated livelihood and ecosystem

management o No. of national, provincial or city based MRV systems developed o No. of sector projects in environment and natural resources development o No. of countries where national support unit (NSU) is fully operational

29

o Share of professionals that are from the region; Share of professionals that are women

• Mainstreaming o Policy/strategy for mainstreaming CC adopted/initiated (yes/no; year) o No. of ministries/government agencies with CC focal points/units participating in

CC mainstreaming o Basin-wide planning process integrated adaptation measures

Group 2: Policy and legal framework • National framework

o National strategy (for CCA, CCM, DDR, IWRM, etc.) is developed and endorsed (yes/no; year)

o Relevant guidelines developed and approved o Policies/laws that are incoherent or promotes maladaptation are identified o A plan to adjust coherency of national legislation/policies is developed (yes/no;

year) o No. of countries that have established new policies to effectively govern climate

change finance o Presence of climate change policy within government medium and annual fiscal

policy formulation (yes/no) • Mainstreaming

o No. of ministries that incorporate CCA/DDR in policy and planning (difficult to measure)

o No. of national or sectors policies review and revised to consider CC /DRR o Hyogo Framework for Action integrated into legislation (yes/no; year) o No. and types of improvements in the (water-related, coastal, etc.) legislation o Explicit strategies for CCA included in sub-national development plans o Sub-national planning guidelines adapted to include formulation of cross-sectoral

CCA strategies o No. of sector or corridor strategies and plans that include environmental and social

considerations • Local framework

o Regional/local adaptation and mitigation strategies developed o No. of regional/national development plans that reflect livelihood needs of

communities (specify livelihood needs, CCA, DRR, etc.) o CC impacts integrated in community CCA strategies (yes/no; no. of districts)

• Transboundary framework o No. of river-basin plans include priorities of local stakeholders o No. of river-basin plans based on development and CC scenarios/ecosystem

approach o Monitoring plans are adopted by relevant stakeholders

Group 3: Implementation • Resource allocation

o Level of government budgets allocated for CCA/DRR o Ministries that incorporate CCA/DDR in budgets (number and share of total) o No. of government staff working with CCA (gender implications of CCA, etc.) o National maintenance plan and staff and budget to carry out repairs after climatic

events in place o Share of priority activities/development projects that are funded (%)

30

o Share of policy, regulatory and legal gaps identified that are funded (%) o CC trust fund run within the framework of government systems o Financial and human resources allocated to local/district levels for areas with

decentralised responsibilities (yes/no; trend) o % of discretionary budgets in target districts allocated to projects responding to

CCA strategies o % of grants funding projects responding to priorities identified by women group in

VRA o Sub-national chart of accounts includes code for revenues earmarked for climate

change adaptation investments o Increase in percentage of national and international climate related finance

included in PFM systems • Monitoring

o Share of development projects from key ministries that meet the agreed national requirements for state area (e.g. CCA, DRR, etc.)

o Share of development projects/activities rated ‘delayed’ (%) o Local level deliveries of disaster management services are enhanced (yes/no;

trend) o No. of NAPA follow-up projects implemented o No. of national and sub-national government entities and CSOs completing at least

50% of activities identified related to CCA/DDR/Ecosystem resilience, etc. o Level of integration of ecosystem values into national development strategy/PRSP

processes o Action plans/projects implemented according to plan (yes/no, delays in following

areas) o Internal implementation mechanism for REDD projects designed and implemented

(yes/no; when) o No. and type of poverty programmes that include specific measures to target,

deliver and monitor and evaluate benefits to climate vulnerable communities o No. of community groups (with at least 35% women beneficiaries) that has

adopted climate change coping strategies o No. of biodiversity conservation corridors where management and operational

plans have been formulated and implemented • Enforcement

o Law compliance monitored; o No. of compliance monitoring activities per year (and trend) o No. of non-compliant activities reported (and trend)

• Incentivising o Policy instruments/incentives are identified to promote specify (e.g.

CCA/ecosystem resilience/low carbon development, etc.) o Policy instruments are introduced/applied (yes/no; year) o Effects of policy instruments are monitored (yes/no; trend) o Policy instruments are effective (yes/no; trend) o Sector specific programs designed and implemented to reduce GHG emissions

• Impact o Share of low-income households with access to improved water and sanitation

facilities (% of people in low-income areas/settlements) o Perceived change in livelihood of vulnerable communities due to intervention

(household income/food security/economic diversification/health/coping strategies, etc.)

31

o Share of targeted population in vulnerable communities aware of climate change risks and appropriate adaptation measures (%)

o Framework for protection against hazards is relevant to the needs (based on interviews)

o Deforestation rate (% and trend) o Share of mangrove forests (or other ecosystems) in target areas restored and in

good health (%) o Proportion of total water resources used (%) o Share of known water resources checked for quality and quantity (%) o Effluent water (from WWTPs or industrial plants) within national pollution

standards (number or share of total; trend) o No. of beneficiaries (men/women) benefitting from projects responding to CCA

strategies

32


Group 1: Education and training:. - No. and type of stakeholder trained - No. of coastal managers trained in ecosystem restoration/rehabilitation measures - No. of training courses conducted Group 2: Awareness - No. of participants to national and regional workshops - No. of national and regional workshops - Share of specialists and users that confirm that they have improved their skills regarding

risks associated with climate change, and apply them in the context of their work

33


Group 1: Methods and tools - Tool launched globally, regionally & nationally (yes/no year) - Degree of user satisfaction with provided tools - Monitoring tools developed & in use Group 2: Study and research - Report/baseline/analysis/review/assessment (or similar) conducted (yes/no: year) - No. of reports/baselines/analysis/reviews/assessments (or similar) developed - Change in number of published peer-reviewed scientific articles. - Degree of user satisfaction with provided information as stated in conference. - Ensured quality monitoring of greenhouse gas, air quality, inland lakes

34


Group 1: coordination - Share of participants familiar with the regional environmental policies (%) - State organisation and its partners jointly identify new themes for analysis and support to

advance the development and implementation of state area (e.g. community forestry) - No. of coordination meetings or activities

Group 2: Participation - No. of countries that integrates the well-being of local people and environmental

sustainability in their national forest programs - No. of regional and national multi-stakeholder working groups that are operational - Consultations held to develop the activity plan (yes/no; when) - Stakeholders’ needs and views are integrated in and promoted by project design - Forest area under community management (% or m2; trend)

Group 3: Research-to-policy - No. of position papers published by each category of participants - % of participant countries who have adopted options recommended by regional policies - No. of policy statements received by decision makers - % of decision makers satisfied - No. of requests (from policy makers) for collaboration, advice or services - No. of consultations on recommendations held with member states representatives - No. of face-to-face communications with policy makers

Group 4: Community of practice - Work programmes of community-level organisations adapted based on information

derived from state initiative - Categories of barriers and opportunities to Integration Climate-Poverty identified (yes/no) - No. of regional community forestry policy forums held per year - Lessons learned throughout the region are documented (yes/no) - Lesson learned are disseminated through suitable communication tools, techniques, and

approaches (the tools, techniques and approaches should be specified) - No. of dialogue sessions with government officials to develop sustainable WASH

initiatives - No. of new projects developed in peri-urban areas that are based on lessons from pilot

projects

35

Annex 6: Analysis of objectives and indicators (separate file)

Documents

Climate change adaptation indicators - Sida's Helpdesk …sidaenvironmenthelpdesk.se/wordpress3/wp-content/uploads/2013/05/... · 1 Climate change adaptation indicators . A Logic