MINISTERE DES INFRASTRUCTURES - waapp-ppaao.org€¦ · Web view- World Bank : « Template and ... (iii) the tracking of the results indicators (results monitoring); (iv) Evaluation

WEST AFRICA AGRICULTURE PRODUCTIVITY PROGRAM(WAAPP)

MONITORING & EVALUATION OPERATIONS MANUALVOLUME 1 – Main document

(Final Version)

January 2014

WAAPP M&E MANUAL

TABLE OF CONTENTPage

1 AN OVERVIEW OF THE WAAPP..................................................................................................1

1.1 GENERAL CONTEXT..........................................................................................................................11.2 OBJECTIVE AND COMPONENTS OF THE WAAPP...............................................................................11.3 WHO ARE THE BENEFICIARIES AND WHAT ARE THE TARGET AREAS ?.............................................31.4 WAAPP IMPLEMENTATION ARRANGEMENTS..................................................................................3

2 INTRODUCTION TO SOME BASIC ELEMENTS OF M&E......................................................4

2.1 PURPOSE OF M&E............................................................................................................................42.2 MONITORING VERSUS EVALUATION.................................................................................................42.3 WHAT IS AN INDICATOR ?................................................................................................................52.4 THE RESULTS MEASUREMENT FRAMEWORK (RMF).......................................................................6

3 PURPOSE OF THIS M&E MANUAL..............................................................................................8

3.1 OBJECTIVES OF THIS MANUAL..........................................................................................................83.2 MAIN CONTENTS AND STRUCTURE OF THE M&E MANUAL.............................................................9

4 THE M&E STAKEHOLDERS ROLES AND INFORMATION NEEDS...................................11

4.1 INSTITUTIONS INVOLVED IN THE IMPLEMENTATION OF WAAPP PROJECT....................................114.1.1 At country level......................................................................................................................114.1.2 At Regional level....................................................................................................................11

4.2 THE MAIN ACTORS OF THE WAAPP M&EIS AND THEIR INFORMATION NEEDS..............................15

5 THE M&E DATA: INDICATORS AND THE RESULTS MEASUREMENT FRAMEWORK20

5.1 THE WAAPP RESULTS MEASUREMENT FRAMEWORK....................................................................205.2 WAAPP RESULTS INDICATORS......................................................................................................20

5.2.1 Indicators baseline values : the initial values of indicators..................................................205.2.2 Most important steps for data collection...............................................................................20

5.2.2.1 Data Sources......................................................................................................................205.2.2.2 Method of Data Collection.................................................................................................255.2.2.3 The Frequency of Data Collection.....................................................................................255.2.2.4 Responsible Entities for Data Collection...........................................................................25

6 WAAPP M&E PROCEDURES.......................................................................................................27

6.1 GENERAL.....................................................................................................................................276.2 OPERATIONAL PLANNING...............................................................................................................28

6.2.1 Preparation of the Project Implementation Plan (PIP).........................................................286.2.2 Preparation of the Annual Work Plans and Budgets (AWPB)..............................................28

6.2.2.1 Content of the AWPB........................................................................................................286.2.2.2 The AWPB preparation process.........................................................................................29

6.3 MONITORING OF THE IMPLEMENTATION OF THE AWPB (PROCESS MONITORING)........................306.3.1 Objective and scope...............................................................................................................30

6.4 RESULTS MONITORING...................................................................................................................326.4.1 Organizational set up for data collection on results.............................................................32

i

WAAPP M&E MANUAL

6.4.2 Source / Method of data collection........................................................................................326.4.3 Dashboard (tableau de bord) for results monitoring............................................................32

6.5 REPORTING ON THE PROJECT PROGRESS TOWARDS RESULTS.........................................................336.5.1 The Framework of reporting..................................................................................................336.5.2 M&E data processing............................................................................................................33

6.6 EVALUATION OF WAAPP PROJECT................................................................................................346.6.1 Overview................................................................................................................................346.6.2 Pre-Project or ex-ante evaluation.........................................................................................346.6.3 Mid-term Evaluation..............................................................................................................346.6.4 Final Evaluations...................................................................................................................34

7 THE TOOLS OF THE M&E SYSTEM..........................................................................................35

7.1 PRESENTATION OF THE TOOLS........................................................................................................357.2 THE WEB BASED M&E SOFTWARE.................................................................................................35

7.2.1 General characteristics..........................................................................................................357.2.1.1 System flexibility...............................................................................................................357.2.1.2 System scalability..............................................................................................................357.2.1.3 System security..................................................................................................................367.2.1.4 System simplicity...............................................................................................................36

7.2.2 The various modules to be provided......................................................................................36

8 DATA QUALITY ASSURANCE.....................................................................................................38

ii

WAAPP M&E MANUAL

LIST OF TABLESPage

Table 1 : Example of a Results Measurement Framework.............................................................6

Table 2 : Implementing entities for the WAAPP components and sub-components.....................12

Table 3 : WAAPP stakeholders roles and their information needs................................................16

Table 4 : Summary of M&E responsibilities...................................................................................19

Table 5 : indicators data sources and frequency of data collection as indicated in the PAD........22

Tableau 6 – Template for AWPB Implementation Monitoring Form (IMF) for the Present Period (Quarter or Semester)...................................................................................................................31

Table 7 – Template for accumulated values of AWPB Implementation Monitoring......................31

Table 8 : Template for Reporting & Data Quality Assessment - Regional M&E Unit level............40

iii

WAAPP M&E MANUAL

LIST OF FIGURESPage

Figure 1 : Links between the logic of intervention and the chain of results.....................................7

Figure 2 : M&E data sources.........................................................................................................21

Figure 3 : WAAPP Logic Model.....................................................................................................26

Figure 4 : The general architecture of the WAAPP M&E Information management System........52

iv

WAAPP M&E MANUAL

LIST OF ANNEXES(See M&E Operations Manual – Volume 2)

A. ANNEX 1 : RESULTS MEASUREMENT FRAMEWORKS FOR WAAPP

B. ANNEX 2 : RESULTS INDICATORS TRACKING TABLE

C. ANNEX 3 : RESULTS INDICATORS REFERENCE SHEETS

PDO INDICATORS INDICATORS FOR COMPONENT A INDICATORS FOR COMPONENT B INDICATORS FOR COMPONENT C INDICATORS FOR COMPONENT D

D. ANNEX 4 : RESULTS MONITORING FORMS (RM FORMS)

FORMS FOR ODP INDICATORS FORMS FOR COMPONENT A INDICATORS FORMS FOR COMPONENT B INDICATORS FORMS FOR COMPONENT C INDICATORS FORMS FOR COMPONENT D INDICATORS

E. ANNEX 5 : FORMAT REPORT

F. ANNEX 6 : AWPB FORMAT

E. ANNEX 5 : AWPB MONITORING

v

WAAPP M&E MANUAL

LIST OF ACRONYMS AND ABREVIATIONS

ARIs Agricultural Research InstitutionsAWPB Annual Work Plan and BudgetCAADP Comprehensive Africa Agricultural Development ProgramCARGS Competitive Agricultural Research Grant SchemesCAS Country Assistance StrategiesCAVA Cassava-Adding Value for AfricaCERAAS Centre d’Etude Régional pour l’Amélioration de l’Adaptation à la Sécheresse / Re-

gional Center for Drought Adaptation StudiesCGIAR Consultative Group on International Agricultural ResearchCORAF/WECARD Conseil Ouest et Centre Africain pour la Recherche et le Développement

Agricoles / West and Central African Council for Agricultural Research and Development

COSTAB Project’s Costs TableCSIR Council for Scientific and Industrial Research in GhanaCM&EO Country M&E OfficersEAs Environmental AssessmentsECOWAP Economic Community of West African Agricultural PolicyECOWAS Economic Community of West African StatesFNRAA Fonds National de Recherches Agricole et Agro-alimentaire in Senegal

FTMR Financial and Technical Monitoring Report IER Institut d’Économie Rurale in MaliISRA Institut Sénégalais de Recherche Agricole ISTA International Seed Testing AssociationIP National Implementing Partners

NCoS National Centers of Specialization

NPCU National Project Coordination Unit

NSC National WAAPP Steering Committee

PIM Program Implementation manual

PIP Program Implementation Plan

RM&EO Regional M&E Officers of WAAPP

RMF Result Measurement Framework

RITT Result Indicator Tracking Table

RPCU regional Program Coordination Unit

vi

WAAPP M&E MANUAL

I. MAIN CONCEPTS UTILIZED1

Planningg reflects the process of identification and implementation schedule for a series of actions to be implemented to achieve the objectives. Operations may cover the term of "components", "sub-components," "activities" or "sub-activities."

The project concept refers to a set of investment activities and operations planned in order to achieve an objective or set of objectives. This is a response mechanism chosen to achieve an objective or objectives identified during the planning process. The project includes a cycle that involves the identification, formulation, financing, implementation and monitoring and evaluation.

Monitoring and Evaluation System is defined as an integrated package that includes some aspects of the planning process, of systematic data collection, analysis, exploitation, synthesis, and dissemination of information, and provides the means and skills to implement the system to improve the basis for decision making in the management and implementation of a project and the capitalization of experiences and lessons learned.

Implementation Monitoring is a systematic and regular assessment of the resources used (input) based on forecasts on the one hand, the achievements (output) and the results of the project (results and intermediate results) in respect of targets and timing on the other. It is based on the development of Annual Work Plans and Budgets (AWPB). Its main deliverables are the technical reports, accounting and financial reporting, as well as activity progress reports.

The impact assessment component seeks to measure or understand the scope of the project actions on the well-being of beneficiaries, and the major changes and outcomes induced by these actions. It includes the establishment of the baseline, evaluations, surveys and periodic or specific indepth studies, as well as the final evaluations. The similarity of the pattern of the baseline survey and the project completion survey is critical to enable a comparison between the baseline described at the beginning of the project and the follow-up situation at the end of it.

Results based Management (RGM): Results-based management (RBM) is a life-cycle approach to management that integrates strategy, people, resources, processes, and measurements to improve decision making, transparency, and accountability. The approach focuses on achieving outcomes, implementing performance measurement, learning, and adapting, as well as reporting performance.

1 The definitions related to the concepts of planning and strategic planning refers primarily to the work of Michel Godet (1985): "Forecasting and Strategic Planning” » ; ECONOMICA, 330 pages including annexes.

2. Definitions related to the concept of Monitoring and Evaluation make reference to various documents: 3. - World Bank : « Template and Guidelines for the Project Appraisal Document (PAD) ; Guidelines for prepar -

ing the technical annex of the PAD ; Technical annex 3 : Results framework and monitoring », p.14-18., Washington, D.C.

4. – World bank, Department of Operations Evaluation : Project design. Monitoring and Evaluation, Vol.2, n°8. 5. - CIDA : CIDA Evaluation Guide, Performance Review Branch, 20006. - IFAD: Managing for Impact for Development, A guide for Project Monitoring and Evaluation, Rome, 2002.

Annexe A : Glossary of Monitoring and Evaluation Concepts and terms7. - IFAD, For impact oriented management : A practical M&E Guide for rural development projects; Rome, 2003

; 8 modules et 5 annexes.8. - PICCIOTTO R. : La coopération pour le développement et l’évaluation des performances. Le défi de Monter-

rey. Banque Mondiale, Washington, 2002.

vii

WAAPP M&E MANUAL

Logic model (LM): Sometimes also called a "results chain," a LM is a description of the causal or logical relationships between inputs, activities, outputs, and the outcomes of a given policy, program or investment.

Result: A result is a describable or measurable change that is derived from a cause-and-effect relationship

Outcome: is a result statement that outlines what a policy, program, or investment is expected to achieve or contribute to.

Output: Products and services generated through inputs and activities

Beneficiary: The set of individuals and/or organizations that experience the change of state at the ultimate outcome level of a LM although they could also be targeted in the immediate and intermediate outcome levels.

Stakeholder: An individual, group, institution, or government with an interest or concern, economic, societal, or environmental in the framework of the project.

Partner: The individuals and/or organizations that collaborate to achieve mutually agreed upon expected results.

viii

WAAPP M&E MANUAL

1 AN OVERVIEW OF THE WAAPP 1.1 GENERAL CONTEXT

Meeting the food needs of a rapidly growing population is one of the main challenges facing the countries of West Africa. The recent food, fuel and financial crises have demonstrated the need to focus food security in West Africa on the development of regional agricultural potential. This option would contribute both to economic growth and poverty reduction.

It is possible to achieve rapid growth in agriculture with emphasis on improving productivity. For example, the adoption of new varieties and improved farming practices would help increase millet yields by 30% (600 kg/ha to 780 kg/ha), the yield of irrigated rice by almost 50% (from 1.560 kg/ha to 1.950 kg/ha) and that of cassava yield of 40% (14 tons/ha to 20 tons/ha)2.

The West Africa Agricultural Productivity Program (WAAPP) has been designed as a two-phase program with a total duration of 10 years, to support the implementation of CAADP’s agricultural research and development (R&D) pillar.

As a technical body of ECOWAS in the area of Agricultural research and Development, the Coordination the WAAPP was assigned to the West and Central African Council for Agricultural Research and Development (CORAF/WECARD).

Since 2007, the ECOWAS countries with the support of the World Bank and others donors ( ), funded three series of projects, including the WAAPP-1A (for Mali, Senegal, and Ghana) in 2007, the WAAPP-1B (for Burkina Faso, Ivory Coast and Nigeria) in 2010, and WAAPP-1C (for Benin, Guinea, Liberia, Niger, Sierra Leone, Gambia and Togo) in 2011. Based on the results obtained by the WAAPP-1A, the World Bank and countries have decided to support the extension of the program in its second phase for Senegal, Mali and Ghana (WAAPP-2A).

Thus, WAAPP-2A will continue to provide to the ECOWAS countries, a regional cooperation framework for the implementation of national and regional agricultural strategies for development, dissemination and use of technologies. These interventions are expected to improve agricultural productivity and economic growth while reducing poverty in recipient countries.

1.2 OBJECTIVE AND COMPONENTS OF THE WAAPP As stated in the Project Appraisal Document of WAAPP 2A, the Development Objective (PDO) is “to scale-up the generation, dissemination and adoption of improved technologies in the participating countries’ priority agricultural commodity areas”. Reaching this objective implies:

(i) strengthening the regional cooperation in the technology generation and dissemination;

(ii) scaling-up dissemination and training programs focused on regional and national priority commodities;

(iii) developing collaborative mechanisms and synergies with the governments’ subsidy programs as well as with other projects and programs intervening along the targeted value chains.

2 The 1st series of projects under the 2nd phase of the WAAPP, Project Appraisal Document of the West Africa Agricultural Productivity Program (waapp-2a), February 17, 2012, World Bank.

1

WAAPP M&E MANUAL

As stated in the in the Results framework of the Project Appraisal Document, the achievement of the PDO can be verified by the following key outcome indicators:

1. Direct project beneficiaries of which 40 % female disaggregated by country

2. Beneficiaries who are using technologies generated by other countries - disaggregated by country

3. Technologies generated by the Project with at least 15percent productivity increase over the control – disaggregated by country

4. Producers with knowledge of technologies generated/released by the Project- disaggregated by country

5. Area under new technologies disaggregated by country

6. Processors/ producers who have adopted at least one new technology -disaggregated by country

The project activities are implemented through the following four components:

1. Component A: Enabling Conditions for Regional Cooperation in the Generation, Dis-semination and Adoption of Agricultural Technologies.

This component finances support to strengthening the mechanisms and procedures for the exchange of technologies, including: (i) Implementation of ECOWAS common regulations on genetic materials, pesticides, fertilizers and veterinary products; and (ii) Implementation of regional WAAPP strategies, which consist of mainstreaming regional strategies into national action plans.

2. Component B : National Centers of SpecializationThis component aims at strengthening regional cooperation in technology generation by aligning national priorities with the regional ones. It will support the upgrading of the National Centers of Specialization (NCoS) into Regional Centers of Excellence (RCoE). It will also strengthen relevant national research centers associated with NCoS in the development of the value chain and facilitate synergies with relevant Consultative Group on International Agricultural Research (CGIAR) and other international research centers. It will mainly finance construction and rehabilitation of core infrastructure, ISO certification of the NCoS, capacity building, grants to implement core research activities and capacity strengthening for CORAF/WECARD,

3. Component C: Support to Demand-driven Technology Generation, Dissemination and Adoption

This component finances the strengthening of priority-focused demand-driven agricultural R&D, scale-up technology dissemination and adoption within participating countries and facilitating access to improved genetic material. It includes three sub-components: (i) Demand-driven Technology Generation that will continue to strengthen the regional CARGS managed by CORAF/WECARD and the existing CARG scheme in each of the participating countries; (ii) Support to Accelerated Adoption of Released Technologies that will scale-up the adoption of released technologies to continue bridging the gap between farmers’ and researchers’ yields for selected priority commodities and (iii) facilitating Access to Improved Genetic Material that will increase the availability of and producers’ access to improved genetic materials (seed, planting materials, and animal breeds) for the targeted priority value chains.

4. Component D Project Coordination, Management and Monitoring and Evaluation. The Project will continue to be coordinated: (i) at the national levels by existing national coordinating units; and (ii) at the regional level by CORAF on the basis of a well-defined mandate agreed by the Regional Steering Committee (RSC) under the Annual Work Plan and Budget.

2

WAAPP M&E MANUAL

1.3 WHO ARE THE BENEFICIARIES AND WHAT ARE THE TARGET AREAS ?The Project’s primary beneficiaries are the producers, processors and all other actors of the selected value chains. The beneficiaries include also the key participants in the generation and dissemination of technology, including researchers, extension staff, public and private extension services and advisory agencies, research institutions, universities, NGOs, and government agencies involved in value chains management.

1.4 WAAPP IMPLEMENTATION ARRANGEMENTS

WAAPP is a regional programme. CORAF/WECARD has been mandated by ECOWAS and countries to coordinate the WAAPP at regional level. Each country has a national steering committee and a project coordination unit (PCU). A Regional Steering Committee, chaired by ECOWAS, is also established and meets once yearly to review the AWPB implementation progress of the current year, to approve the AWPB of the next year and to provide guidance for the project.

At national level, the institutional arrangements in each country are made for the Ministry in charge of Agriculture or Research to be responsible for the oversight of the Project; with the coordination of the Project being entrusted to a “Project Coordination Unit (PCU)”. The PCUs are responsible for the internal monitoring of Project indicators and outcomes as defined in the Results Framework. Each coordination unit (M&E officer) produces semi-annual progress reports along with notes synthesizing information on the project activities based (among others) on the reports produced by implementation agencies. The NCoS in participating countries will be responsible for measuring the impact of technologies released and transferred under Component 2.

At regional level, project coordination will continue to be entrusted to CORAF/WECARD which will have the responsibilities of monitoring program outcomes and impact at the regional level through:

(i) Carrying out update studies on the region’s agricultural productivity;

(ii) producing an annual consolidated report, based on the countries’ reports and their specific studies, to be shared with all ECOWAS countries and stakeholders;

(iii) informing participating countries on a yearly basis on implementation progress and the use of funds transferred from country proceeds,;

(iv) maintaining the web-based databases on grant administration and results while developing and maintaining an agricultural research resource database in terms of publications and research skills available in the region.

In order to speed up dissemination of project results and ensure achievement of the PDO, CORAF/WECARD will build further synergies with some regional organizations notably, the African Forum for Agricultural Advisory Services (AFAAS), the Network of Farmers' and Agricultural Producers' Organizations of West Africa (ROPPA), the West Africa Seed Alliance (WASA), the relevant CGIAR centers (IITA, ICRISAT, AfricaRice), the International seed testing association (ISTA), the African Seed Trader Association (AFSTA), the “Association des femmes de l’Afrique de l’Ouest” (AFAO), the Association pour la promotion de l’élevage au Sahel et en Savane (APESS), etc.

3

WAAPP M&E MANUAL

4

WAAPP M&E MANUAL

2 INTRODUCTION TO SOME BASIC ELEMENTS OF M&E2.1 PURPOSE OF M&EMonitoring and Evaluation (M&E) is essential for a results-based approach to program management. After being a key component of project design, M&E remains incorporated into all facets of the project cycle through its completion. The M&E system to be established will enable the WAAPP management team to clearly demonstrate to the key stakeholders whether the project is achieving the stated goal, outcomes and outputs in accordance with the targeted timeframe.

Therefore, the M&E systems should provide the means to robustly analyse the relevant data and information in order to:

monitor the project outcome indicators as reflected in the Results Measurement Framework;

comprehensively track the implementation of the Annual Work Plan and Budget (the project inputs, activities and outputs), using mainly the set of indicators (or milestones) outlined under each component and sub-component at frequent intervals,

Establish a process to alert implementers, stakeholders and WAAPP managers to any problems in project implementation and provide the basis for making the necessary adjustments.

Outline the flow of data and information from the project sites through the various stakeholders, both for the use of the general public and to inform decision-making.

2.2 MONITORING VERSUS EVALUATION

Even though monitoring and evaluation have complementary roles, it is important to distinguish one from the other.

Monitoring is the collection, analysis and systematic use of information / data to feed the project management and decision-making in order to achieve the project objectives. Through monitoring, we seek to establish to what extent the provision of inputs, work programs, and other necessary actions and expected outcomes are achieved / executed so that corrective action can be taken in time in case of 'significant deviations from expected quantitative and temporal targets.

In general, through monitoring one can determine whether the implementation of activities is going in the right direction at the right pace. It provides key stakeholders of a project with the information needed to:

(i) monitor the allocation and use of resources (physical, material, financial);(ii) assess the achievements (products, services, etc) obtained and measure the achieve-

ment of results with respect to the initial estimated targets;(iii) detect any deviations recorded during implementation;(iv) Facilitate decision making regarding the continuation of the policy, program or project

(technical shifts, reallocation of funds, and adjustment of program objectives).

Monitoring activities include, among others:

(i) the centralization of periodic reports prepared by stakeholders and reporting on their pro-gress;

(ii) carrying out field missions to verify the conditions/status of implementation of activities and immediately verifiable progress;

5

WAAPP M&E MANUAL

(iii) conducting specific surveys (which may also include discussions with all stakeholders of the project;

(iv) the regular updating of report for the managers/stakeholders;(v) the periodic summary reports for policy makers;(vi) specific rapid analysis fast, such as for example, SWOT and risks analysis an assessment

of strengths, weaknesses, opportunities and risks

Evaluation is a measure, as systematic and objective as possible, of the results of a project, program or policy, to determine its relevance and coherence, the efficiency of its implementation, its effectiveness and impact and the sustainability of the outcomes obtained in respect of the objectives. Thus, evaluation is to assess the efficiency, effectiveness, impact, relevance and sustainability of projects.

MONITORING EVALUATIONQuestion: Does the project implement its activities effectively?

Question : Does the project have the expected outcomes and impact ?

Regular / Continuous data collection Occasional data collection for im-pact assessment

Tracking of the implementation of AWPB (Process Monitoring)

Tracking of results indicators of the RMF (Results Monitoring)

In depth data analysis of socio-eco-nomic survey data

Measurement of effectiveness (Planned vs achieved)

Measurement of outcome and im-pact (Impact evaluation / assess-ment))

2.3 WHAT IS AN INDICATOR ?An indicator can be defined as "information used to account for a situation, a state at a given time." This is a factor or quantitative or qualitative variable, which is a simple and reliable way to measure or assess the changes associated with an intervention, or performance of a project, programme or organization. There are:

Impact indicators which relate to the consequences of the project beyond the immedi-ate outcome on its direct beneficiaries. There are two types of impact: on the one hand, the specific impacts that occur after a while but are still directly related to the action taken (Number of households benefitting from project services) and on the other hand, the general impacts that occur in the longer term and affect a larger population ( im-provement of living conditions of the population). The evaluation of this type of impact is complex and it is often difficult to establish a clear causal relationship and attribution. As indicated in the figure 1, the impacts are at a higher level than the outcomes.

Outcome indicators relate to the direct and immediate effect of the intervention. They provide information on changes, for example, on the behavior, the capacity or perform-ance of the direct beneficiaries. These indicators can be of direct (performance, etc..) or indirect nature (gross income of producers, etc.).

Output Indicators (or implementation indicators) relate to the activities. They are meas-ured in physical or monetary units (Technologies generated by NCoS; Number of demonstration plots established, etc.).

Input (or process) indicators relate to the activities and resource budget allocated to each level of intervention. The physical and financial indicators are used to monitor the progress of the implementation of activities in terms of commitments (quarterly, yearly)

6

WAAPP M&E MANUAL

and disbursements (quarterly, yearly) of the funds available for a project or program in relation to its cost.

In most cases, impact and outcome indicators require additional studies or surveys to measure them. Whereas, output and process indicators must be analyzed jointly to assess the "performance" in the management of the project and establish a dash board. The outputs will be analyzed in relation to the quantities forecast. Input Indicators are typically project resource monitoring indicators.

2.4 THE RESULTS MEASUREMENT FRAMEWORK (RMF)The Results Measurement Framework (RMF) is a structured plan to systematically collect and analyze relevant data on results indicators, over the lifetime of a Project/Program, to assess and demonstrate progress made in achieving expected results. The RMF is divided into columns: expected results/Activities, indicators, baseline data, targets, data sources, data collection methods, frequency, and responsibility for data collection. The “Expected Results“ column can be divided into several columns rows: outputs, immediate outcomes, intermediate outcomes, and ultimate outcome.

Table 1 : Example of a Results Measurement Framework.

Expected Resukts / Activities

(1)

Indicators) (2)

Baseline (3)

Target val-ues

Data sources / collection Methods

Frequency collection

Re-spons-

ible

Description

Goal / Development Objective (Impact)

Impact or PDO in-dicators

Specific Objectives(Outcomes)

Intermediate Res-ults indicators

Outputs /-Products Outputs indicat-ors

7

WAAPP M&E MANUAL

Figure 1 : Links between the logic of intervention and the chain of results.

There is often confusion between: indicators, outcomes and objectives:

Outcome: a statement that is clear and provides realistic description of the desired development change.Example: “400,000 yam producers in Ghana have adopted the new improved

varieties released by the project”.

Indicator: the unit of measurement that will be used to assess the result, without specifying the desired direction for the change .

Example: Number of yam producers in Ghana who have adopted the new improved varieties released by the project

Objective: Specifies a target value for the indicator of a result to be accomplished over time and the resources of the Project.Example: Improve agriculture productivity and production in Ghana by 15% at

the end of the project.

8

WAAPP M&E MANUAL

3 PURPOSE OF THIS M&E MANUAL3.1 OBJECTIVES OF THIS MANUAL

This manual is developed as part of the implementation of the West Africa Agriculture Productivity Program (WAAPP). Although the design of this manual is based mainly on the WAAPP-2A, it will be also be the framework for monitoring all series of the WAAPP (2A, 1B, 1C and 1D), as countries, CORAF/WECARD and the World Bank will propose the restructuring of WAAPP 1B and 1C to align them to the WAAPP 2A. Therefore, the M&E Manual is proposed to serve as a reference and a guide to the staff at various levels, directly involved in the monitoring and evaluation of the project:

(i) at regional level, the staff in charge of WAAPP M&E activities at CORAF/WECARD; (ii) at national, the WAAPP M&E officer and others actors involved in M&E activities3.

The guiding principles underlying the proposed methodology for implementation of this regional WAAPP Monitoring and Evaluation Information System (WAAPP-M&EIS) are:

Integration of regional WAAPP M&E system with those existing within CORAF/WE-CARD;

The shared understanding of results and indicators already predefined in the “Results Framework and Monitoring” of the Project Appraisal Document;

Consultation with key stakeholders through participatory review/updating of the Results Framework and the definition of appropriate tools for data collection, the organizational M&E framework and the flow of information between actors at various levels.

The proposed approaches and tools and the developed mechanisms are based on: The guidelines contained in the Project Appraisal Document, The participative review of the existing project RMF and indicators during a 4 days work-

shop attended by by national and regional WAAPP M&E stakeholders, The rapid assessment of the existing information systems and of the M&E operational

capacity among the staff,

This M&E Operations Manual shows and formalizes the processes and tools to help structure the plan for collecting, analysing and disseminating information on the progress of the project towards its expected outcomes, as well as the operational monitoring of activities predefined in the AWPBs. Thus, it establishes the roles and responsibilities of the staff both at regional and country levels in terms of production and / or supply of data, and usage levels. The functional relationships between the various entities within the regional WAAPP M&E management Information System (M&EIS) are described.

The regional WAAPP-M&EIS will allow: Operational planning of activities; Report on achievements of the project; Assessment of the impact, outcomes or major changes attributable to the actions of the

project.

3 In particular the WAAPP M&E focal point within the Country M&E Unit (M&E/FP)

9

WAAPP M&E MANUAL

Specifically, the regional WAAPP-M&EIS will achieve four (4) main functions:

(i) Activities planning (Strategic Plan over the project life span/phase and AWPBs,

(ii) Implementation monitoring of the AWPBs,

(iii)Results indicators monitoring (iv) Results measurements (outcome and impact evaluations).

In view of the AWPB, the implementation monitoring will: develop and maintain the dashboard of the project by center of responsibility or executing

agency; establish the rate of physical and financial implementation of the activities; Explain the discrepancies and propose corrective measures in time.

The Results Indicator Monitoring will be based on the Results Measurement Framework (RMF) and produce dashboards on the tracking of the results indicators.

As for the impact evaluations, they will have to assess the changes brought about by the actions of the project on improved productivity, increased income and increasing the food security of project beneficiaries.

The development and implementation of the WAAPP-M&EIS has been designed to be a participatory process that involves all stakeholders in the collection, analysis and dissemination of information. Such M&E system will allow transparency and efficiency in decision-making, as well as capitalizing on lessons learned during the implementation of the project.

Like any information system, the WAAPP-M&EIS is a combination of 4 interrelated key elements:

(i) Actors : who will use the M&E data? To make what decisions? and who would provide this information?

(ii) Data : What are the information needs? and what data will therefore have to be collected? How should we organize the collection of data to avoid duplication and inconsistencies that could complicate the treatment and analysis?

(iii) Procedures : Who does what and when, so that information is collected, entered into the system database, processed and disseminated to the right actor?

(iv) Tools : which tools are needed to implement the procedures?

3.2 MAIN CONTENTS AND STRUCTURE OF THE M&E MANUAL

The Manual is subdivided into eight (8) chapters and 6 annexes.

In addition to Chapter 1 which presents an overview of the WAAPP project, chapter 2 which provides some introductory basic elements on M&E and Chapter 3 which presents the purpose and structure of this M&E Manual, the next four chapters (4-7) of the Manual are dedicated to the four (4) elements of the WAAPP-M&EIS mentioned above, namely the actors, data, procedures and tools, and their interrelationships, as follows:

Chapter 1 : An overview of the WAAPP project The general context of the WAAPP project, notably a short presentation of WAAPP (objectives and components; target areas and beneficiaries)

Chapter 2 : This chapter presents:Some basic concepts/terminologies used in M&E and Result Base Man-agement;

10

WAAPP M&E MANUAL

Chapter 3 : This chapter presents the purpose of this M&E Operations Manual and how it is structured.

Chapter 4 : The Actors and their information needs

Chapter 5 : The Data : The project Results Measurement Framework (RMF) and indicators

Chapter 6 : The project M&E proceduresThis chapter is the “heart of the Manual”. It is designed for everyday use. It contains practical guidance on the 4 basic M&E functions: (i) the AWPB planning; (ii) Implementation monitoring of the AWPBs (process monitoring); (iii) the tracking of the results indicators (results monitoring); (iv) Evaluation of outcomes and impact. For each of these 4 functions the data collection and reporting mechanisms are described.

Thus, this chapter focuses on (i) the institutional and organizational framework for data collection, (ii) the analysis and processing of data by the designated stakeholders, (iii) the content of the dashboard for the implementation of the project and (iv) the standardized format for reporting.

Chapter 7 : The tools of the WAAPP-M&EIS

Chapter 8 : The resources to be mobilized for the operations of the M&E system

ANNEXES : They offer flexible templates and formats for some data collection tools, M&E dashboards.

11

WAAPP M&E MANUAL

4 THE M&E STAKEHOLDERS ROLES AND INFORMATION NEEDS

4.1 INSTITUTIONS INVOLVED IN THE IMPLEMENTATION OF WAAPP PROJECT

The implementation of the project is based on the following institutions:

4.1.1 At country level The National Steering Committee of the WAAPP (NSC/WAAPP), established in each

participating country, provides political guidance and orientation to the national coordination and implementation units of the Project. The NSC is chaired by the representative of the Ministry in charge of Agriculture, and/or livestock, or Research. The WAAPP Coordinator ensures the Secretariat of the NSC.

The Ministry in charge of the WAAPP (Ministry of Agriculture (MoA), Ministry of Livestock, Ministry of Fisheries, Ministry of Research, etc.) : is responsible for the oversight of the Project. In each country, the overall coordination, monitoring and evaluation of the Project is entrusted to a National Project Coordinating Unit (NPCU) established by the MoA. NCPU is responsible for the internal monitoring of Project indicators and outcomes as defined in the Results Framework.

The Table 2 presents the agencies and entities responsible for implementing the WAAPP components and sub-component.

In most of the countries, a National Centers of Specialization (NCoS) is responsible for measuring the impact of technologies released and transferred under Component 2. To this end, the NCoS will carry out farm surveys to measure the indicators on productivity improvement and technology adoption In addition, external service providers will be recruited to organize the required baseline survey at year 1 of the project, as well as follow-up surveys in years 3 and 5.

Each NPCU produces semi-annual/annual progress reports based on their own activities and results and those achieved by WAAPP Implementing Agencies, and synthesizing information on the national priority sector and information on research grants and results.

4.1.2 At Regional level The Regional WAAPP Steering Committee (RSC/WAAPP) .

ECOWAS chairs the Regional Steering Committee that meets once yearly to review the AWPB and implementation progress. The CORAF/WECARD executive secretariat ensures the Secretariat of the NSC.

The West and Central African Council for Agricultural Research and Development (CORAF/WECARD): is responsible for the coordination of WAAPP at regional level. Each country concludes a subsidiary grant agreement with CORAF/WECARD, detailing the nature of technical support and coordination expected from CORAF/WECARD. CORAF/WECARD has responsibilities of monitoring program outcomes and impact at the regional level. .

In order to speed up dissemination of project results and ensure achievement of the PDO, CORAF/WECARD will build further synergies with some regional organizations, that have a large network coverage, notably: AFAAS, ROPPA, WASA, the relevant CGIAR centers (IITA, ICRISAT, AfricaRice) and the International seed testing association (ISTA).

12

WAAPP M&E MANUAL

13

WAAPP M&E MANUAL

Table 2 : Implementing entities for the WAAPP components and sub-components4

Components Sub-components/ Activities Responsible for implementationMonitoring

Component A: Strengthening the enabling environment for regional cooperation for the generation and adoption of agricultural technologies

Sub-Comp A.1: Implementation of ECOWAS common regulations

A.1.01: Registration of genetic materials, pesticides, fertilizer, veterinary and animal products

CORAF/WECARD and the NPCUs on the basis of AWPBs and (MoU) and/or contracts concluded with each of concerned directorates/Implementing Agencies of the the Ministries in charge of Agriculture, Livestock or Fisheries or other government implementing partners

Sub-Comp A.2: Mainstreaming Regional Strategies into National Action Plans

A.2.01: Knowledge management, information, and communications systems

A.2.02: Mainstreaming Gender

A.2.03: Mainstreaming Climate Change

A.2.04: Sustained mechanism to financing agriculture research and development

A.2.05: Intellectual Property Rights

A.2.06: Environmental and social management safeguards

4 SEE annexe 3 OF PAD WAAPP 2A “Implementation Arrangements », page 42 to 46 and others WAAPP, for more details

14

WAAPP M&E MANUAL


Component B: National Centers of Specialization

(i) Upgrading core facilities & equipment of NCoS and key national research centers associated with the NCoS

(ii) ISO certification of NCoS(iii) Building the capacity of researchers

and facilitating regional partnerships(iv) Support to priority agricultural research

program(v) CORAF’s support to the strengthening

of the NCoS

Council for Scientific and Industrial Research in Ghana

Institut d’Economie Rurale in Mali,

Institut Sénégalais de Recherche Agricole in Senegal

Centre National de Recherche Agronomique (CNRA) in Côte d’Ivoire the Centre National de Recherche Agronomique (CNRA) in Côte d’Ivoire The Institut National de l’En-vironnement et de la Recherche Agricole

(INERA) in Burkina Faso National Institute for Fresh Water

Fisheries Research (NIFFR) in Nigeria

Institut National de Recherche Agricole du Bénin ( INRAB) in Benin

Liberia Agronomic Research Institute (LARI) in Liberia

Institut National de Recherche Agronomique du Niger (INRAN) in Niger

Sierra Leone Agricultural Research Institute (SLARI) in Sierra Leone

National Agronomic Research Institute (NARI) in The Gambia

Institut Togolais de Recherche Agronomique (ITRA) in Togo

Component C: Support to Demand-driven Technology Generation, Dissemination and Adoption

Sub-Comp C.1 : Demand-Driven Technology GenerationC.1.01 At national level (i) annual stakeholders’ workshops to develop, follow-up and update national R&D action plans (ii) grants to multidisciplinary adaptive research teams to address priority needs of stakeholders along the value chains; and test and adapt technologies generated by NCOs in the sub-region

C.1.02 At Regional levelFunding of Regional Competitive Agricultural Research Grant Scheme (CARGS)

Council for Scientific and Industrial Research (CSIR) in Ghana,

Comité National de la Recherche Agronomique (CNRA) in Mali

Fonds National de Recherches Agricole et Agro-alimentaire (FNRAA) in Senegal.

FIRCA in Côte d’Ivoire CNRST in Burkina Faso ARCN in Nigeria the PCU in Benin until the Fonds

National de Développement Agricole (FNDA) is established and operational

, in Niger by the Conseil National de Recherche Agronomique (CNRA

in Togo by the Fonds National d’Appui Institutionnel Agricole (FNAIA)

In Liberia, Benin, Sierra Leone and The Gambia the PCU will

15

WAAPP M&E MANUAL


implement the national window and transfer the technical management to the national CARGS when it will be set up

This regional window will be implemented by CORAF/WECARD

Sub-component C.2 : Support to Accelerated Adoption of Released TechnologiesC.2.01 At national level (i) annual extension stakeholders’ workshops at zonal and national levels to develop dissemination action plans; (ii)demonstrations of released technologies responding to beneficiary needs; (iii) development of efficient extension methodologies (iv) promotion of released technologies through various information systems and communications media (iv) participatory training on extension methodologies

C.2.02 At regional level (i) Policy research and action towards promoting efficient participative and pluralistic extension systems; (ii) support implementation of technology awareness action plans developed by relevant regional organizations,

Managed by CPCUs on the basis of contractual arrangements to be concluded with suitable public and private service providers

Managed by CORAF on the basis of contractual arrangements to be concluded with suitable public and private service providers

Sub-component C.3 : Facilitating access to improved genetic materialC.3 .01: Development of seed production policies and systems C.3 .02: Development or improvement of the gene banks t C.3 .03: Upgrading of the capacity of public research stations to produce and store breeder and foundation seeds C.3 .04: Scaling-up of the production of certified seeds by private enterprises, farmers’ organizations; C.3 .05: Provision of mini kits of improved seeds to producers to jump-start dissemination, along with training and technical assistance for seed companies and seed producers.

Managed by CPCUs on the basis of contractual arrangements to be concluded with suitable public and private service providers

Component D: Program Coordination, Management and M&E

(i) At the national level (ii) At the regional level

By NPCUsby CORAF

16

WAAPP M&E MANUAL

4.2 THE MAIN ACTORS OF THE WAAPP M&EIS AND THEIR INFORMATION NEEDS

The actors involved in the WAAPP M&E system can be subdivided into the system managers and the users. The managers of the WAAPP-M&EIS are primarily the Regional Program Coordination and in particular the WAAPP Regional M&E Officer (M&EO), supported by the National M&E officers and M&E focal points in the various Implementation Agencies, notably M&E data collection and reporting.

The table 3 below presents the WAAPP stakeholders, their roles/responsibilities and their information needs.

To make the WAAPP M&E Information System (M&EIS) operational, it is essential to determine and satisfy the information needs of each actor. Thereafter, the M&EIS should be designed and built to produce regularly the required M&E information for the use of the actors in their decision making process. At both levels (National and regional levels), based on the roles, responsibilities and needs of each stakeholder group, this M&E Manual will specify the type of monitoring, the tools to be uses, the sources of M&E information, the users of the information and frequency of data collection.

The information needs identified in Table 3 below will be the basis for the production of dash boards, M&E reports and financial monitoring reports that meet the expectations of those involved in the project implementation and who will contribute at the same time to the functioning of a results-based M&E information system. Table 3 below meets those requirements.

17

WAAPP M&E MANUAL

Table 3 : WAAPP stakeholder’s roles and their information needs

ACTORS/STAKEHOLDERS RÔLES / RESPONSIBILITIES INFORMATION NEEDSMonitoring report, document/Source

AT REGIONAL LEVEL

Regional Steering Committee (RSC)

Oversees all WAAPP activities at regional level Examines and Approves AWPBs presented by RPCU Reviews and determines policy and legal issues relating

to WAAPP implementation Organizes yearly regional forums to review experience

associated with WAAPP implementation at regional level

Annual Work Plan and Budget WAAPP outcome indicators Implementation progress

Project Annual report M&EIS dash board Draft regional AWPB

CORAF / WECARD

Regional Program Coordination Unit (RPCU)

Overseeing the activities of the WAAPP at regional level,

Drafts policies and guidelines for consideration by RSC, providing reports and evaluations to RSC Monitoring of AWPB implementation Serves as secretariat to the Regional Steering Commit-

tee Meetings Monitoring outcomes and impact at the regional level by: (i) to update studies on the region’s agricultural pro-

ductivity; (ii) Producing an annual consolidated report based on

the countries’ reports and their specific studies, to be shared with all ECOWAS countries; and partners

(iii) Informing participating countries on a yearly basis on implementation progress and the use of funds trans-ferred from country proceeds;

(iv) Maintaining the web-based databases research res-ults and maintaining an agricultural research re-source database in terms of publications and re-search skills available in the region.

Evolution of WAAPP outcome and impact indicators Implementation rates (physical and financial rates) of the AWPB Problems encountered in project implementation Problem in coordination and M&E activities Financial problems (disbursements, etc.). Problems with procurement Proposed solutions to problems Overall disbursement rate and per donor, Countries Performance IPs Performance

Project Annual report Project Quarterly and

semi annual reports Impact Evaluation stud-

ies M&EIS dash board

Financial and technical Supervise and monitor the implementation of the project Evolution of WAAPP outcome and impact indicators Implementation rates (physical and financial rates) of AWPBs

Project Annual report Project Quarterly re-

18

WAAPP M&E MANUAL


Partners (World Bank, MoA, etc)

Manage the credit Control of procedures (financial management, procure-

ment, etc.) Approve the AWPBs Supervise Program implementation on regular basis

Problems encountered in project implementation Problem in coordination and M&E activities Financial problems (disbursements, etc.). Problems with procurement Proposed solutions to problems Overall disbursement rates, including matching funds

ports Impact Evaluation stud-


WAAPP M&E working group (meeting twice a year, out of RSC and wrap up meeting)

Self appraisal Experience sharing Challenges & jointly devised solutions PMP Updates- jointly and comparatively Joint action planning Capacity strengthening

Evolution of WAAPP outcome and impact indicators Implementation rates (physical and financial rates) of AWPBs Problems encountered in project implementation Problem in coordination and M&E activities

Project Annual report Project Quarterly re-

ports Impact Evaluation stud-


AT NATIONAL LEVEL

National Steering Committee

Oversees all WAAPP activities at national level Examines and Approves AWPBs presented by CPCU Reviews and determines policy and legal issues relating

to WAAPP implementation at national level Organizes annual forums to review experience and is-

sues associated with WAAPP implementation at national level

Evolution of WAAPP outcome and impact indicators Annual Work Plan and Budget WAAPP outcome indicators Implementation progress

Project Annual report M&EIS dash board Draft regional AWPB Reference and impacts

studies

Ministry in Charge of WAAPP

Ensure that the WAAPP program is implemented in ac-cordance with guidelines, and with the specific require-ments of development partners

External monitoring

Evolution of WAAPP outcome and impact indicators Implementation rates (physical and financial rates) of the AWPB

Ssemi-annual; Annual reports;

M&EIS dash board

National Project Coordination Unit (NPCU)

Overall responsibility for the implementation of the WAAPP at national level Preparation of the national AWPBs;

Establish and monitor results-based Memoranda of Un-derstanding and/or contracts with Implementing Partners for the implementation of WAAPP activities

Evolution of WAAPP outcome and impact indicators Implementation rates (physical and financial rates) of the AWPB Problems encountered in implementation at national level Problem in coordination and M&E activities at national level Financial problems (disbursements, etc.) at national level Problems with procurement Proposed solutions to problems

Project semi-annual and Annual reports

M&EIS dash board Impact evaluation stud-

ies (Baseline and fol-low-up studies)

19

WAAPP M&E MANUAL


Operational support on financial management, contract administration, procurement, administrative and opera-tional cost of Implementation of the WAAPP project at national level;

Direct oversight responsibility for project administration and financial management tasks at national level;

Monitoring indicators and outcome of WAAPP as defined in the Result Framework

Monitoring and Evaluation of WAAPP activities; Preparation of progress Reports. Acting as the Secretariat of National Steering Committee

Implementing Agencies (IAs):Implementation and monitoring of their activities

Implementation rates (physical and financial rates) of the AWPB Problems encountered in implementation at national level Problem in coordination and M&E activities at national level Financial problems (disbursements, etc.) at national level Problems with procurement Proposed solutions to problems

Project semi-annual and Annual reports

M&EIS dash board Impact evaluation stud-

ies (Baseline and fol-low-up studies)

20

WAAPP M&E MANUAL

Table 4 : Summary of M&E responsibilities

Activity Responsibility M&E Reports

Internal monitoring at national level PCU of countries Semi-annual/annual progress report Annual progress report Project completion report

Internal monitoring at regional level CORAF/WECARD Semi-annual progress report Annual progress report Project completion report

External monitoring (supervision, mid-termreview)

Government of countriesWorld BankCORAF/WECARD

Supervision & implementation sup-port reports (aide-mémoire)

Mid-term review reports

Impact evaluation CNS/CROE Supply chain analysis andbenchmarking studies

CORAF/PCU will prepare ToR and PCU will recruit external consultants

Baseline surveys , Follow-up surveys Impact Evaluation reports

21

WAAPP M&E MANUAL

5 THE M&E DATA: INDICATORS AND THE RESULTS MEASUREMENT FRAMEWORK

5.1 THE WAAPP RESULTS MEASUREMENT FRAMEWORK

The WAAPP M&E Information System is framed and constructed using the logic model approach that classifies indicators as process milestones, output, outcome and impact (goal indicators). The WAAPP logic model is presented in figure 3.

The WAAPP Results Measurement Frameworks (RMF) in Annex 1 provides relevant details on the results indicators. For each result indicator, It provides descriptions for the indicator metadata by specifying: (i) the title; (ii) the definition; (iii) the unit of measurement; (iv) the baseline and target values; (v) the data source/method of data collection; (vi) the frequency of collection; (vii) party or parties responsible.

Indicators and their metadata, including: baseline values, target values, the source, method, frequency, and responsible for the data collection, are the main data of the WAAPP-M&EIS.

In this RMF presented in annex 1, the output/activity indicators are also presented in order to allow for the monitoring of implementation of the AWPBs.

5.2 WAAPP RESULTS INDICATORS5.2.1 Indicators baseline values : the initial values of indicatorsBecause it is difficult to set targets and measure performance and impact of a project without knowing the situation that prevailed at its beginning, baseline data reflecting the situation at project start must be collected as soon possible. Baseline data provide as accurate a picture as possible of the indicator values at the beginning of the project/program and serve as benchmark to measure changes that occur through the action of the project. Table 5 provides a list of WAAPP results indicators for which specific data should be obtained through a baseline survey and follow-up surveys in the project intervention areas.

5.2.2 Most important steps for data collectionThere are four main steps to be followed for adequate and reliable M&E data collection

• Develop standard forms and method for data collection

• Define and agree on data reporting responsibility and accountability between institutions at all levels

• Define data flow mechanisms from different sources

• Incorporate data verification into data collection process

5.2.2.1 Data SourcesThe sources of data collection are the places/documents etc, where information verifying the status of an indicator is collected. The Figure 2 summarizes the most common data sources depending on whether process monitoring or results monitoring is concerned. Data sources have been identified and vetted for all the indicators listed in the RMF in Annex 1. In general, process monitoring data will be obtained from various sources, ranging from the Country PCUs, to Implementing Entities and to Service Providers. As for results monitoring data, they are usually obtained through rapid appraisal and survey methods..

22

Immediateoutcome

Ultimateoutcome

Intermediateoutcome

PROCESS MONITORING RESULTS MONITORING

INTERNALCommon Source: Administrative Data

EXTERNALCommon Source: Rapid Appraisal & Survey methods

Financial & Human resources indicators

Progress in the implementation of planned activities

Output indicators Indicators of access & use Indicators of improvement in living standards

Data collected from the Programme Coordination Unit, implementing agencies, implementing partners, the Government Management Information Systems

Analyze the implementation performance (planned vs achieved) of programs and projects AWPBs

Data collected directly from the beneficiaries

Monitor the effectiveness of government programs and projects

WAAPP M&E MANUAL

Figure 2 : M&E data sources

23

ActivitiesInput Outputs

WAAPP M&E MANUAL

Table 1 : indicators data sources and frequency of data collection as indicated in the PAD

Results Indicators* Frequency Data Source / MethodologyResponsibility

for Data Collection

PDO Level Results Indicators

IDO1. Direct project beneficiaries of which 40% female- disaggregated by country Annual Assessment survey made by M&E Unit if

necessary in collaboration with consultants PCU

IDO2: Beneficiaries who are using technologies generated by other countries Annuel Annuel assessment PCU/NCOS

IDO3.Technologies generated by the Project with at least 15% productivity increase over the control – disaggregated by country

Annual Annual assessment PCU /NCOS

IDO4.Producers with knowledge of generated /released technologies by the Project- disaggregated by country Year 1, 3, 5 Benchmark and supply chain surveys PCU/NCOS

IDO5. Area under new technologies disaggregated by country Annual Annual assessment survey PCU / NCOS

IDO6.Processors/ producers who have adopted at least one new technology -disaggregated by country Annual Annual assessment / survey PCU / NCOS

COMPONENT A: Strengthening the enabling environment for regional cooperation for the generation and adoption of agricultural technologies

24

WAAPP M&E MANUAL


for Data Collection

IRA.01: Pesticide products/ and genetic materials register red by national committees - disaggregated by country

Semi-annual NPCU progress reports NPCU

IRA.02: Laboratories with ISTA/ISO certification - disaggregated by country


IRA.03: Technologies with property rights (disaggregated by country)


IRA.04: Varieties generated by NCOS/RCOE registered in Regional catalogue - Disaggregated by country

Semi-annual

IRA.05: Hits for the regional/national Web-based information system on agricultural technologies and research skills


IRA.06: National action plans developed on gender, communication and climate change-disaggregated by country


Component B: Strengthening National centers of specialization

IRB.01: Qualifying criteria met by the NCOS to become RCOE - disaggregated by country


IRBA02: NCOS/RCOE with ISO certificate - disaggregated by country


IRB.03:: Technologies generated by NCOS and demonstrated in at least three ECOWAS countries outside the country of origin - disaggregated by country


IRB.04: Study sponsorship- desegregated by country and MSc and PHD


25

WAAPP M&E MANUAL


for Data Collection

IRB.05: Scientific exchange visits - disaggregated by country Semi-annual NPCU progress reports NPCU

Component C: Support to demand-driven technology generation, dissemination and adoption

IRC.01: Multi-country research proposals financed by the regional CARGS - disaggregated by country


IRC.02: National demand-driven research projects financed by the national CARGS - disaggregated by country


IRC.03: Demonstration plots established (breakdown by country )


IRC.04: Number of technologies demonstrated – disaggregated by country


IRC.05: Percentage of targeted clients (male and female farmers or businesses) satisfied with extension services


IRC.06: Certified foundation seeds and improved breed stock produced – disaggregated per country and commodity


Component D: Project Management and Coordination

IRD.01: Percentage of procurement activities executed in conformity with the timing of the procurement plan


IRD.02: Disbursement rate of IDA funds. Semi-annual NPCU progress reports NPCU

IRD.03: National M+E system with database functional (structure, data collection & organization, up-to-date, reporting)


26

WAAPP M&E MANUAL


for Data Collection

IRD.04::% of subproject granted with environmental management plan that have been implemented effectively


27

WAAPP M&E MANUAL

5.2.2.2 Method of Data Collection The data for higher level indicators (some PDO indicators) will be drawn from surveys conducted by the RPCU and the NPCUs, while the lower level indicators will be drawn from reporting from CPCUs, IAs and IPs. On the basis of indicators data collection forms (see annex 5), data will be reported on through the M&E Management Information System. Data will be reported to WAAPP M&E Officers on a quarterly or semi-annual or annual basis, depending on the indicator’s requirements.

Countries with the support of CORAF/WECARD will also commission surveys to collect special socio-economic data, if necessary. Data collection instruments will be designed in a participatory manner with the project stakeholders. To ensure the relevance and compliance of data collected with the specific requirements of the national entity responsible for Statistics, the WAAPP will collaborate closely with the National Statistic System. Commissioned special studies will standardize the data collection tools and methods and will use sample frames designed and/or approved by the National Statistic System.

5.2.2.3 The Frequency of Data Collection For each indicator, the RMF provides the frequency of data collection. Data will be collected on a quarterly, semi-annual or annual basis, depending on the indicator (See table 5). Baseline and periodic follow-up surveys have also to be scheduled and arrangements established with various institutions to regularly provide data for the impact assessment of the project activities on the various beneficiaries. The frequency and timing of such surveys will generate information that the Project will use to establish the baseline situation, the mid-term review, and the Final Evaluation.

5.2.2.4 Responsible Entities for Data Collection The RMF identifies at least one entity to be responsible for data collection for any given indicator, The entities responsible for providing data on each indicators will be the RPCU, the CPCUs, Implementing Entities and their partners,. The Tables 2 and 5 describe responsibilities for data collection in more detail.

28

WAAPP M&E MANUAL

Figure 3 : WAAPP Logic ModelIM

PACT

LEV

ELO

BJEC

TIVE

LEVE

LO

UTC

OM

E LE

VEL

OU

TPU

T LR

VEL

29

COMPONENT 1the enabling Conditions for Regional

Cooperation in the Generation, Dissemination and Adoption of

Agricultural Technologies strengthened

COMPONENT 2National Centers of Specialization / Regional

Centers of Excellence strengthened

COMPONENT 3Demen-driven technology generation,

diffusion and adoption developed

COMPONENT 4Program Coordination, Management

and Monitoring and Evaluation performed

WAAPP OVERALL OBJECTIVE:the generation, dissemination and adoption of

improved technologies in the participating countries’ priority agricultural commodity areas are scaled up

WAAPP GOAL:<

Increased agriculture productivity and production inpriority value chains in participating countries

OUTPUT UNDER C1regional regulationsseed and pesticides committeeslaboratoriesseed cataloguesregional and national action planinformation system

OUTPUT UNDER C2NCOS/RCOEtechnologiestrainingexchange visits

OUTPUT UNDER C3technologiesregional and national research projectsseeds/genetic materialsdemonstration plots

OUTPUT UNDER C4M&E systemfinancial systemprocurement systemEnvironment and social

WAAPP M&E MANUAL

6 WAAPP M&E PROCEDURES 6.1 GENERALOperationally, the implementation of WAAPP M&E is organised in a general frame at 2 levels:

1. At regional level ,

The RPCU has responsibilities of monitoring outcomes and impact of the Program at re-gional level, as well as the implementation of its own AWPB. The RPCU will carry out the results monitoring activities by:

(i) subcontracting with appropriate regional and international agencies to update stud-ies on the region’s agricultural productivity;

(ii) producing semiannual and consolidated annual report based on the annual coun-tries’ reports and their specific studies, to be shared with all participating countries; ECOWAS, WB and members of Regional Steering Committees

On the process monitoring side, the RPCU will produce semi annual progress reports fo-cusing on implementation of the AWPB. It will also inform participating countries on a yearly basis on implementation progress and the use of funds transferred from countries.

(i) Monitoring of the implementation of the AWPB (Process monitoring) at re-gional level is under the overall responsibility of the RPCU. It is carried out through the review of semi annual/ annual CORAF and country implementation reports fo-cusing on implementation progress and financial information.

(ii) Monitoring of the outcome indicators (Results monitoring) is under the overall responsibility of RPCU with the support of NPCU. The results monitoring is based on the semi-annual data collection and reporting by the country PCUs and key IAs and IPs at country level. On the basis of these progress reports, the M&E Focal Points within the country PCUs and CORAF/CARD updates the M&E database, fo-cussing WAAPP indicators.

(iii) Impact evaluation of the WAAPP is also under the responsibility of RPCU (general guidance to harmonize methodology, and specific studies), which will need to sub-contract the necessary impact assessments.

2. At country level

The NPCUs and key Implementing Agencies (IAs) at country level support the Results monitoring activities at regional level mentioned above. The Country M&E Focal Points (at NPCUs and IAs levels) rely on the National Implementing Partners (NIPs) in order to fulfill their tasks and to develop semi annual and annual report. As part of the M&E data quality control, they perform regular routine field visits to make sure that the activities are imple-mented as planned and that the reporting is accurate. Those field visits will also provide op-portunities to collect and cross check the M&E data collection and reporting forms filled by the IPs.

30

WAAPP M&E MANUAL

The impact assessment of technologies released and transferred under Component 2 is under the responsibility of NCoS in participating countries. To this end, the NCoS carry out farm surveys to measure the indicators on productivity improvement5and technology adoption6. Such surveys will be conducted annually and will include if possible, both a control group and beneficiaries of released technologies.

These two levels7 work together to achieve the production of AWPBs, the process and results monitoring and the outputs of the M&EIS such as the dashboard, the semi-annual and annual M&E reports.

Therefore, the M&E plan to be implemented by WAAPP will have five main functions:

a) Operational and Strategic planning of the project activities (Activity planning); b) Monitoring of the implementation of the activities of the AWPB (Process and outputs

monitoring); c) Monitoring of results (Outcomes, Impacts) (Results monitoring);

d) Reporting on the project progress (On AWPB implementation as well as on outcomes)

e) Measurements of the outcomes and impacts (Impact evaluations)

For each of these five functions, this M&E Manual needs to define WHO? does WHAT?, WHEN? and HOW?

6.2 OPERATIONAL PLANNING 6.2.1 Preparation of the Project Implementation Plan (PIP) The Project Implementation Plan (PIP) must be prepared at the start of the project, using a participatory approach and the basic documents of the project, notably the PAD, the RMF and the Project Implementation Manual (PIM). The PIP is prepared taking into account the whole duration of the Project and all the activities that are necessary to achieve the objectives. The PIP takes into account all the activities to be carried out over the life-time of the project in order to achieve project objectives. The Officers in charge of WAAPP M&E could help to review the PIP. This work must be done in close collaboration with the other stakeholders, both at regional and national levels, who are involved in the project implementation.

6.2.2 Preparation of the Annual Work Plans and Budgets (AWPB) 6.2.2.1 Content of the AWPBThe Annual Work-Plan and Budget (AWPB) describes the planned activities and expenditures of the project for each financial year. The AWPB must show at least:

1. Annual targets related to output/activity indicators (including target, achievement in previous years, planned achievement in current year and revised target by end of project if it is the case);

2. Status of activities:

5 See indicator IDO3, Annex 1 “technologies generated by the project with at least 15% productivity increase”6 See indicator IDO6, Annex 1 “Processors/ producers who have adopted at least one new technology”7 Namely, national, provincial and commune levels.

31

WAAPP M&E MANUAL

3. Summary of planned expenditures by component and by implementing entity (RPCU, NPCUs and IAs);

4. Summary of expenditures by category (project budget total for category, amount spent in previous years, amount planned in current year, and planned amount outstanding at the end of the current year).

5. Detailed AWPB for each implementing agency;

6. Updated Procurement Plan.

6.2.2.2 The AWPB preparation processThe preparation process of WAAPP AWPB has eight (8) main steps at the level of CORAF/WECARD Executive’s Secretariat:

Step 1: The Executive Secretariat of CORAF/WECARD8 or the RPCU/WAAPP hold a preparatory meeting no later than August 31 of each year (N), to establish the guidelines and priorities for the AWPB of the next year (N+1), on the basis of:

The project performance indicator as reflected in the RMF The technical and financial progress reports of the first 3 quarters of the current year

(N); The technical and financial progress reports of the previous year (N-1);

Step 2: the Planning Officer of CORAF/WECARD write a concept note of the AWPB for the year N+1 and submit it no later than 5 th September for approval by the Executive Director.

Step 3: From 6 th to 30 th September the WAAPP draft AWPB of the year N+1 is elaborated

Step 4: The draft AWPB of the is consolidated with the other draft AWPBs of the various cost centers of CORAF/WECARD and transmitted to the Executive Director no later than the 5 th October .

Step 5: The draft AWPB of CORAF/WECARD (including) the draft WAAPPP AWPB is sent to members of the Scientific and Technical Committee (STC) no later than 15 th October of each year

Step 6: The draft AWP validated by the Executive Secretariat is submitted to:

The STC of CORAF/WECARD for scientific approval no later than 30 th

October The RSC of WAAPP which shall meet during a budgetary and technical

session no later than 15 th November . The Governing Board of CORAF/WECARD which shall meet during an

ordinary session no later than 30 th November .

8 The Executive Director, the Programs Director, the Administrative & Finances the Information & Communications Director and their collaborators

32

WAAPP M&E MANUAL

Step 7: The adopted documents shall be transmitted to IDA for no objection no later than 5 th

December.

IDA transmit his No Objection Letter no later than 31 December

Step 8: The AWPB approved by the SCT of CORAF/WECARD, the WAAPP RSC, the Governing Board of CORAF/WECARD and IDA, is transmitted to the WAAPP accountant for data input, updating and setting up in the financial management system no later than 5 th January .

The AWP will include the following documents:

The activities plan and the indicators by component and sub-component;

The schedule of implementation of the activities and tasks undertaken by the NPCUs.

The budgetary resources management plan, including the financing plan, the procurement plan and the cash flow plan.

6.3 MONITORING OF THE IMPLEMENTATION OF THE AWPB (PROCESS MONITORING) 6.3.1 Objective and scopeProcess monitoring is one of the most important M&E tasks. It refers to the progress and implementation of the different components, sub-components and activities of the project. It is closely related to the elements of planning. The main objective is to allow the WAAPP Management at regional level and national level and other relevant stakeholders to be able to know, for each given activity, whether it has been implemented according to the plan (When? By whom ?) and how much it cost.

A link must be established between the level of disbursement (or commitment) and the physical implementation of each activity. It should be possible to link each disbursement to an implemented activity of the project. Harmonization in coding has to take place between the financial management and the M&E unit of the project.

33

WAAPP M&E MANUAL

Tableau 5 – Template for AWPB Implementation Monitoring Form (IMF) for the Present Period (Quarter or Semester)

Code

Activity Indicator or milestone

Physical achievementFor present period

Financial achievement for present period Analysis

Level of implementation Quantity

achieved

% achieve

dBudgeted spent % spent Problems

encountered Actions to be taken comments

Level of implementation: (i) On-going; (ii) completed, (iii) Cancelled (Explain why under comments)

Table 1 – Template for accumulated values of AWPB Implementation Monitoring

Code

Activity Indicator or milestone

Physical achievementUp to dtae

Financial achievement up to date Analysis

Level of implementation Quantity

achieved

% achieve

dBudgeted spent % spent Problems

encountered Actions to be taken comments

Level of implementation: (i) On-going; (ii) Completed, (iii) Cancelled (Explain why under comments)

34

WAAPP M&E MANUAL

35

WAAPP M&E MANUAL

6.4 RESULTS MONITORING

6.4.1 Organizational set up for data collection on results Project Result monitoring means the whole set of procedures implemented to produce, on a regular basis, the information necessary to establish the value of each outcome indicator (immediate, intermediate, and ultimate outcome indicators). Results monitoring of WAAPP is under the overall responsibility of the RPCU and the NPCU. The results monitoring is based on the tracking of results indicators listed in the RMF which provides the guidance for data collection and reporting by the RPCU, NPCUs, key Implementing Agencies (IAs) and Implementing Partners (IPs) at country level.

This is facilitated by the use of a Results Indicators Tracking Table (RITT) shown in table 10.The WAAPP regional M&E Officer makes sure that the RITT is properly and timely updated by the M&E Focal Points within NPCUs, key Implementing Agencies (IAs) and Implementing Partners (IPs) at country as well as at regional levels.

6.4.2 Source / Method of data collection As indicated in figure 2, results monitoring data are generally obtained through socio-economic studies and impact assessment surveys. However, some data on results indicators may be obtained from other sources or methods as indicated in the RMF (Annex 1).

Generally, data for higher level indicators9 will be drawn from surveys conducted by WAAPP, Implementing Entities, and Service Providers, while the lower level indicators will be drawn from various Project implementers.

Ideally, the indicators should be reported on through a web based M&E Management Information System (MIS). In such case, the data input system could be decentralized, with the M&E Focal Points (M&E/FP), at country and regional levels, providing the data directly into the MIS in real time.

6.4.3 Dashboard (tableau de bord) for results monitoring The regional M&E Officer will provide the tracking of the result indicators using the templates presented in the following tables 10 and 11

9 intermediate and ultimate outcomes indicators

36

WAAPP M&E MANUAL

6.5 REPORTING ON THE PROJECT PROGRESS TOWARDS RESULTS

6.5.1 The Framework of reporting

Report / document Date of submission

Responsible Entity

Recipient

Semi-annual Technical dashboards

45 days after the semester

NPCURPCU

CORAF/WECARD; WAAPP CoordinatorWAAPP regional Implementing partners

Semi-annual Technical and Financial progress reports

45 days after the semester

NPCURPCU

CORAF/WECARDWAAPP CoordinatorWB, ECOWAS, WAAPP Implementing partners

Quarterly financial reports 15 days after each quarter

NPCURPCU

CORAF/WECARDWAAPP CoordinatorWB

Annual report 60 days after the period

NPCURPCU

CORAF/WECARDWAAPP CoordinatorWB, ECOWAS, WAAPP Implementing partners

AWPB 1st novembrer of each year

NPCURPCU

Regional Steering Committee; WB TTLCORAF;

6.5.2 M&E data processing The processing of the WAAPP M&E should be based on relational database software allowing decentralised data entry and consultation. This will enable the WAAPP managers, as well as implementing partners to directly input the M&E data into the system. This would also enable key stakeholders and managers with authorized access to have the possibility to regularly and at any time, have an overview of the project implementation.

The next section 7.3 gives more details on the technical specifications of the tool to be implemented.

37

WAAPP M&E MANUAL

6.6 EVALUATION OF WAAPP PROJECT 6.6.1 OverviewEvaluation is an essential element of the WAAPP M&E system. It intends to assess the Project’s rationale, effectiveness, efficiency, and sustainability. It strives to estimate the impacts on the targeted Land recipients’ livelihood. The evaluations will provide WAAPP management, the RSC, WB, ECOWAS and other stakeholders with information on whether or not the intended outcomes are likely to be achieved (or are achieved), and at the project’s end on whether the impacts from those outcomes are attributable to the Project. Evaluations will be guided by the economic analysis, estimated Project impacts, and research questions. They will draw upon WAAPP basic documents10 but can also include discussions with key stakeholders such as relevant WB staff, WAAPP staff, implementers, and beneficiaries.

The Evaluations are implemented at three periods:

6.6.2 Pre-Project or ex-ante evaluation This evaluation has already been done facilitated by countries with the support of WB and CORAF/WECARD. It involved an economic analysis estimating the ERR for the Project. This analysis evaluated the investment’s likely economic value using a cost-benefit framework; the project’s costs were assessed with its corresponding quantified benefit streams. Assumptions used in the Project’s benefit streams have been documented in the PAD. Assumptions in the economic analysis will be examined in the Final Evaluations, which are discussed later in this section.

6.6.3 Mid-term Evaluation The Mid-term Evaluation will assess progress towards meeting the project’s goals, objectives and outcomes. This evaluation will provided feedback for Project management on implementation experience, reinforce positive impact and mitigate adverse impact through modifications to design and implementation.

6.6.4 Final Evaluations Final evaluations will be carried out countries with the support of WB and CORAF/WECARD, at the completion of the Project period by independent evaluators. The focus will be on the outcomes, objectives and the sustainability of the project. Upon completion of the project CORAF, the participating countries and the WB wish to comprehensively assess the following fundamental questions:

i. Did the Project meet its development objective and its expected outcomes? ii. What lessons can be learned from the implementation experience (both procedural and

substantive)?

10 Documentation includes, the PAD, work plans, contractors’ reports, WAAPP commissioned data collection, externally available data, this M&E Manual, and other relevant information

38

WAAPP M&E MANUAL

7 THE TOOLS OF THE M&E SYSTEM7.1 PRESENTATION OF THE TOOLS

The WAAPP M&E information system will make use of the tools necessary for strategic planning (or preparing Project Implementation Plan), operational planning and budgeting (the Annual Work Plan and Budget), and the data collection, processing and sharing of information tools. The three (3) main M&E tools to be mobilized to support the M&EIS of the project are:

1. The Strategic and operational planning tools (Project Implementation Plan & the AWPB)

2. The Implementation Monitoring Records of the AWPB activities;

3. The Management Information System (MIS) using a relational database11 tailored to the needs of the WAAPP M&E and for a decentralized management, planning and monitoring and evaluation of the project.

ToolM&E Function content

Data collection

toolReporting tool

PIP Planning: The main purpose of the PIP would be to serve as a reference and a guide for the preparation of the AWPBs

The PIP is a work plan and budget developed over the lifetime of the project

AWPB Process monitoring activities, sub activities, budget, responsible, schedules of activities and other relevant details to facilitate the planning

Data collection forms

AWPB Implementation Monitoring Form (IMF)

RMF Monitor all indicators of the Result Framework

Levels of results and indicators (output, intermediary, outcome, impact)Indicators documented (targets, achievements, percent of achievement) for each reporting period

Data collection formsIndicators description sheets

Result Indicator Tracking Table (RIIT)

PAD; RMF

Evaluations Ex-ante evaluation A technical, social, environmental and economic analysis

Mid-term Evaluation progress towards meeting

Impact assessment surveysAnnual Reports

Evaluation reports

11 Preferably a web based M&E Management Software

39

WAAPP M&E MANUAL

the project’s goals, objectives and outcomes to provide feedback for Project management

Final evaluations

focus on outcomes, object-ives and the sustainability (i) Did the Project meet its de-velopment objective and its expected outcomes, (ii) What lessons can be learned

7.2 THE WEB BASED M&E SOFTWARE 7.2.1 General characteristicsThe database and the data processing application software form a set that facilitates the planning, monitoring and evaluation of the project. It should be a web based and project oriented M&E information management system fully customized for the WAAPP M&E needs. The software should be accessible via internet to allow decentralized access for data entry, consultation and secure access anywhere Internet is available, at least the country level. The software must have the following features and qualities:

7.2.1.1 System flexibilityThe system must be highly configurable thanks to the provision of advanced management features. It should allow changing the settings at any time. For example, the analytical framework should help manage an unlimited number of levels of disaggregation. Also, the value lists of values should be fully modifiable at any time (eg, field of activity, type of training, etc.)..

7.2.1.2 System scalabilityThe system must support a n-tier architecture to separate the presentation layer, the business logic layer and the data layer. This facilitates changes to one of the layers if the need arises. To do this, the programming environment must be able to create models, views and controllers that represent the data layer, the layer of film viewing and treatment.

7.2.1.3 System security The system should provide solid security measures (database, interface, communication channel, server) to prevent data loss, protect confidential data, ensure data integrity and prevent intrusion. To do this, several security measures including data backup, the use of SSL certificates, validation of data formats, etc., must be implemented.

7.2.1.4 System simplicity The system must be easy to use, configurable and manageable by non professionals. For example, reports and forms strongly resembling the ones that the project staff is already using should be developed. Hence, users move easily through the application and quickly learn to use its various features.

40

WAAPP M&E MANUAL

7.2.2 The various modules to be providedA geographical plan and an analytical plan must be developed in order to link the components and activities and the indicators. These plans should not be static, but evolve along with the progression of the implementation of project activities. The M&E Officers at regional level manage the database and ensure that performance indicators and impact identified are well documented on time, and that the implementation of the AWPB is monitored rigorously and in real time. The finalized M&E Information Management Software should include the following modules:

1. The Planning module is used to enter and track work plans, budgets and funds commit-ted;

2. The module "Stakeholders Management" allows entering and storing relevant informa-tion on all major project stakeholders: Project staff, IPs, services providers, etc.

3. The module "Indicators" captures the values of the indicators and keep track of each in-dicator, in addition to presenting the dash board of overall set of indicators, indivcators values should be entered at the end of each quater.

4. The module "Procurement" captures the procurement plan and tracks its implementa-tion in real time;

5. The module "Training / Meeting" is used to enter and track M&E data on: workshops, meetings, training, field missions.

6. The module "Reports / Consult" to edit the reports of a particular analytical level, and the various M&E summary tables or dash boards.

7. The module "Filing" allows you to file, archive documents in the M&EIS, especially those required as means of verification.

8. The module "Geo-View" allows having a geographic visualization of some of M&E ele-ments including: delivery rate indicator, schemes status, etc.

The general architecture of the monitoring system (organization of the database, input and output data) is as shown in the figure 4 on next page.

41

WAAPP M&E MANUAL

Figure 4 : The general architecture of the WAAPP M&E Information management System

42

WAAPP M&E MANUAL

8 DATA QUALITY ASSURANCEThe WAAPP emphasizes data quality because the program is explicitly evidence based and results oriented. Good data are needed to inform the design of interventions and to monitor and evaluate the quantitative progress toward pre-determined program targets. Ultimately, the program is committed to accuracy of information for purposes of accountability and, more importantly, for use of quality data to improve programs and to measure program benefits and impacts on targeted beneficiaries.

In order for reported outputs of the system to be used, the quality of the data on which they are based must therefore meet minimum standards that includes accuracy, reliability, completeness, precision, timeliness, and integrity of the information system. For performance data to be useful in managing for results and credible for reporting, five dimensions of data quality standards need to be defined for WAAPP. The data quality standards are explained in Table below.

Data Quality Dimension Operational Definition

Validity Are the indicators defined well and are data reported in an appropriate format?

Do the indicators accurately represent reality?

Do they have a verifiable source? Is all necessary data present?

Adequacy is also taken into account under this dimension:

To what extent do the indicators for particular expected results fully measure them?

Are they sufficient to characterize and/or measure the results?

Can indicators be disaggregated by age, gender, categories etc?

Reliability Do data values give conflicting information? Is there:

1. Measurement Error (for surveys only): Were the questions in the survey/ques - tionnaire clear, direct, and easy to understand? Was an appropriate sampling frame used? Were enumerators well trained?

2. Transcription Error: Is there a process to limit transcription error? Have data er - rors been tracked to their original source and mistakes corrected?

3. Consistency: Is the data gathering process (including instrument and sampling process) same over time and across Projects/regions? Is the data internally consistent (totals equal sum of parts, etc.)?

4. Quality Control: What are the procedures (data collection, maintenance and process) to guard against bias? Are procedures reviewed periodically? Are there random checks at each stage?

5. Transparency: Are the procedures in writing and are problems reported? Do data values give conflicting information?

Timeliness Data are timely when they are up-to-date and when the information is available on time.

Practicality is also included under this dimension: Is the collection of data for the indicator a reasonably viable matter? Is it cost effective? Are there reasons that make data collection non

43

WAAPP M&E MANUAL

Data Quality Dimension Operational Definition

feasible?

Precision This means that the data have sufficient detail.

Is the margin of error reported? Is the margin of error less than expected change in the indicator? Is the margin of error acceptable for decision-making, given cost/benefit? Does the target include margin of error?

Integrity Are data subject to political and/or personal manipulation? Is there independence in key data collection, management and assessment? Is there an impartial review of entire data gathering process? Is there integrity (between records)? Are data maintained in accordance with international or national confidentiality guidelines?

Data Quality Reviews (DQR) must be critical component of WAAPP M&E System. DQR enable the analysis of data sets and data management system using the criteria listed above to inform on how best needed information is obtained and utilised.

The DQR will aim at:

Understanding the program and corresponding activities.

Depicting clearly the data flow.

Identifying the weaknesses (if any) of the indicators, their data gathering and calculation methods.

Understanding the data management systems, including the roles and responsibilities of all relevant actors.

Identifying gaps in data quality and making recommendations for overcoming weaknesses.

Data quality assurance methods and tools have to be integrated at all levels of the M&E information management system12 to address the entire process by which the data are collected, transmitted, managed, and reported.

An independent person (or entity) from the unit which produces the data should be used to undertake data quality assurance: M&E data quality assurance must be conducted at 4 levels:

The Implementing Agencies (IAs) should be responsible for data quality assessment at local level where the primary data is collected;

The national M&E Officer at NPCUs should be responsible for data quality assessment at the levels of IAs;

The regional M&E Officer should be responsible for data quality assessment at the levels of NPCUs;

An entity independent from WAAPP should conduct data quality reviews, at least once during the lifetime of the program, preferably few months before the Mid-term Review Mission.

The following tables provide templates for data quality assurance and reporting.

Table 6 : Template for Reporting & Data Quality Assessment - Regional M&E Unit level

12 From the local level where primary data is collected, from the Implementing Agency, the NPCU to the RPCU level

44

WAAPP M&E MANUAL

Date of Review:……………………………………………….

Reporting Period Verified:…………………………..

Component of the M&E System

Answer Codes:1. Yes - completely2. Partly3. No - not at all4. N/A

REVIEWER COMMENTS(Please provide detail for each

response not coded "Yes - Completely".

Detailed responses will help guide strengthening measures. )

Part 1: Reporting performance

1 How many reports should there have been from NPCUs? [A]

2 How many reports are there? [B]

3 Calculate % Available Reports [B/A]

4 Check the dates on the reports received. How many reports were received on time? (i.e., received by the due date). [C]

5 Calculate % On time Reports [C/A]

6 How many reports were complete? (i.e., complete means that the report contained all the required indicator data). [D]

7 Calculate % Complete Reports [D/A]

Part 2. M&E Systems Assessment

I - M&E Structure, Functions and Capabilities

1 There is a documented organizational structure/chart that clearly identifies positions that have data management responsibilities at the M&E Unit

2 All staff positions dedicated to M&E and data management systems are filled.

3 Current human resources at the M&E Unit are sufficient to ensure good quality M&E

4A senior staff member (e.g., the Program Manager or the WAAPP Coordinator) is responsible for reviewing the aggregated numbers prior to the submission / release of reports from the M&E Unit.

5 There are designated staff responsible for reviewing the quality of data (i.e., accuracy, completeness, timeliness and confidentiality ) received from PIUs

6 There is a training plan which includes staff involved in M&E and data collection and reporting at all levels in the reporting process

7 All relevant staff have received training M&E and on the data management processes and tools

II-Reporting Guidelines

8 The M&E Unit has documented the definition of the indicator(s).

9The M&E Unit has shared the definition of the indicator(s) with all relevant levels of the re-porting system (e.g., NPCUs, IAs).

10

The M&E plan shows a description of the services (activities) that are related to each indic-ator measured by the Program

11

There is a written policy that states for how long source documents and reporting forms need to be retained

1The M&E Unit has provided written guidelines to all PIUs on reporting requirements and deadlines

45

WAAPP M&E MANUAL






2The M&E Unit has provided written guidelines to NPCUs …

13

,,, what they are supposed to report on.

14

… how (e.g., in what specific format) reports are to be submitted

15

… to whom the reports should be submitted

16

… when the reports are due.

III - Data-collection and Reporting Forms and Tools

17

The M&E Unit has identified standard reporting forms/tools to be used by all reporting levels

18

All implementing agencies (IAs) use the same reporting forms and report according to the same reporting timelines.

19

….The standard forms/tools are consistently used by IAs

20

Clear instructions have been provided by the M&E Unit on how to complete the data collec-tion and reporting forms/tools

21

All source documents and reporting forms relevant for measuring the indicators are available for auditing purposes (including dated print-outs in case of computerized system).

IV- Data Management Processes

22

The M&E Unit has clearly documented data aggregation, analysis and/or manipulation steps performed at each level of the reporting system

23

Feedback is systematically provided to PIUs on the quality of their reporting (i.e., accuracy, completeness and timeliness)

24

There are quality controls in place for when data from paperbased forms are entered into a computer (e.g., double entry, post-data entry verification, etc).

25

There is a written back-up procedure for electronic data and information

26

...If yes, the latest date of back-up is appropriate (e.g., back-ups are weekly or monthly).

46

WAAPP M&E MANUAL






27

There is a written procedure to address late, incomplete, inaccurate and missing reports; in-cluding following-up with IAs on data quality issues

28

If data discrepancies have been uncovered in reports from IAs, the M&E Unit has docu-mented how these inconsistencies have been resolved

29

The M&E Unit can demonstrate that regular supervisory site visits have taken place and that data quality has been reviewed

47

Documents

MINISTERE DES INFRASTRUCTURES - waapp-ppaao.org€¦ · Web view- World Bank : « Template and ... (iii) the tracking of the results indicators (results monitoring); (iv) Evaluation