END LINE EVALUATION REPORT OF INCREASING SMALLHOLDER ... · Training Centers (ISAP) project was funded by Alliance for a Green Revolution in Africa (AGRA) and implemented by Oxfam

OXFAM AMERICA Evaluation Report

END LINE EVALUATION REPORT OF INCREASING SMALLHOLDER AGRICULTURAL PRODUCTIVITY THROUGH IMPROVED FARMER TRAINING CENTERS (ISAP) PROJECT FINAL REPORT

Submitted by: SEGEL Research and Training Consulting PLC

BOLE SUBCITY, WOREDA 6 KAL BUILDING, ROOM # 406

P.O.BOX: 100662 Addis Ababa, Ethiopia

Tel + 251- 9 26 80 51 46 Tel + 251- 11 851 4535

Email: [email protected] Website: www.segelconsult.com

http://www.segelconsult.com/

2

As part of our commitment to accountability and learning, Oxfam will share conclusions and recommendations from evaluations. Internally we will share with relevant stakeholders, ensuring that they have an opportunity to participate in discussion of those results in meaningful ways. We will also publish the evaluation reports on our website in accessible language. As a rights-based organization, accountability, particularly to the communities we seek to serve, is of the highest importance to us. For Oxfam, accountability requires Oxfam to regularly and honestly assess the quality of its work, share and learn from its findings with primary stakeholders, and apply that learning in future work.

This is an evaluation of Oxfam America’s Increasing Smallholder Agricultural Productivity through Improved

Farmer Training Centers (ISAP) Project. The project has been operating in Ethiopia since January 1, 2014 and

this evaluation covers the work undertaken between 2014 and 2016.

The major evaluation activities took place in December, 2016. The evaluation was carried out by SEGEL Research and Training Consulting PLC (www.segelcosult.com/) through a competitive process and reflects the findings as reported by him/her/them as validated with stakeholders. The evaluation was managed by ADIL YASSIN, MEL COORDINATOR, and BERHANU DIRIRSA, AG PROJECTS COORDINATOR, from Oxfam America, and commissioned by MANDEFRO NIGUSSIE, DEPUTY REGIONAL DIRECTOR, ETHIOPIA. For additional information regarding the evaluation Terms of Reference, please refer to the report appendices.

i

TABLE OF CONTENTS

Table of Contents ................................................................................................. i

List of Tables ................................................................................................... iii

List of Figures .................................................................................................. iii

Appendix Table ................................................................................................ iii

List of abbreviations and Acronyms ................................................................. v

Executive Summary ............................................................................................. 1

Overview of intervention ................................................................................... 1

Objectives of the evaluation ............................................................................. 1

Evaluation methodology ................................................................................... 1

Summary of Findings/conclusions .................................................................... 2

Main recommendations .................................................................................... 5

Introduction ......................................................................................................... 5

Overview of context and project ....................................................................... 6

Major project interventions and key stakeholders ......................................... 7

Purpose of the Evaluation ................................................................................ 8

Evaluation DESIGN ............................................................................................. 9

Key Evaluation Questions ................................................................................ 9

Evaluation Team ............................................................................................ 12

Evaluation Methodology ................................................................................. 12

Evaluation Limitations .................................................................................... 14

Evaluation Findings ........................................................................................... 16

1. Demographic and socioeconomic profile of sample households . 16

2. Relevance/appropriateness of the project interventions ............... 17

2.1 Compliance of the project interventions with policies/strategies .......... 17

2.2 Project relevance to the priorities and needs of local communities ...... 17

3. Effectiveness of the project interventions ...................................... 19

3.1 Improving the organizational and technical capacity of the FTCs and local institutions in delivering demand-driven extension services ........ 19

3.2 Strengthening FTCs income generation capacity ................................ 21

3.3 Improving access and use of integrated soil fertility management technologies and practices .................................................................. 22

ii

3.4 Enhancing capacity of FTCs and skills of DAs to improve access to market information for farmers ............................................................ 24

3.5 Enhancing capacity of FTCs and skills of DAs to organize youth and women ................................................................................................ 25

3.6 Identify and document best practices, success stories, and lessons learned for use in future scale-up efforts ............................................. 27

4. Efficiency ........................................................................................... 28

4.1 Timeliness and quality of implementation ............................................ 28

4.2 Appropriateness of the implementation approach ............................... 28

5. Impact ................................................................................................ 29

5.1 Percentage increase in income of the smallholder farmers ................. 29

5.2 Percentage increase in productivity of major crops and livestock ........ 29

6. Sustainability ..................................................................................... 30

7. Lessons learnt ................................................................................... 31

Conclusions and Recommendations ................................................................. 33

Conclusion ................................................................................................... 33

Recommendations ....................................................................................... 34

References ........................................................................................................ 35

Appendix 1 ........................................................................................................ 38

Appendix 2 ........................................................................................................ 45

iii

LIST OF TABLES Table 1: Increasing Smallholder Agricultural Productivity through Improved

Farmer Training Centers (ISAP) project end line evaluation matrix ...... 10

Table 2: Sample FTCs and households selected for the evaluation ................... 12

Table 3. Qualitative data collection tools and number of partciapnts .................. 13

Table 4: Demographic characteristics of sampled households .......................... 16

Table 5: Sample households land ownership status .......................................... 17

Table 6. Use of agricultural extension service in sample kebeles before and after ISAP project ......................................................................................... 18

Table 7. Use of yield increasing inputs by male and female farmers in sample kebeles before and after ISAP project .................................................. 21

Table 8: Sample FTCs annual income statement, Birr (2015/16) ....................... 22

Table 9. Use of component of Integrated Soil Fertiliy Management (ISFM) practcies by male and female farmers in in sample kebeles before and after ISAP project ................................................................................. 23

Table 10. Use of different soil fertlity management practcies in the sample kebeles after ISAP project ................................................................... 23

Table 11. Access to improved market information service by male and female farmers in sample kebeles before and after ISAP project ..................... 25

Table 12.Changes in the income of smallholder farmers in the sample in the sample kebeles before and after ISAP project ..................................... 29

Table 13. Average productivity of major crop and livestock products in the sample kebeles before and after ISAP project ..................................... 30

LIST OF FIGURES Figure 1. Use of agricultural extension service by male and female farmers in

sample kebeles before and after ISAP project ................................... 19

Figure 2. Use of yield increasing inputs in the selected kebeles before and after ISAP project ...................................................................................... 20

Figure 3. Access to improved market information service in sample kebeles before and after ISAP project ............................................................ 25

APPENDIX TABLE

iv

Appendix Table 1. FTC service participation through ISAP Project in 2007/8 EC Cropping season .................................................................. 36

Appendix Table 2. Average productivity of major livestock products in the sample kebeles before and after ISAP project .................................. 36

v

LIST OF ABBREVIATIONS AND ACRONYMS DAs Development Agents FTCs Farmer training centers FTC-MC FTC Management Committee GRD Grand Renaissance Dam HUNDEE Oromo Grass Root Development Initiatives (English equivalent

for the Oromifa meaning of HUNDEE) ISAP Increasing Smallholder Agricultural Productivity through

Improved Farmer Training Centers ISFM Integrated Soil Fertiliy Management MoANR Ministry of Agriculture and Natural Resources SMSs Subject Matter Specialists ORDA Organization for Rehabilitation and Development in Amhara P/FTCs Pastoral/Farmer training centers REST Relief Society of Tigray SEPDA South Ethiopia Peoples’ Development Association WoANR Woreda office of Agriculture and Natural Resources

1

EXECUTIVE SUMMARY

OVERVIEW OF INTERVENTION The Increasing Smallholder Agricultural Productivity through Improved Farmer Training Centers (ISAP) project was funded by Alliance for a Green Revolution in Africa (AGRA) and implemented by Oxfam America - Horn of Africa Region Office (OA-HARO). The project has been implemented in 62 kebeles in four (Oromia, Amhara, Tigray and SNNPRs) National Regional States of Ethiopia between 2014 and 2016. The project has been implemented in partnership with four local NGOs (HUNDEE, ORDA, SEPDA and REST) and government counterparts including the Ethiopian Ministry of Agriculture and Natural Resources (MoANR) with the respective decentralized office up to kebele level. The project is designed to strengthen Farmer Training Centers (FTCs) based agricultural extension services towards demand driven services for increased productivity and income of smallholder farmers in the project targeted areas.

OBJECTIVES OF THE EVALUATION The objective of this report is to document lessons that the ISAP project offers in increasing agricultural productivity and income of smallholder farmers of the target population. It assesses project achievements and impact that can be attributed to project interventions and the extent to whether key outcome indicators as specified and defined in the projects’ results framework have been met. This will enable OA-HARO to identify best practices, success stories, and lessons learned to guide future scale-up efforts by the MoANR and beyond.

EVALUATION METHODOLOGY This endline evaluation adopted the before and after evaluation approach, making use of the baseline data collected for this project. The evaluation used both quantitative and qualitative methods coupled with a desk review of secondary data to measure changes in conditions of beneficiaries in the FTCs before and after the ISAP project. The quantitative data for the evaluation was collected through a household survey, which interviewed 664 sampled households, of which 17.2 % were female headed households. The survey existed of a structured questionnaire that was adapted from the baseline questionnaire taking into consideration the evaluation questions as outline in the Terms of Reference (ToR). The household

2

survey was conducted through digital data collection, using KOBO Collect software installed on mobile phones. Qualitative data wee generated through Focus Group Discussions (FDGs), Key informant interviews (KIIs) and Case studies. Quantitative data were entered into the computer and analyzed using SPSS while the qualitative data (recorded KIIs & FGDs) were transcribed verbatim and analyzed thematically. The results from the household survey were analyzed across baseline study and end line study results, and substantiated and triangulated with the qualitative data.

SUMMARY OF FINDINGS/CONCLUSIONS Relevance of the project

Compliance of the project interventions with policies/strategies. The ISAP project was found to be consistent with and supportive of the country’s agriculture growth and transformation plans and strategies that clearly put strengthening agricultural extension services as the main gateways for increased smallholder agricultural productivity. The FTC needs assessment conducted before the inception of the project ensured that the project’s capacity building activities were based on a real need expressed by FTCs. The evaluation revealed that the sampled FTCs are upgraded from the ‘basic’ to the ‘intermediate’ FTC classification level.

Relevance to the priorities and needs of local communities. Farmers decide or consider using FTC services when they deem it important and to satisfy their demand for information and/or improved technologies. The different capacity building activities for FTCs and local institutions have enabled FTCs to provide demand-driven extension services to farmers. This has been demonstrated by a statistically significant increase in the use of FTCs based extension services both by male and female headed households over the project period.

Effectiveness

Improving the organizational and technical capacity of the FTCs and local

institutions in delivering demand-driven extension services. The evaluation demonstrated that the various demonstrations and field operations in FTCs, including the field days organized through the support of the ISAP project, enabled farmers to be actively involved in the evaluation of new agricultural technologies and practices adaptable to local conditions. This provided farmers with the opportunity to formulate their demand for improved technology while building the capacity of the local extension providers (the FTCs and DAs) towards client oriented extension services in the respective kebeles. This has been demonstrated by statistically significant increase in farmers’ use of yield increasing inputs such

3

as improved seed, inorganic fertilizer and chemical inputs over the ISAP project period.

Strengthening FTCs income generation capacity. Strengthening the capacity of FTCs to generate their own revenue from demonstrations and field operations in FTCs is considered as one of the major strategies to sustain the provision of improved and farmer-driven extension services in the respective kebeles. The ISAP project support has enabled the FTCs to increase their average annual income to 19,327 Birr, which is three fold of what the FTCs generated before the ISAP project. In this cropping season, the FTCs have profited 15,239 Birr (see Table 8). These income generation activities have also been a source of knowledge and technology (seeds) for the farmers.

Improving access to and use of integrated soil fertility management

technologies and practices. The ISAP project created awareness on integrated soil fertility management practices through farmer training, introducing demonstrations in FTCs and the use of ICT based innovation (Digital Green Videos). This has significantly improved access to and use of integrated soil fertility management practices in the sample kebeles. The endline evaluation revealed that this awareness raising process contributed to a statistically significant increase in use of at least one component of integrated soil fertility management practices in farmers’ fields, both in female and male headed households.

Enhancing capacity of FTCs and skills of DAs to improve access to market

information for farmers. Despite some delay in the provision of mobile phones and accessories to establish a market price board in FTCs, the project support has contributed to enhanced farmers’ access to improved market information compared to the situation before the start of the project.

Enhancing capacity of FTCs and skills of DAs to organize youth and women.

The project trained DAs and Subject Matter Specialists (SMSs) on group formation, business plan development and value chain approach, enabling them to assist in setting up and strengthening youth and women groups. In two sample kebeles in Amhara region, 18 saving and credit groups with a membership of 261 males and 17 females were set up. These groups have mobilized a capital of 64,582 Birr over the project period.

Identify and document best practices, success stories, and lessons learned

for use in future scale-up efforts. The project put in place a strong monitoring system to track project performance and provide opportunities for real time decision making on issues requiring corrective action. This enabled the project to timely track project performance, and report achievements and challenges in due course. Despite weak documentation and sharing of the project’s best practices,

4

project implementing partners and stakeholders such as WoANR have already identified lessons from the project implementation for use in future scale-up efforts.

Efficiency

Timeliness and quality of implementation. Project implementation was managed by the implementing partner. This has provided enough space for the implementing partners to meet project deadlines. Project inputs were delivered in a timely manner. However, government stakeholders in the sample woredas mentioned that there were some delays in the renovation of FTCs and that the quality of procured farm tools for the FTCs was sometimes poor.

Appropriateness of the implementation approach. The project adopted an innovative implementation approach by engaging local NGOs as implementing partners. This enabled the project to carry out activities in an efficient manner and according to plan. Government stakeholders were involved in the planning of the project activities.

Impact

Percentage increase in income of the smallholder farmers. There are positive signs of of change regarding the income from crop, livestock and off-farm activities of smallholder farmers as a consequence of ISAP interventions in the sample kebeles. This could relate to the increased access to demand driven agricultural extension services through the project, leading to increased use of improved agricultural technologies and practices.

Percentage increase in productivity of major crops and livestock. The evaluation findings demonstrate that there are signs of increases in the productivity of both crop and livestock of the local and improved commodities. The productivity of improved crop varieties (see table 13) demonstrated a statistically significant change over the project period. This again relates to enhanced access to demand driven extension services due to ISAP project.

Sustainability

The following prospects for sustainability were demonstrated by the evaluation:

FTCs and the respective local management structure are capacitated to self-sustain the FTC services.

The WoARN extension system is learning from the project and already aligned the project lessons into the regular extension program planning.

5

The successful youth and women groups have demonstrated that access to finance is an important input to engage in business and thereby create rural employment.

Lessons learnt

The major lessons that can be learned from the project and scaled up to the regular extension program include:

Strengthening organizational and technical capacity of FTCs and local institutions ise essential for providing demand driven extension services to farmers.

Strengthening FTCs income generation capacity is needed to enable self- sustainment of the FTC services.

MAIN RECOMMENDATIONS Based on the findings from the evaluation the following recommendations are suggested:

To put a stronger exit strategy in place to increase chances of sustainability of the project outputs but this requires a stronger exit strategy.

To strengthen access to improved market information and linkage services. To document and share project lessons for scaling-up.

6

INTRODUCTION

OVERVIEW OF CONTEXT AND PROJECT In view of valuable contributions that the agricultural extension sector can make to agricultural development, governments have traditionally taken the dominant role in its provision. Hence, usually a professional body of agricultural experts, often government employees, are involved in teaching improved methods of farming, demonstrating innovations, and helping farmers to organize and solve their problems. Over the past two decades, when many African governments have shown limited commitment to supporting smallholder agriculture and when many have neglected agricultural extension services in particular, the Government of Ethiopia has consistently invested in both. Ethiopia’s agricultural extension service is the third biggest public extension system in the world next to China and India.1

The Ethiopian agricultural extension system has focused on establishing kebele level pastoral/farmer training centers (P/FTCs) that are serviced by trained development agents (DAs). Currently, the government is progressing well in establishing one P/FTC in each Kebele to meet the national target of 18,000 FTCs, where about 12,500 P/FTCs were already established in 2015. The P/FTCs are established to promote knowledge and information sharing, to develope best-bet practices and to create self-sustained P/FTCs management systems with the ultimate aim of introducing smallholder farmers to modern agricultural technologies that can cope with changing climate. Government and non-government organizations have been supporting the P/FTCs to become local knowledge hubs for the kebeles. However, the established P/FTCs in the country are at different functionality levels and most are not in a position to deliver the expected demand driven agricultural extension services to farmers. Oxfam America responded to this gap with the support from Alliance for a Green Revolution in Africa (AGRA), and implemented a three-year project “Increasing Smallholder Agricultural Productivity through Improved Farmer Training Centers (ISAP)” from 2014-2016.

The project was designed with the goal of increasing the agricultural productivity and income of smallholder farmers (including women) in 62 kebeles in four National Regional States of Ethiopia (Oromia, Amhara, Tigray and SNNPRs). To achieve the project’s goal, the project set the following four objectives, each with their own set of outcomes, outputs, and activities:

1 Davis, K.; Swanson, B.; Amudavi, D.; Mekonnen, D.; Flohrs, A.; Riese, J.; Lamb, C. and Zerfu, E. (2010) In-Depth Assessment of the Public Agricultural Extension System of Ethiopia and Recommendations for Improvement. IFPRI

Discussion Paper 01041. Washington, DC: International Food Policy Research Institute (IFPRI).

7

1. To strengthen the capacity of 62 targeted FTCs (to become information & knowledge hubs) and local institutions to provide demand-based extension services to smallholder farmers with the following outcomes.

2. To increase smallholder farmer and DAs knowledge and skills, and use of integrated soil fertility management practices, improved seeds and good agronomic practices in the 62 kebeles.

3. To enhance the capacity of FTCs and skills of DAs to organize farmers, improve access to market information and post-harvest management.

4. To identify and document best practices, success stories, and lessons learned for use in future scale-up efforts.

The details of the project objectives, the expected outcomes and their indicators have been consolidated in the Key Performance Indicator Table. This evaluation based itself on the performance indicators as outlined in the Table and presented in Appendix 1.

MAJOR PROJECT INTERVENTIONS AND KEY STAKEHOLDERS The project designed a number of interventions to achieve the anticipated goal and objectives. The 62 target project intervention FTCs were divided over two phases, 31 each for the first and second phase of project implementation. Objective 1 established FTC Management Committees (FTC-MCs) and trained them for self-governance through establishing FTC based enterprise development, linking with demonstration and field operations. The targeted FTCs were to be renovated and achieve Intermediate FTC classification for training and demonstrations. This required the provision of improved farm tools for demonstration; development of FTC training materials including Digital Green (DG) videos, and capacity building support for DAs, SMSs and implementing NGOs.

Objective 2 established improved technologies and agronomic and soil fertility management practices demonstrations in FTCs, and promoted ICT based innovations called Digital Green to enhance technology dissemination and adoption. Objective 3 established market information centers in FTCs and provided DAs with mobile- and IT accessories for improved marketing & linkages. DAs and SMSs were trained on mobile-based market information, computer applications and post-harvest management thereby they trained farmers/farmer organizations/groups on the same. This objective also required the training of DAs and SMSs on business plan development and value chain approach so that they could introduce and support business & saving schemes for women and youth groups. Objective 4 worked towards providing real time performance information for learning and enhanced project effectiveness. This also required the identification and documentation of lessons learned from the project.

8

The project has been implemented in four Regional States of Ethiopia namely; Oromia, Amhara, SNNPRS, and Tigray through an innovative partnership model that engaged local NGOs as implementing partners. These NGOs offer unique technical expertise and specialized knowledge, and are likely more flexible to innovate and scale up lessons from the project. Accordingly, HUNDEE, ORDA, SEPDA and REST were the key stakeholders facilitating the project implementation in Oromia, Amhara, SNNPR and Tigray regions, respectively. In addition, the MoANR and its decentralized structure up to the level of FTCs were also key stakeholders who implemented the various project interventions. The project intervened to strengthen the capacity of FTCs and FTC-MCs to provide demand driven agricultural extension service to smallholder farmers. Thus, the primary change agents of the project were male and female farmers, and their representatives who managed the FTCs.

PURPOSE OF THE EVALUATION The objective of this report is to document the lessons that the ISAP project offers in increasing agricultural productivity and income of smallholder farmers (including women) in 62 kebeles in four National Regional States of Ethiopia. This final evaluation sampled four woredas in Amhara and Oromiya Regional States. The analysis constitutes an independent evaluation of identifying to what extent the ISAP project has achieved its stated objectives. The evaluation team adopted a mixed methods approach combining quantitative evidence based on household data collected in two rounds: from the baseline survey in 2014 and the end line survey in 2016. This was supplemented and complemented with qualitative evidence from key informant interviews and focus group discussions. This end-line evaluation was initiated by Oxfam America Horn of Africa Regional Office (OA-HARO) to generate information on the performance of the ISAP project implementation. This will enable OA-HARO to identify best-practices, success stories, and lessons learned to guide future scale-up efforts by the (MoANR) and beyond. As a result, OA-HARO commissioned SEGEL Research and Training Consultancy PLC to undertake the end line evaluation of the ISAP project. This report presents the evaluation design, findings, conclusions, and recommendations. The findings of the evaluation are expected to be utilized mainly by the MoANR and others who work towards establishing demand-driven agriculture extension service in Ethiopia and beyond.

9

EVALUATION DESIGN

The main purpose of this end-line evaluation, as outlined in the Terms of Reference for this evaluation (see Appendix 2), is to assess project achievements and impact that can be attributed to project interventions and to what extent key outcome indicators as specified and defined in the projects’ results framework have been met.

The specific objectives are:

1. Measure the changes in the productivity and income of smallholder farmers (male and female) before and after improving the capacity of FTCs;

2. Assess the project’s contribution to improved soil fertility of targeted smallholder farms;

3. Asses the changes in income of smallholder farmers (male and female) before and after improving the market information and value addition support by the project;

4. Measure the changes in the income of youth and women before and after the project intervention as well as changes in access to market and linkages; and business and/or saving schemes; and

5. Compile lessons learned from the project to be shared to stakeholders for future implementation and scale up.

KEY EVALUATION QUESTIONS The key evaluation questions for this end-line evaluation were guided by the objectives of the evaluation and the evaluation indicators/questions identified in the ToR. In addition, evaluation questions related to project relevance, effectiveness/impact, efficiency and sustainability were employed based on OECD-DAC2 criteria. This enabled the evaluation team to establish the project end line evaluation matrix. In addition, the evaluation took a close look at cross-cutting and visibility issues. Table 1 presents the ISAP project end-line evaluation matrix, mapping the intervention’s performance indicators, evaluation questions and evaluation criteria to data sources and collection approach. The matrix guided data collection and analysis.

2 Organization for Economic Cooperation and Development, Development Assistance Committee (OECD-DAC) Evaluation Quality Standards (2006).

10

Table 1: Increasing Smallholder Agricultural Productivity through Improved

Farmer Training Centers (ISAP) project end line evaluation matrix

Evaluation

Question

Definition of the evaluation

question

ISAP project end line

evaluation Sub-

questions

Data source and

collection

approach

Relevance:

Relevance evaluates the extent to

which the project is consistent with, and supportive of, the policy and project framework of the country’s agricultural extension policies and strategies.

The extent to which the project activity is suited to the priorities and policies of the target group, national policies, Oxfam America and AGRA.

Stakeholder’s participation and the extent to which the objectives of the project are consistent with the target beneficiaries' needs for demand-driven agricultural advisory services.

Is the intervention in line

with the priorities and policies of the people and communities it is working with?

Household

survey Key informant

interviews with project staff and targeted institutions.

FGD with selected project beneficiaries.

Project reports

Effectiveness/

Impact:

Measuring the extent to which the

project achieved results in terms of defined project performance indicators in the result framework.

Examining the reasons for any deviance in accomplishment of targets, including the process by which the change was brought about.

Evaluation of effectiveness is therefore linked to evaluation of impact and longer-term effects of the project. Impact refers to the total effects of the project interventions: positive and negative, intended and unintended.

To what extent are

targeted FTCs & local institutions delivering demand-driven extension services leading to increased productivity and/or income of smallholder farmers including women?

Has the project contributed and improved soil fertility of targeted smallholder farmers’ plots through the use of integrated soil fertility management (ISFM) technologies and best farming practices?

Has increased access to improved market information and linkages led to increased productivity and/or income?

Has income of youth & women groups in target FTCs increased through business and/or saving schemes designed by the project?

What are the reasons behind achieving or not achieving the objectives

Household

survey Key informant

interviews with project staff and targeted institutions.


Project reports

11

of the project/ and or the learning questions?

Evaluation

Question

Definition of the evaluation

question

ISAP project end line

evaluation Sub-questions

Data source and collection

approach

Efficiency:

It measures the extent to

which the FTC capacity building is done in a timely manner and whether the execution of these services were within the planned budget and cost efficient.

The extent to which interventions can be justified by results, taking alternatives into account.

Has the implementation adopted or taken into account new tools (innovation) in the transfer of knowledge?

Have the funds been used

economically, effectively and efficiently?

Household survey Key informant interviews

with project staff and targeted institutions.


Sustainability:

Sustainability assesses

whether the positive outcomes of the project continue after the conclusion of the project and considers the following factors: (1) The intended and unintended effects of the project implementation. (2) Efforts to replicate or scale up the objectives/activities and communities’ ownership of the socio-economic development efforts by the project.

Has the project documented

best practices and lessons during implementation period and shared with relevant stakeholders (MoANR and others) for future and wider scale-up efforts?

What are the prospects for sustainability and scaling up the project?

Household survey Key informant interviews

with project staff and targeted institutions.

Interviews with Woreda officials and SMSs

Discussion with project partners (HUNDEE and ORDA)

Cross-cutting

issues

Whether cross-cutting

issues - youth and women - are addressed by direct interventions or mainstreamed as a result of the project interventions..

What is the impact of the

project on the direct beneficiaries disaggregated by sex and age?

Has income of youth & women groups in target FTCs increased through business and/or saving schemes designed by the project?

Household survey; KII;

FDG; Project reports

12

EVALUATION TEAM SEGEL Research and Training Consulting PLC is an independent research, evaluation and training firm located in Addis Ababa, Ethiopia with the mission to help organizations deliver effective and quality services. The evaluation design, data collection, analysis and reporting steps in this evaluation employed participatory approaches that included guidance and input from OA-HARO staff and stakeholders. Three senior evaluators, four supervisors and twelve data collectors took part in the evaluation process.

EVALUATION METHODOLOGY As stipulated in the ToR, the approach to the evaluation was based on the before and after evaluation method, making use of the baseline data collected for the same project. The intention was to establish panel data where baseline sample respondents are interviewed again to measure the attribution of the project. SEGEL employed both quantitative and qualitative methods coupled with a desk review of secondary data to measure changes in conditions of beneficiaries in the FTCs before and after the ISAP project.

This end-line evaluation adopted the same sampling procedure employed for the baseline data. Accordingly, 664 sample households were selected through a random sampling method from four districts of Machakelle and Gozammin from Amhara, and Wuchale and Honkolo Wabe from Oromiya region. Oromia and Amhara regions were purposively selected since they represent more than 60% of project FTCs. Table 2 presents the sample FTCs and households for this evaluation. The baseline survey’s sample household list was used to ensure interviewing of the same households for this evaluation, with possible replacement of a comparable household in the case that the former interviewee was not available during the survey.

Table 2: Sample FTCs and households selected for the evaluation

Region Woredas FTCs Households

Oromia Wuchale Bidaru 89 Gimbichu 77

Honkolo Wabe Mechitu Lemman 92 Teji Wolkitee 88

Amhara Machakelle Yewela 157 Gozammin Yebo 161

Total 4 6 664

13

The quantitative data for the evaluation was collected through a household survey, which interviewed 664 sampled households, of which 17.2 % were female headed households. The survey instrument was a structured questionnaire that was adapted from the baseline questionnaire taking into consideration the project evaluation matrix as summarized in Table 1. The questionnaire was prepared in English and conducted using the local languages of the sample areas (Appendix 3). The household survey was conducted through digital data collection, using KOBO Collect software and applied on mobile phones.3 Mobile technologies facilitated the collection of the evaluation data by enabling the design of a survey with skipped patterns and variable checks and by facilitating direct uploading of the data to the platform server of the evaluator. In this way the data were available immediately for data cleaning and analysis. At field level, the enumerators were selected based on their familiarity with Android phones, level of education, familiarity to the study area and previous experience with data collection. The IT person and the lead evaluator gave orientation and training for data collectors.

Qualitative data are generated through Focus Group Discussions (FDGs), Key informant interviews (KII) and Case studies. FDGs were conducted with project beneficiary and non-beneficiary farmers and youth in the project areas involving both men and women of different age and wealth status. The KIIs were held with Woreda SMSs, Kebele DAs and FTC-MC representatives. Selected cases were interviewed to assess the relevance and benefits obtained from the implementation of the project activities. This helped in illuminating the outcomes and immediate impacts of the program and in drawing useful lessons. Respective checklists were developed to guide the qualitative data collection. Table 3 summarizes the number and type of qualitative data collection methods.

Table 3: Qualitative data collection tools and number of partciapnts

Type of data

collection tool

Number of qualitative data collected

Kebele/FTC Woreda Partner Total

FGD 12 12 KII 12 2 2 16 Case stories 4 4

Total 30 2 2 32

Quantitative data were entered into the computer and analyzed with SPSS while the qualitative data (recorded KII & FGD interviews) were transcribed verbatim and analyzed thematically. The results of the household survey were analyzed across baseline study and end line study results where we applied test statistics (t and chi-square test) to compare if these differences were statistically significant. The

3SEGEL has successfully carried out multiple surveys using smartphones and KOBO software, http://www.kobotoolbox.org/

http://www.kobotoolbox.org/

14

findings from each source were triangulated using the qualitative findings. Preliminary meetings with OA staff were conducted for briefing on the project activities, explanations on the ToR, and agreement on the evaluation methodology and sampling process. The qualitative data were collected until the required information reached saturation level and the quantitative data were cross referenced in the field and data cleaning was done at the same time. Preliminary results were briefed to OA staff and written down into a draft report. The final report is revised based on inputs and supportive evidence brought forward during the validation workshop, held with government representatives, OA-HARO and implementing partners to validate the evaluation methods, findings, lesson learnt and the way forwards.

Formal support from OA-HARO was obtained to get easy access to key stakeholders at field level. During fieldwork, respondents were explained about the purpose of the evaluation and informed consent was gained orally. For qualitative interviews approval was sought from respondents to note down findings from the interview using paper and pen. Interviewers took notes of key points and key quotes raised during the conversation and drew up a summary of the interview as soon as possible following the interview. Respondents were informed that they had the right to withdraw from the study at any time or choose not to respond to certain questions if they did not feel comfortable in answering them and interviews were conducted at times and places chosen by them. All information obtained during the course of the evaluation was dealt with in a confidential way.

EVALUATION LIMITATIONS The methodology adopted for this evaluation knew some limitations:

1. There were some difficulties in tracing baseline survey respondents. In cases where the former interviewee was not available to participate in this survey, a comparable household was sought for replacement..

2. The evaluation adopted the Before-and-After approach, which measures the changes in outcomes of interest from an intervention over time. It uses the outcome information of the pre-intervention (baseline data) situation and post-intervention (end line data) situation, and attributes the difference to the impact of the intervention. However, attribution seldom accurately measures impact given the inadequate measurement of the changes in without intervention scenario. Interpretation of the evaluation findings should therefore be done with caution.

3. The baseline and the end-line data collection were undertaken in different production seasons. This may affect the quality of the information as in both cases; responses are recalled based on respondents’ perceptions and information.

15

Although challenging, these limitations did not significantly affect the quality and findings of the evaluation.

16

EVALUATION FINDINGS

This section presents the findings of the evaluation. It begins by describing the sample of households based on selected socio-economic characteristics. The details of the evaluation findings are presented following the evaluation criteria: relevance, effectiveness, impact, efficiency, and sustainability.

1. DEMOGRAPHIC AND SOCIOECONOMIC PROFILE OF SAMPLE HOUSEHOLDS

The sex composition of the sampled households reveals that 82.8% of sampled households were male headed households and 17.2% were female headed households (Table 4). This composition is in line with the baseline survey result yet less than the national population mean of 26.1% in 2011. This is related to the difficulty of finding female headed households to participate in the survey in the project areas. With regard to marital status, from the total sample 91.9%, 1.2%, 2.0%, 4.4% and 0.6 % were married, single, divorced, widowed and separated, respectively. In Ethiopia, female headed households mainly constitute unmarried households (single, divorced, widowed and separated), and the high percentage of married households in the sample suggests that some of the female headed households interviewed during the baseline survey are now in a married status.

Table 4: Demographic characteristics of sampled households

Demographic variable N %

Sex composition – Male headed

550 82.8

Female headed 114 17.2

Marital status – Married

610 91.9

Single 8 1.2 Divorce 13 2.0

Widowed 29 4.4 Separated 4 0.6

Source: Survey Data (2016)

The socioeconomic profile of the sample of households is expressed with the average land size (ha) they owned. The result shows that the sampled households on average owned 1.39 ha (1.33 ha of rain fed and 0.06 ha irrigated). The majority (84.5%) of the land owned by the households is rain fed land and only 16.5 percent have irrigated land with an average of 0.39 ha. The result is similar to the majority of the land use in the country. The majority of the respondents practiced mixed

17

crop and livestock production where 96.8% owned livestock as a source of income during periods of draught. In addition, about a quarter of the households participated in some form of non-farm activities as additional source of livelihood.

Table 5: Sample households land ownership status

Household land size category N % Mean total land size (ha)

Rainfed land size 664 100 1.33 Irrigated land size 109 16.5 0.39 Total land size 664 - 1.39

Source: ISAP Survey – Endline

2. RELEVANCE/APPROPRIATENESS OF THE PROJECT INTERVENTIONS

2.1 Compliance of the project interventions with policies/strategies

The ISAP project support is found to be consistent with and supportive of the country’s agriculture growth and transformation plans and strategies. The current as well as the recently completed Agriculture Sector Growth and Transformation Plan clearly put strengthening agricultural extension services as a main gateway for increased smallholder agricultural productivity. Establishing and strengthening Farmers/ Pastoralist Training Centers (FTCs/PTCs) is the main strategy to provide demand driven extension services and the aim is to have 18,000 functioning P/FTCs by the end of 2021. To realize this, a strategy/guideline exists that support the transformation of FTCs into farmer-owned/farmer-driven entities and enterprises. Among others, the strategy sets criteria to determine the level of functioning of an FTC as basic, intermediate or advanced based on its infrastructural and service delivery standards. The ISAP project intervention was designed and implemented to transform the targeted 62 FTCs from their classification as ‘basic’ into ‘intermediate’ functioning level. The FTC needs assessments conducted before the inception of the project ensured that the project was responding to the needs of FCTs, namely renovating FTC, establishing demonstration plots and developing FTC income generation. Based on field observations and findings from the FGDs with FTC-MCs, DAs and SMSs; the sampled FTCs are upgraded to the ‘intermediate’ FTC classification level as a result of the ISAP project.

2.2 Project relevance to the priorities and needs of local communities

The project was designed to strengthen FTCs to provide demand driven agricultural extension services to the farmers in the kebele. One of the main principles for demand driven service delivery systems is that services are driven

18

by user demand.4 Farmers decide or consider using FTC services when they deem it important and to satisfy their demand for information and/or improved technology. This evaluation revealed that the various demonstrations and field operations in the FTCs, such as the field days organized through the support of the ISAP project, enabled farmers to be actively involved in the evaluation of new agricultural technologies and practices that are adaptable to the local conditions. This provided farmers with the opportunity to express their demand for improved technologies while building the capacity of the local extension providers (the FTCs and DAs) towards client oriented extension services in the respective kebeles. Results from the evaluation demonstrate statistically significant increases in the use the FTC based extension services both by male and female headed households over the project period (Table 6). The most common agricultural extension services were contact with extension agents through training and/demonstration at FTCs. The contact has significantly increased to 90.3% in this cropping season from 17.94% in 2014/15. The other services are DAs extension visits to farms and farmers visit to FTCs for advice, through which 42.9 and 24.3% of the users have been serviced.

Table 6. Use of agricultural extension service in sample kebeles before and

after ISAP project

Accessed FTC based extension

service

Baseline Endline Test statistics

(χ 2)

Significance

N % N %

Male Headed Households 450 81.8 506 92.0 25.06 *** Female Headed Households 74 64.9 96 84.2 11.19 *** All Sample 524 78.9 602 90.7 35.52 ***

Remark: ***, **, and * statistically significant at 1 %, 5 %, and 10 % probability level, respectively Source: ISAP Survey- Baseline and Endline In an attempt to enhance the participation of women and youth in the provision of agricultural extension services, the ISAP project involved a considerable number of female farmers - both household heads and house wives - in technology demonstration and field days (Appendix Table 1). This provided the opportunity to create linkages between extension workers and female farmers. The evaluation results demonstrated that there is a 23.6% increase in the percentage of female headed households using agricultural extension services over the project period (see Figure 1). There is diversity in the use of agricultural services across sample kebeles and respondent types. In kebeles where there was low level of female participation at baseline, such as in Mechitu Lemma, the project contributed significantly to female participation.

4 Chipeta, Sanne. Demand driven agricultural advisory services. NEUCHÂTEL GROUP 2006.

19

Figure 1: Use of agricultural extension service by male and female farmers

in sample kebeles before and after ISAP project

Source: ISAP Survey- Baseline and Endline

3. EFFECTIVENESS OF THE PROJECT INTERVENTIONS

Effectiveness of the project was assessed in terms of to what extent it achieved its objectives and outcomes. Wherever data allowed, comparison between baseline and end-line survey results is made to measure the project’s contribution on the selected effectiveness indicators.

3.1 Improving the organizational and technical capacity of the FTCs

and local institutions in delivering demand-driven extension

services

The capacity of the FTCs to deliver demand-driven agricultural extension services to farmers is measured in terms of farmers’ adoption of improved agricultural technologies. Farmers adopt a certain technology if he /she has enough awareness about the technology, is convinced about the advantage of the new technology over the existing ones, finds it affordable not just in terms of the absolute price, when it is available for purchase at that price when farmers have money in their pockets based on their cash-flow cycles, and when it is accessible when needed.

The ISAP project provided different trainings for DAs on how to design and establish demonstrations of improved agricultural technology and practices in the FTCs. For this, the relevant type and amount of inputs (improved seeds, fertilizer, etc.) were procured and delivered to FTCs for timely demonstrations. The evaluation demonstrated that the sample FTCs have undertaken demonstrations

0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%90.0%

100.0%

Yebo Yewula GimbichuGodonee

Bidaru MechituLemman

Taji Wolkitie All Sample

Female Baseline Female Endline Male Baseline Male Endline

20

and organized field days around the demonstration for at least two cropping seasons. In addition, the project involved local institutions in supporting client oriented services to farmers in the respective kebeles through establishing and/or strengthening FTC Management Committees. FTC-MCs were trained and provided with seed capital for the management of their FTCs. These have equipped the MCs with the motivation and enhanced technical & managerial ability to establish and strengthen income generation activities in the FTCs. The different field days organized around the demonstrations and income generation activities in the FTCs provided farmers with the opportunity to select improved agricultural technologies and practices that adapt well to their local conditions, in addition to creating awareness and convincing the advantage of these technologies and practices over existing local practices. Subsequently, farmers requested the regular extension system to provide access to the selected yield increasing inputs in next cropping season. These again provided farmers access to the design of local extension programs and thereby prospect for institutionalizing demand driven agricultural extension service in the respective kebeles. This has been demonstrated by increased farmers’ use of yield increasing inputs such as improved seed, inorganic fertilizer and chemical inputs over the ISAP project period (Figure 2).During the current cropping season ( 2016/17), major improved seeds applied constitute tef, wheat, maize, barely and faba bean, while the chemical inputs include pesticide, herbicides and fungicide.

Figure 2: Use of yield increasing inputs in the selected kebeles before and

after ISAP project


The changes in the use of these yield increasing inputs over the project period have been statistically significant for both male and female farmers, with slight differences on the use of improved seeds (Table 7). The use of improved varieties of Tef and wheat is equally important for both men and women farmers where both have significantly increased its use over the project period. The use of improved

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Yebo Yewula GimbichuGodonee

Bidaru MechituLemman

Taji Wolkitie All Sample

Baseline Inorganic Fertilizer Endline Inorganic FertilizerBaseline Improved Seed Endline Improved SeedBaseline Chemical Inputs Endline Chemical Inputs

21

variety of maize is only significant to female farmers given the limited use of the seeds at baseline. The results reveal that male farmers have been equally using improved varieties of maize before ISAP project. The use of improved varieties of barely by female farmer has not significantly increased over the project, probably due to limited experience of using improved barely varieties in the country in general and in the project areas.

Table 7: Use of yield increasing inputs by male and female farmers in

sample kebeles before and after ISAP project

Use of yield increasing inputs Baseline Endline Test

statistics

(χ 2)

Sig.

N % N %

Inorganic fertilizers Male farmers 521 95.1 544 98.9 13.86 *** Female Farmers

81 75.7 110 96.5 20.34 ***

Chemical inputs Male farmers 113 20.5 394 71.6 288.90 *** Female Farmers

5 4.4 50 43.9 48.52

***

Improved seeds Male farmers 243 44.2 334 60.7 30.19 *** Female Farmers

49 43.0 63 55.3 3.44 *

Improved variety of Tef Male farmers 70 12.7 106 19.3 8.77 *** Female Farmers

5 4.4 19 16.7 9.13

***

Improved variety of wheat Male farmers 120 21.8 246 44.7 65.01 *** Female Farmers

16 14.0 45 39.5 18.82

***

Improved variety of Maize Male farmers 179 32.5 196 35.6 1.17 NS Female Farmers

37 32.5 50 43.9 3.14

*

Improved Variety of Barely Male farmers 21 3.8 50 9.1 12.66 *** Female Farmers

1 0.9 1 0.9 0.00

NS

Improved Variety of

Fababean

Male farmers 5 0.9 4 0.7 0.11 NS Female Farmers

0 0.0 2 1.8 2.02

NS

Remark: ***, **, and * statistically significant at 1 %, 5 %, and 10 % probability level, respectively NS- Not statistically Significant Source: ISAP Survey- Baseline and Endline

3.2 Strengthening FTCs income generation capacity

Strengthening the capacity of FTCs to generate their own revenue from demonstrations and field operations is considered an important strategy in sustaining the provision of improved and farmer-driven extension services in the respective kebeles. The project baseline survey data indicated that the sampled FTCs already started generating revenues before the start of the project with an average annual income of 6,436 Birr from different income generation schemes. During the project lifespan, FTC-MCs were established and strengthened through trainings and technical support. The trainings provided incentives to the MCs and

22

enhanced their leadership and technical skills to manage the FTCs’ income generation schemes. The sample FTCs in Oromia region have been supported with a seed capital of 17,000.00 Birr per FTC and this required FTC-MCs to open a bank account. The bank based transaction ensured transparent financial management procedures of the FTC money. The support provided through the ISAP project enabled the FTCs to increase their average annual income to 19,327 Birr, which is three fold of what the FTCs used to generate before the ISAP project. In this cropping season, the FTCs made a profit of 15,239 Birr (Table 8). The FCTs were a source of knowledge and technology (seeds) for the farmers. The FTC-MCs require further capacity building support (in the form of training, exposure visits and seed capital) to improve the profitability of the enterprises and so to self-sustain the provision of demand driven services at FTCs.

Table 8: Sample FTCs annual income statement, Birr (2015/16)

Keble/FTC Income generation scheme Expense Revenue Profit

Yebo Crop production, beehives 2,000 5,510 3,510 Yewula Crop production, beehives, dairy 900 12,100 11,200 Gimbichu Godonee Crop production, fattening-cattle 7,000 18,000 11,000 Bidaru Crop production, saving (GRD Bond for 5,000

birr) 6,500 21,049 14,549

Mechitu Lemman Crop production 6,000 39,975 33,975 Taji Wolkitie Crop production 7,000 24, 200 17,200 Average 4,900 19,327 15,239

Source: ISAP Survey –Endline

3.3 Improving access and use of integrated soil fertility management

technologies and practices

The ISAP project created awareness on integrated soil fertility management practices through farmer training, introducing demonstrations in FTCs and the use of ICT based innovation (Digital Green Videos). DAs and SMSs were actively involved in the demonstrations, and relevant amount and types of inputs have been procured and delivered to FTCs for timely demonstrations. The trainings and demonstrations/field days involved a considerable number of female farmers, both heads of households and wives of households heads (Appendix Table 1). This has significantly improved access to and use of integrated soil fertility management practices in the sampled kebeles. The endline evaluation revealed that this awareness raising process contributed to a statistically significant increase in use of at least one component of integrated soil fertility management practices in farmers’ fields, both in female and male headed households (Table 9).

23

Table 9: Use of component of Integrated Soil Fertiliy Management (ISFM)

practcies by male and female farmers in in sample kebeles before and after

ISAP project

Applied at least a component of

ISFM practices


(χ 2)

Significance

N % N %

Male Headed Households 436 79.3 492 89.5 21.61 *** Female Headed Households 62 54.4 87 76.3 12.11 *** All Sample 498 75.0 579 87.2 32.23 ***

Remark: ***, **, and * statistically significant at 1 %, 5 %, and 10 % probability level, respectively Source: ISAP Survey- Baseline and Endline

Different soil fertility management practices are currently applied on farmers’ fields, and the majority of the users believe that the practices contribute to improved soil fertility. Crop rotation is the most significant soil fertility management practice applied by the majority of the sample farmers (69.9%) and covering 74.70% of the respondents land size (Table 8). In terms of number of farmers applying the practice, application of animal manure is the second most significant followed by soil conservation and compost application. Soil and water conservation (SWC) practices are the second most important in terms of land coverage. The SWC practices implemented on farm plots entails soil and stone bunds, terraces, check dams and trenches. They are among the sustainable land management practices being promoted nationally as part of the effort towards sustainable natural resource management to address the problem of land degradation that also contributes to soil fertility degradation in the country.

Table 10: Use of different soil fertlity management practcies in the sample

kebeles after ISAP project

Management practices

Applied the practice in 2016/17

cropping season Percentage of land

covered with this practice N %

Compost 192 28.9 25.48 Animal manure 302 45.5 30.48 Crop rotation 464 69.9 74.90 Soil and water conservation 236 35.5 54.94

Source: ISAP Survey- Endline In addition to the practices discussed above, application of sufficient quantities of inorganic fertilizer is considered as a solution to land degradation induced nitrogen constraint in Ethiopian soil. As discussed in section 3.1, there is a significant increase in the use of inorganic fertilizer in the sample of households. However, the “non-nitrogen” constraints on soil quality tend to decrease the efficiency of inorganic fertilizer in contributing to crop output and thereby decrease the effective demand for inorganic fertilizer. Thus, the use of large quantities of inorganic fertilizer only may not be enough to benefit from current yield gains offered by improved crop varieties because they farm on depleted soils that are non-

24

responsive to fertilizer application. A more holistic and integrated land management strategy is needed, which focuses on raising organic matter, moisture retention, and other forms of soil rehabilitation in addition to greater inorganic fertilizer use are preconditions for sustainable agricultural productivity growth in densely populated rainfed farming systems of Africa5. Of course, inorganic fertilizers are an entry point because the amount of organic fertilizers (livestock manure and crop residues) used currently is limited. There is consensus among agricultural experts that an integrated soil fertility management (ISFM) approach, one that combines organic and inorganic fertilizers, is the most practical way of sustainably and profitably restoring soil health.6 By definition, ISFM entails a set of soil fertility management practices that include the use of fertilizer, organic inputs and improved crop varieties combined with the knowledge on how to adapt these practices to local conditions. This evaluation demonstrated an increase in use of organic fertilizers which is promising for a reduction in the use of inorganic fertilizers in the future. In the longer term this can have positive implications on the use and application of ISFM.

3.4 Enhancing capacity of FTCs and skills of DAs to improve access

to market information for farmers

The ISAP project trained DAs and SMSs on the use of mobile-based market data collection and provided them with mobile phones and accessories for the purpose. Some FTCs created a market price board on which DAs posted up to date market information to improve farmers’ access to timely market information. This has contributed to enhanced farmers’ access to improved market information compared to the baseline situation (Figure 3). In addition to FTC market information boards, sampled farmers also accessed market information from the radio. The evaluation could not demonstrate whether all the sample kebeles used market price boards and the radio for accessing market information as there was a delay in the delivery of mobile phones to the respective kebeles. For example, in Teji Wolkitie and Mechitu Lemman kebeles (in Honqolla Wabe woreda), the mobile phones were not yet provided to DAs at the time of the evaluation and the price boards were not (yet) created in the respective FTCs.

5 Kihara, J., Nziguheba, G., Zingore, S., Coulibaly, A., Esilaba, A., Kabambe, V., Huising, J. (2016). Understanding variability in crop response to fertilizer and amendments in sub-Saharan Africa. Agriculture, Ecosystems and Environment, 229, 1–12. 6 Vanlauwe, B., Descheemaeker, K., Giller, K. E., Huising, J., Merckx, R., Nziguheba, G., & Zingore, S. (2015). Integrated soil fertility management in sub-Saharan Africa: unravelling local adaptation. SOIL, 1(1), 491–508.

25

Figure 3: Access to improved market information service in sample kebeles

before and after ISAP project


In addition to delays in the provision of mobile phones to the respective FTCs, limited use of the procured phones by DAs in accessing market information contributed to a lack of statistical significance difference in accessing improved market information over the project period (Table 9). Gimbichu Godonne and Bidaru sample FTCs in Wuchale woreda of Oromiya region are an exception. Here the DAs have effectively utilized the mobile apparatus to disseminate timely market information to farmers.

Table 11: Access to improved market information service by male and

female farmers in sample kebeles before and after ISAP project

Accessed improved market

information service


(χ 2)

Significance

N % N %

Male Headed Households 141 25.6 152 27.6 0.56 NS Female Headed Households 20 17.5 23 20.2 0.26 NS All Sample 161 24.2 175 26.4 0.78 NS


3.5 Enhancing capacity of FTCs and skills of DAs to organize youth

and women

The project trained DAs and SMSs on group formation, business plan development and value chain approach, enabling them to assist in creating and strengthening youth and women groups. In Yebo and Yewula kebeles in east Gojjam, located in Amhara region, the project trained facilitators to mobilize members and they established 18 saving and credit groups in the two kebeles. The groups received follow up training, in addition to technical and material (like registration books and saving boxes) support from the respective DAs and ORDA. This enabled them to develop regulations, legally register and open bank accounts. The 8 groups in

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

Yebo Yewula Gimbichu Godonee Bidaru Mechitu LemmanTaji Wolkitie All Sample

Female Baseline Female Endline Male Baseline Male Endline

26

Yebo together mobilized 124 male and 11 female members who regularly save every month and who have increased their capital to 28,652 Birr. In Yewula, the 10 groups together mobilized 137 male and 6 female members who saved about 37,930 Birr over the project period. The ISAP project selected 4 out of the 18 groups based on their performance in terms of organizing a large number of members and realizing high savings. The selected 4 groups were granted with financial support of 9250 Birr each and linked to the Amhara Credit and Saving Institution (ACSI). This enabled the groups to provide credit services to its members. The evaluation demonstrated that two groups, called Gojjam and Edigetelelimate, provided credit to their members to an amount of 15,000 Birr (3000 Birr to 5 members) and 6,000 Birr (to one member), respectively. Both groups have managed to collect all the repayments. Individuals who received the credit were engaged in income generation activities like animal production and fattening. This has contributed to increased household income that enabled the households to cover their children’s schooling expenses and unexpected expenses such as expenses for health problems.

In Oromia the experience of women and youth groups has been different. About 13 groups were organized in the two sampled woredas (Wuchale and Honqolo Wabe). The groups received training that enabled them to legally register as Micro and Small Enterprises and they were linked to the Oromiya Credit and Savings Share Company (OCSSCO) to benefit from seed capital support through the ISAP project. In Wuchale, all the 6 organized groups manage to withdraw the seed capital deposited at OCSSCO and distributed thus to 15 members. In Honqola Wabe only 4 out of the 7 organized groups managed to benefit from the seed capital. Individuals who received the financial support were operating independently and engaged in different income generation activities, and experienced changes in their livelihood situation (Box 1). The evaluation was not in a position to demonstrate that the groups are functioning both as places for learning as well as for saving purposes. This relates to challenges regarding

https://www.povertyactionlab.org/partners/oromiya-credit-and-savings-share-company-ocssco

https://www.povertyactionlab.org/partners/oromiya-credit-and-savings-share-company-ocssco

27

empowerment of women within a patriarchal system and limited follow up and technical support from the project and government implementing partners.

Box 1. Birhane Beyene’s story of becoming a member of a women’s group through the

ISAP project

Woizero Birhane Beyene is a 31 year old female, married and a mother of five. She lives in Badoye Gimbichu Kebele and is a member of the Abdi Gudina Womens’ development association that was established with support of the ISAP project. The association has 15 women female members and received 15, 000 Birr from the Oromiya Credit and Savings Share Company through the ISAP project in 2014. Each group member received 1000.00 Birr and engaged in individual income

generation activities. W/ro Birhane, as member of the group, received the 1000.00 Birr and managed to change her livelihood. Immediately after she received 1000.00 Birr, she started a small scale trade of vegetables in the local market of the kebele where she lives. As a result of her hard work, she generated profit from her businesses. With the help of her husband and children, she has been able to deposit 250 Birr each week (through a local saving scheme called EKUB) and collected 12,000 Birr in the

middle of 2016. She has allocated the 1000.00 Birr to a buy a bull for purposes of fattening and selling it. Now, she was able to sell the fattened bull for 8000 birr at the local market (picture). From the profit she made she bought an additional ox (picture) for five thousand Birr that she uses for draught power. Berhane’s household used to rent an ox for draught power services which decreased the family’s’ return from crop production. Birhanse still has some cash at hand with which she is doing business in the village. During our visit to her family, she told us that they are leading a happy life. She feels inspired to expand her business activities and engage in large sale cattle and sheep fattening and vegetables trade.

3.6 Identify and document best practices, success stories, and

lessons learned for use in future scale-up efforts

The project established a strong monitoring system to track project performance and so provide opportunities for real time decision making for issues requiring corrective action. For this, the project identified performance indicators before the start of the project and baseline data was collected accordingly. In addition, an indicator performance tracking table (IPTT) was established. Project staff members were trained on the different monitoring tools (IPTT) and reporting methods applied in the project. This enabled the project to timely track project performance, and report achievements and challenges in due course.

28

Project implementing partners and stakeholders such as WoANR have already identified lessons from the project implementation for use in future scale-up efforts. For example, WoANR identified capacity building of FTC-MCs, feeding feedback from field days into regular extension program planning and engaging youth in agriculture based on their needs as important lessons from the project. The evaluation could not demonstrate any evidence that the project’s best practices have been documented and shared.

4. EFFICIENCY

The efficiency of the ISAP recovery project was measured in terms of the quality of outputs achieved as a result of inputs (in terms of timeliness). This generally requires comparing alternative approaches to achieving an output, in order to see whether the most efficient approach has been used.

4.1 Timeliness and quality of implementation

The implementation of the project was managed by the implementing partner. This has greatly contributed to meeting project deadlines. Project inputs were delivered in a timely manner. This may be a result of the FTC needs assessment that was conducted before the start of the project. In addition, procurement of projects inputs was coordinated and handled by the implementation partner and that shortened the bureaucratic chain compared to when it had been handled by government offices. Demonstration inputs were available on time and demonstration plots were established at the right time. However, government stakeholders in the sampled woredas mentioned that there were some delays in the renovation of FTCs and that some of the farm tools that were procured were of poor quality.

4.2 Appropriateness of the implementation approach

The project adopted an innovative implementation approach by engaging local NGOs as implementing partners. This contributed to managing project activities in an efficient manner and according to plan. This has contributed positively to the timeliness of the project implementation (see section 4.1). Government stakeholders were involved in the planning of project activities, in particular in the identification, needs assessments and renovation of FTCs. However, these stakeholders have limited information about the details of the project implementation plan and budgets. Evidence on cost effective planning and implementation was not documented, even not at the level of governmental implementing partners.

29

5. IMPACT

Project impact was measured in terms of detecting any signs ofchange on the project impact indicators as presented in the project performance indicator table (Appendix 1). Accordingly, changes in the income of smallholder farmers and their productivity of crop and livestock commodities were searched for.

5.1 Percentage increase in income of the smallholder farmers

Crop and livestock production and off-farm activities usually form the main sources of income of smallholder farmers in Ethiopia in general and in the project intervention areas in particular. The evaluation found evidence that the income of smallholder farmers from crop, livestock and off-farm activities has increased as a result of ISAP interventions in the sampled kebeles (Table 12). This could relate to the increased access to demand driven agricultural extension services through the project and increased use of improved agricultural technologies and practices. However, it remains difficult to say whether the increases in income can be attributed completely to the project interventions given the short duration of the ISAP project in the sample kebeles and lack of impact measurement in non-ISAP kebeles.

Table 12.Changes in the income of smallholder farmers in the sample in the


Source of income Baseline Endline Test statistics

(t)

Significance

Crop production (Birr) 16,186.89 22,863.53 5.36 *** Livestock Production (Birr) 511.94 5,006.88 13.55 *** Nonfarm (Birr) 527.64 5,112.10 2.17 * Total household income

(Birr) 17,209.46 32,982.51 6.13 ***


5.2 Percentage increase in productivity of major crops and livestock

Changes in the productivity of crop and livestock commodities were measured by comparing the productivity at baseline and endline. The evaluation findings demonstrate that there are positive changes in the productivity of both crops and livestock, both the local and the improved ones (Table 13 and Appendix Table 2). The productivity of improved crop varieties demonstrated a statistically significant change over the project period. This again may be a result of enhanced access to demand driven extension services established by the ISAP project. Still it remains difficult to say whether all these changes in productivity can be attributed to the project interventions, given the short project span in the intervention areas and lack of similar impact measurements in non-ISAP sites. Also, there was favorable

30

rainfall across the highlands of Ethiopia in this production season7 and interpretation of changes in crop production therefore requires caution.

Table 13. Average productivity of major crop and livestock products in the


Crop/livestock commodity Productivity

measure


(t)

Significance

Local Tef Qt/ha 9.68 12.97 7.58 *** Improved Tef Qt/ha 20.71 21.58 30.16 *** Local Wheat Qt/ha 15.16 22.69 21.23 *** Improved Wheat Qt/ha 22.57 29.34 1.11 NS Local Maize Qt/ha 13.80 26.20 5.05 ** Improved Maize Qt/ha 22.62 30.51 6.32 ** Local Barely Qt/ha 13.10 16.01 21.89 *** Improved Barely Qt/ha 20.50 25.32 6.52 ** Local Faba Bean Qt/ha 7.14 10.27 9.94 ** Improved Faba Bean Qt/ha 20.40 25.32 22.62 ***


6. SUSTAINABILITY

Sustainability measures the prospect for ownership and scaling-up of the project achievements in to the regular extension program planning in the targeted woredas. The evaluation demonstrated the following prospects for sustainability.

FTCs and the respective local management structure are capacitated to self-

sustain the FTC services. Institutionalizing community owned management structures at FTCs could enable them to continue to engage in income generation schemes that could self-sustain the FTCs’ services. The established livestock oriented demonstrations in FTCs are one source of sustainable income for the FTCs. However, this calls for an appropriate exit strategy given that the majority of the FTC-MCs in the sample kebeles are being restructured with new members who are not trained on the FTC guidelines and procedures.

The WoARN extension system is learning from the project and already

feeding the project lessons into the regular extension program planning. The government stakeholders are actively engaged in implementing the majority of the project activities. They organized the various demonstrations and organized field days. Farmers, both females and males, who participated in the field days evaluated the technologies shown in the demonstration and selected the most adaptable ones for the next cropping season. Based on these feedbacks, WoANR

7 GIEWS, 2016. GIEWS Coutry Brief Ethiopia. Global information and early warning system on food and agriculture. FAO , November 22, 2016

31

ensured that the selected technologies are taken up in the regular extension program planning in the woredas. This reveals there are positive prospects for demand driven agricultural extension services. In the meantime, documenting these lessons and preparing a policy brief is of key importance to enable scale-up into the national extension system.

The successful youth and women groups have demonstrated that access to

finance is an important condition for engaging in business and thereby

create rural employment. Given the increasing number of unemployed youth and women in rural Ethiopia, the project demonstrated that providing access to finance and engaging youth and women in enterprises based on their own interest is key to creating rural employment. The sustainability of the groups depends on an appropriate exist strategy to engage them in more profitable enterprises demanding additional financial support.

7. LESSONS LEARNT

The major lessons that can be learned from the project and scaled up to the regular extension program are the following:

Strengthening organizational and technical capacity of FTCs and local

institutions is essential for providing demand driven extension services to

farmers. The ISAP project demonstrated that enhancing the capacity of FTCs through training DAs and establishing/strengthening FTC-MCs enabled the FTCs to organize demonstrations of improved technologies and organize field days, thereby involving end-users in technology selection and evaluation. The demonstrations have served as sources of knowledge about new technologies, transforming the FTCs into local knowledge hubs. In addition, FTCs needed the input from farmers to determine which technology should become part of the regular extension system, so that the the extension services are driven by demand, one of the characteristics of demand driven extension services.

Strengthening FTCs income generation capacity is needed to self-sustain

the FTC services. One of the major limitations of FTCs identified by the needs assessment prior to the start of the project was lack of operating expenses to establish demonstrations and limited capacity to link trainings with demonstrations. Establishing and/or strengthening local community members for the management of FTCs has enabled FTCs to engage in income generation activities. Engaging FTCs in income generation could enhance sustainable income for the FTCs and thereby create possibilities for self-sustainment of the FTC services. This also requires farmers to contribute resources for the financing of the services. This is

32

an innovative financing mechanism for demand driven agricultural extension services where users co-finance the service.

33

CONCLUSIONS AND

RECOMMENDATIONS

CONCLUSION

The Increasing Smallholder Agricultural Productivity through Improved Farmer Training Centers (ISAP) project is designed to strengthen FTCs to provide demand driven agricultural extension services to the farmers in the kebele. The project interventions have proofed to be a successful strategy to provide demand driven agricultural extension services to end-users. The project is consistent with the national policies/strategies/plans and community priorities. The various demonstrations and field days organized with the support of project enabled farmers to be actively involved in evaluating to what extent certain technologies are adaptable to local conditions. This has transformed FTCs into local knowledge hubs and enhanced their capacity to provide extension services that are based on the actual needs of the respective kebeles. This was demonstrated by increased use of FTC services by farmers, and thereby increased use of improved agricultural technologies and soil fertility management practices. Establishing and strengthening FTC-MCs and so improving the organizational and technical capacity of FTCs also contributed in this regard. The project interventions have also contributed to increased access to improved market information in areas where DAs are using the mobile phones and accessories for the purpose. The evaluation revealed that there are increases in the productivity of major crops and livestock and thereby in the income of smallholder farmers after ISAP interventions in the sampled kebeles. The interpretation of these changes requires caution given the short duration of the project interventions in the sampled kebeles.

The project implementation approach involving local NGOs contributed to effective delivery of the project outcomes. The project used the existing government and community level structures and focused on building capacities in order to sustain the activities. More efforts are required to document and share the project lessons for scaling them up into the regular extension program planning.

34

RECOMMENDATIONS

Based on the findings from the evaluation the following recommendations are suggested:

• There is a positive prospect for sustainability of the project outputs but

this requires a stronger exit strategy. The role of FTC-MCs is instrumental in self-sustaining the FTC services. The high DA and kebele leadership turnover limits the prospect for sustainability of the FTC-MC activities. An appropriate exist strategy that involves government stakeholders is required in this regards.

• Strengthening access to improved market information and linkage

services. The project attempted to improve access to timely market information. However, further efforts are required to strengthen farmers’ access to improved value chain and linkages services through establishing market oriented agricultural extension services.

• Document and share project lessons for scaling-up. Government implementing partners have already identified some lessons but documentation of lessons is limited. Documentation and sharing of these lessons will facilitate the scaling up of the lessons into the regular extension program. Efforts are required to align the project lessons on providing access to improved market information and demand-driven agricultural extension services to marry into market oriented agricultural extension services.

35

REFERENCES

Chianca, T. (2008). The OECD/DAC criteria for international development evaluations: an assessment and ideas for improvement. Journal of Multidisciplinary Evaluation, 5(9), 41-51.

Davis, K.; Swanson, B.; Amudavi, D.; Mekonnen, D.; Flohrs, A.; Riese, J.; Lamb, C. and Zerfu, E. (2010) In-Depth Assessment of the Public Agricultural

Extension System of Ethiopia and Recommendations for Improvement. IFPRI Discussion Paper 01041. Washington, DC: International Food Policy Research Institute (IFPRI).

Kihara, J., Nziguheba, G., Zingore, S., Coulibaly, A., Esilaba, A., Kabambe, V., Huising, J. (2016). Understanding variability in crop response to fertilizer and amendments in sub-Saharan Africa. Agriculture, Ecosystems and Environment, 229, 1–12.

Vanlauwe, B., Descheemaeker, K., Giller, K. E., Huising, J., Merckx, R., Nziguheba, G., & Zingore, S. (2015). Integrated soil fertility management in sub-Saharan Africa: unravelling local adaptation. SOIL, 1(1), 491–508.

36

Appendix Table

Appendix Table 1: FTC service participation through ISAP Project in 2007/8 EC Cropping season

FTC activities Yebo

Yewula

Gimbichu Godonee

Bidaru

Mechitu Lemman

Taji Wolkitie

Training Crop production

Male 101 461 185 215 40 57 Female

10 39 87 45 20 3

Youth 5 0 45 75 0 Livestock production

Male 38 59 125 178 - - Female

3 7 60 72

Youth 0 25 68 Integrated soil fertility management practices

Male 326 59 325 152 - Female

38 4 90 74

Youth 3 0 70 48 Demonstration and/or Field days

Crop production

Male 61 445 250 185 135 Female

24 34 75 35 26

Youth 5 0 45 89 - Livestock production

Male 24 27 150 145 Female

14 6 72 85

Youth 0 0 30 28 Integrated soil fertility management practices

Male 26 30 250 125 Female

2 4 75 82

Youth 2 45 45 Source: ISAP Survey- Endline

Appendix Table 2: Average productivity of major livestock products in the


Kebele

Milk (lt/cow/month) Egg

(No./chicken/month)

Honey (Kg/beehive/year)

Baseline

(local and

crossbred

cows)

Endline

(crossbred

cow )

Endline

(local

cow) Baseline Endline

Improved

+local

beehives

honey

(improved

beehives)

Honey

(local

beehives)

Yebo 72.00 18.19 15.58 17.00 15.22 Yewula 43.57 21.73 15.05 22.50 46.91 Amhara 8.33 6.48 7.42

Gimbichu Godonee

161.50 55.26 17.80 5.00

Bidaru 120.00 43.98 18.42 10.00 Mechitu Lemman

130.00 48.28 16.84 9.50

Taji Wolkitie

116.68 41.00 19.27 22.80

Oromia 5.03 6.24 2.00


37

38

APPENDIX 1

Key Performance Indicator Table

Program Logic / Result Chain Indicators Definition Level of Disaggregation

Means of Verification

Frequency of Measurement

Baseline Target Cumulative

Target Yr. 1 Yr. 2 Yr.3

Project Goal: To increase agricultural productivity and income of smallholder farmers (including women) in 60 kebeles in four National Regional States of Ethiopia.

Project Impact: Enhanced agricultural productivity and increased income of smallholder farmers

Percentage increase in income of the smallholder farmers

Increase in income of female & male headed households those involved in this project & driving their livelihood from Ag sector

Location, household, sex

Household survey Baseline and End–line evaluation

TBD 20% 25% 30%

20-30% increase for

about 80% of target farmers

Percentage increase of productivity and profitability of crops and livestock

Productivity implies the increased amount of product per plot of land or per livestock Profitability implies revenue/income from the products produced and/or decreased production costs

Location, Crop/Livestock, Sex

Household survey Baseline and End–line evaluation

TBD 20% 25% 30%

20-30% increase for

about 80% of target farmers

Objective 1: To strengthen the capacity of 60 targeted FTCs (to become information & knowledge centers) and local institutions to provide demand-based extension services to smallholder farmers.

Outcome 1: Improved organizational and technical capacity of the FTCs & local institutions in delivering demand-driven extension services to smallholder farmers

Increase in number of better technologies and best practices adopted by smallholders

Better technologies/best practices refer to agricultural technologies/practices that fit to the local contexts.

Location, sex, technology

On-going data collection, Sample survey

On annual basis, Baseline & End-line evaluation

TBD 4 4 2 10

Output 1.1: FTC MC members are capacitated and FTC management system is strengthened

Number of FTCs with functional leadership

Not only establishment, functionality also matters

Location Periodic reports, On-going data collection

On quarterly basis

0 29 31 0 60

Process/Activities: 1.1.1: Establish FTC Management Committees (MCs) for pilot FTCs

Number of FTCs for which MCs established

MC is in charge of coordinating activities at FTC level and managing FTC resources/facilities etc.


On quarterly basis

0 29 31 0 60

39






1.1.2: Train FTC MCs for self-governance by linking with demonstrations, field practices, and business development

Number of MC members trained

MC is formed from 7 members derived from the community. Hence a total of 420 MC members will be trained during the 3 years

Location, Sex

Periodic reports, On-going data collection

On quarterly basis

0 203 217 0 420

(7 MC members/FTC)

Output 1.2: 60 FTCs upgraded to intermediate functionality level

FTCs meeting Intermediate level classification

There are 3 FTC levels (basic, intermediate & advanced) with defined sets of facilities/services


On quarterly basis

0 29 31 0 60

Process/Activities: 1.2.1: Renovate FTCs to achieve Intermediate FTC classification for training and demonstrations

Number of FTCs for which facilities are renovated

Facilities = FTC buildings, classrooms, demo plots, water schemes, etc. (as appropriate for each FTC)

Location, Facilities


On quarterly basis

0 29 31 0 60

1.2.2: Provide improved farm tools for demonstration

Sets of farm tools provided to FTCs

Farm tools are tools used to do demonstrations at FTCs

Location, List of tools


On quarterly basis

0 29 31 0 60

(one set/FTC)

1.2.3: Develop/provide FTC training materials: Duplicate training manuals & guidelines; Produce & disseminate Digital Green (DG) videos; Produce leaflets & posters

Number of materials developed/duplicated

Manuals and guidelines already developed by OA for similar project

List of materials developed/ provided


On quarterly basis

0 21 (7 from each)

24 (8 from each)

0

45 (15 manuals & guidelines,

15 videos, and 15

leaflets/posters)

1.2.4: Train SMSs, DAs, and FTC MC members on performance-based MEL system (guidelines, tools, & formats)

Number of participants

Six people (3 DAs & 3 SMSs) will participate per FTC

Location, Sex Periodic reports On quarterly basis

NA 174 186 0 360

Output 1.3: Local NGOs are capacitated and in a better position to manage/coordinate provision of demand-based extension services

Number of local NGO staff trained/capacitated

It is to train staff of partner local NGOs implementing the project. A total of 92 staffs will be trained on different issues during the 3 year period

Location/ NGO, Sex

Periodic reports, On-going monitoring

On quarterly basis

0 60 32 0 92

Processes/Activities: 1.3.1: Train staff of local NGOs on project management, financial management, grant report preparation, FTC MC establishment, and value chain approaches


60 local NGO staffs will be trained during project life

Location/ NGO, Sex Periodic reports On quarterly basis

0 40 20 0 60

40






1.3.2: Train staff of local NGOs on MEL tools that ensure provision of demand-based extension services through amplifying farmer’s voice


32 local NGO staffs will be trained

Location/ NGO, Sex Periodic reports On quarterly basis

0 20 12 0 32

Output 1.4: Exit strategy is clearly defined to ensure sustainability and future scale up

Exit strategy document

Exit strategy is about setting the strategy on how the project will be phased out by OA ensuring sustainability of project achievements

NA Final project report

At end of the project

NA 0 0 1 1

Processes/Activities: 1.4.1: Develop an exit strategy to ensure sustainability and future scale up

Exit strategy document

Same as above NA Final project report

At end of the project

NA 0 0 1 1

Outcome 2: FTCs generate revenue on sustainable basis to sustain the provision of improved and farmer-driven extension services

Number of FTCs generating and using revenue on continuous basis

Revenue generated by FTCs comes from selling of products obtained from demonstration of technologies and better practices

Location On-going data collection, Sample survey

On quarterly basis, Baseline & End-line evaluation

0 29 31 0 60

Output 2.1: Enterprise-based revenue generation schemes introduced & operational in the FTCs

Number of FTCs with revenue generation schemes employed on continuous basis

Revenue generation schemes are schemes introduced in FTCs to demonstrate technologies / practices and generate income from selling products

Location, Scheme Periodic reports, On-going data collection

On quarterly basis

0 29 31 0 60

Process/Activities: 2.1.1: Introduce enterprise-based revenue generation schemes for FTCs, focusing on simple and rewarding activities + activities listed above)

Number of FTCs generating revenue

Same as above Location, Scheme Periodic reports, On-going data collection

On quarterly basis

TBD 29 31 0 60

Objective 2: To increase smallholder farmer and DA knowledge and skills, and use of ISFM, improved seeds, and good agronomic practices in the 60 kebeles.

Outcome 3: Increased use of ISFM technologies and best farming practices by smallholder farmers in 60 kebeles in Ethiopia.

Percentage of targeted smallholders using ISFM technologies & best farming practices

ISFM – technologies are technologies used to improve soil fertility (soil management & appropriate application of organic/inorganic fertilizers…)

Location, Type of Technology, Sex

On-going data collection, Sample survey


TBD 25% 50% 75% 75%

41






Output 3.1: Farmers have access to demonstrations at FTC level thereby enhancing their knowledge of better technologies and practices leading to adoption of such technologies and practices

Percentage of target farmers that attended FTC level demonstrations

Demonstration at FTCs is done by DAs to teach farmers on new/improved technologies & practices

Location, Technology, Sex


On quarterly basis

0 48.3%

51.7%

0 100%

Process/Activities: 3.1.1: Demonstrate improved technologies and agronomic and soil fertility management practices on FTC demo plots

Number of FTC that have demonstrated innovations/good practices

Innovations/good practices refers to better technologies & practices on crops & soil fertility management

Location, Technology/ Practice

Periodic reports, Field observation

On quarterly basis

0 29 31 0 60

3.1.2: Promote ICT based innovations (DG) to enhance technology dissemination and adoption - Supply DG video apparatus - Train DAs on computer application

and DG technology

Number of FTCs where DG technology is introduced

DG technology is one of extension approaches – producing videos on best practices & disseminate to farmers. It will be introduced in 4 FTCs


On quarterly basis

0 4 0 0 4

Output 3.2: Select farmers get access to agricultural inputs to employ and demonstrate new technologies/practices in their own plots

Number of smallholder farmers that get access to inputs for demonstration purposes

Agricultural inputs refer to improved seeds, fertilizer, etc. 25 farmers per FTC will have access to inputs for demonstration purposes in their own plots

Location, Input type & quantity, Sex


On quarterly basis

0 700 800 0 1500

Process/Activities: 3.2.1: Provide inputs for demonstrations (improved seeds, fertilizer, etc.)

Number of FTCs supplied with inputs

FTCs will use inputs for demonstration and will also distribute to select farmers for adoption/demonstration on their own plots

Location, Input Type & Quantity


On quarterly basis

0 29 31 0 60

Objective 3: To enhance capacity of FTCs and skills of DAs to organize farmers, improve access to market information and post-harvest management

Outcome 4: Strengthened farmer organizations and increased access to markets by smallholders in 60 kebeles.

Percentage of targeted smallholder farmers get access to good prices for their products

Good price is maximum price available in the markets accessible to smallholders

Location, Sex On-going data collection, Sample survey


0 48.3%

51.7%

0 100%

42






Output 4.1: Smallholder farmers get access to improved market linkages and information

Percentage of targeted smallholder farmers get access to market information

Market information refers to the information (regularly revised) on prices of agricultural products in markets can be accessed by smallholder farmers.

Location, Sex Periodic reports, On-going data collection

On quarterly basis

0 48.3%

51.7%

0 100%

Process/Activities: 4.1.1: Establish market information centers and provide them with mobile devices & accessories for improved marketing & linkages

Number of information centers established

Information centers are centers where latest & dependable market information is available via mobile (using SMS)


On quarterly basis

0 29 31 0 60

4.1.2: Train DAs and SMSs on mobile-based market information


360 DAs & SMSs (6/FTC) will be trained. SMSs stand for subject matter specialists.


On quarterly basis

0 174 186 0 360

4.1.3: Organize community trade fairs Number of trade fairs organized

Trade fairs refers to the events where farmers/producers and customers meet (for fair price)


On quarterly basis

0 4 4 4 16

4.1.4: Establish market price boards at target FTCs to provide latest price information

Number of FTCs with market price boards & providing required information

Price board is the notice board where latest & reliable prices information on agricultural products are displayed


On quarterly basis

0 29 31 0 60

4.1.5: Train farmers/farmer organizations/ groups on market information and post-harvest management

Number of farmers (group members) trained

Group members refers to farmers those who are member of groups organized to get access to or provide agriculture related services


On quarterly basis

0 290 310 0 600 (10/FTC)

Outcome 5: Increased income (from business/saving schemes) of youth & women in 10 kebeles in Ethiopia

Number of groups earning from the business & saving schemes

Earning refers to making some profit from the business and saving schemes

Location, Scheme, Sex Periodic reports, sample survey


0 4 6 0 10

43






Output 5.1: Business & saving schemes introduced & become operational by youth & women groups

Number of functioning business & saving schemes

Business & saving schemes are schemes introduced to promote business incubation by youths & women (see also 5.1.2. below)

Location, Type of business


On quarterly basis

0 4 6 0 10

Process/Activities: 5.1.1: Train DAs and SMSs on business plan development and value chain approach (enabling them assist youth & women groups)

Number of DAs & SMSs trained

A total of 360 DAs(180) & SMSs (180) will be trained during the first 2 years


On quarterly basis

0 174 186 0 360

5.1.2: Introduce business & saving schemes for youth & women groups:

- Organize youth & women groups - Train the groups on business

development & management - Identify feasible businesses and

provide loan guarantee/seed money

Number of groups organised & engaged in business incubation

10 groups (6 youth & 4 women) will be organized in 10 kebeles from the 60 project kebeles. Each group will have 10 members

Location, Scheme, Sex Periodic reports, On-going data collection

On quarterly basis

0 4 6 0 10

Objective 4: To identify and document best practices, success stories, and lessons learned for use in future scale-up efforts. Outcome 6: Best practices & lessons learned during this project are applied to future scale-up efforts by the MoA & others

Best practices taken up by the wider system

Best Practices = any kind of extension-related practice learned from this project

Location Sample survey End-line evaluation

NA 2 2 2 6

Output 6.1: Baseline data established for major indicators

Report generated from baseline study

This report establishes baseline data for each indicator against which project achievement will be tracked

Location Validation workshop report

On quarterly basis

NA 1 0 0 1

Process/Activities 6.1.1: Conduct baseline study for the project

Baseline study report generated on time

See above Location Validation workshop report

Once during project life (1st year)

NA 1 0 0 1

Output 6.2: Issues identified and timely corrective measures are taken

Smooth project implementation / completion of project activities on time achieved

Taking timely corrective measures helps to overcome problems related to project implementation thereby ensuring timely completion

Location Periodic report, Bi-annual review workshop report

On quarterly basis

NA Continuous

44






Process/Activities 6.2.1: Monitor implementation status against plan and make on-time corrections as necessary – applying performance-based MEL system

Completion of project activities as per the action plan

Completion of project interventions meeting the required quality and timeline

Location Monitoring report, Periodic reports

On quarterly basis

NA Continuous

Output 6.3: Lessons learned and best practices from the project are documented and shared

Number of lessons, policy briefs and documents documented and shared to key stakeholders

See below (under 6.3.1, 6.3.2, 6.3.3)

Location, Technology, etc.

Periodic report, Annual review report, special reports

On annual basis, at third year of the project

NA 2

(1+1) 2

(1+1)

11 (4+2+5

) 15 (6+4+5)

Process/Activities 6.3.1: Identify and document lessons learned from the project

Number of lessons

Lesson implies any kind of extension related practice (positive or negative) learned from this project

Location, Technology, etc.

Periodic reports On quarterly basis

NA 1 1 4 6

6.3.2: Prepare and communicate policy briefs to the MoA

Number of policy briefs prepared and communicated

Policy brief is evidence based communication to policy makers to influence the policy making process in the sector

NA Special report on best practices

Annual NA 1 1 2 4

6.3.3: Documentation of project materials and products for wider use

Number of documentation

Documentation implies giving a shape to materials (evaluation (baseline and end line) reports, annual reports, capacity building materials, etc.) for wider use or scaling up of the project

NA Special reports on the documentation

At third year of the project

NA 0 0 5 5

Output 6.4: End-line evaluation conducted

End-line evaluation report

Final evaluation report that will discuss project achievements against set indicators/objectives, describe lessons learned, and recommend next steps

Location End-line report At end of the project

NA 0 0 1 1

Process/Activities 6.4.1: Conduct final/end-line evaluation

Final/end-line evaluation report document

Same as above Location End-line report At end of the project

NA 0 0 1 1

Notes: 1) NA – Not Applicable or Not Available (at this time); 2) TBD – To Be Defined (following baseline study); 3) Smallholders is used interchangeably with Farmers (on an average there are 1,000 farming households per

kebele)

45

APPENDIX 2

Terms of Reference 1. Background, Purpose and Audience Oxfam America (OA) is an international development agency that operates with a vision of creating lasting solutions to poverty, hunger, and injustice by working with others. To achieve its mission and objectives in Ethiopia, Oxfam America is implementing a number of development programs/projects as well as humanitarian interventions for the decades.

Oxfam America with the support from Alliance for a Green Revolution in Africa (AGRA), has piloted and implemented a three-year project “Increasing Smallholder Agricultural Productivity through Improved Farmer Training Centers (ISAP)” from 2014-2016. The project is intended to establish and pilot a new and replicable approach to transforming Ethiopian agriculture by improving the capacities of 60 pilot Farmers Training Centers (FTCs), Development Agents (DAs) assigned in those FTCs and Subject Matter Specialists (SMS) providing backups to DA (See annex 2 for the details of FTC’s distributions per regions). The project has been piloted in four Regional States (Oromia, Amhara, SNNP and Tigray) and the result is expected to significantly increase productivity and income of 30,000 smallholder farmers. The project has also an objective to document best-practices, success stories, and lessons learned to guide future scale-up efforts by the Ethiopian Ministry of Agriculture and Natural Resource (MoANR) and others.

Key project activities, among other includes: establishing FTC Management Committees (MCs); training FTC MCs in self-governance; providing improved seeds and farm tools for demonstration; developing and/or providing FTC training materials; training farmers and DAs on critical knowledge and skill gaps; demonstrating improved technologies and agronomic and integrated soil fertility management practices on FTC demo plots; promoting ICT-based innovations Digital Green (DG) to enhance technology dissemination and adoption; training DAs on computer applications and DG technology; establishing market information centers and providing mobile devices and accessories for improved marketing and linkages; establishing market price boards at target FTCs to provide latest price information; organizing community trade fairs; training DAs/SMSs on mobile-based market information, market chains, rural enterprise development, and assessing risks based on product selection; and documenting key lessons and best practices for latter scale up.

Baseline study has been conducted in June – July, 2014 by means of a household survey, focus group discussion and key informant interview. The survey reached 662 households with a random sampling method from four districts of Machakelle and Gozammin from Amhara, and Wuchale and Honkolo Wabe from Oromiya region. Oromia and Amhara regions are purposively selected since they represent more than 60% of project FTCs. Of the FTCs participated in the study are Yewela, Yebo, Gimbichu, Bidaru, Mechitu Lemman and Teji Wolkitee (see the detail on annex 1). This study can be benchmarked while deciding on evaluation methodology and to compare the programs impact through analysis of before-and- after approaches.

MoA (federal, regional, and project target Woredas and FTC), ATA, AGRA, and other stakeholders which are involved in similar intervention could be audiences for this study. The study will provide lessons and could be used as guidance for future project design and up scaling.

46

2. Specific Object and Objective of the Evaluation

The objects of this evaluation are Farmers Training Centers (FTCs), DAs working at these centers, FTC MCs and farmers residing in and accessing extension services from the FTCs, and women and youth groups organized and targeted by the project. Besides, SMSs who are playing a supervisory and supporting role to DAs from Woreda agricultural office could also be the target of this evaluation as they have been actively participating in the implementation as well as capacity building efforts backstopping DAs in various forms. They are change agents for sharing and scaling-up best practice and lessons to other kebeles and/or woredas. As the project also involves capacity building of partner organizations (government and NGO partners), the study can also address partners to observe whether partnership has been strengthened.

The general objective of this end line evaluation is to assess project achievements and impact that can be attributed to project interventions and whether key outcomes indicators specified and defined in the projects’ results framework have been met or whether there is any signal of change, in this regard. More specifically; the study will find out whether productivity and income of smallholder farmers and women have increased through improving the capacity of FTCs; whether the project has contributed to and improved soil fertility of targeted smallholder farmers?; has increased access to market and linkages; and has income of youth and women increased through business and/or saving schemes designed by the project. The study will be expected to document best practices and lessons of the project to be shared to stakeholders for future implementation and scale up. It will also generate gender disaggregated data at least on the key indicators.

3. Key Evaluation Questions

The study needs to respond to the following evaluation questions applying appropriate evaluation methodologies. In addition to these questions the evaluation should refer to any unintended changes and/or impact of the project.

A. To what extent are targeted FTCs & local institutions delivering demand-driven extension services led to increased productivity and/or income of smallholder farmers including women?

i. How useful and applicable are the technologies and best practices demonstrated at target FTCs? Has the project achieved an increase in number of better technologies and best practices adopted by smallholders’ farmers? How much has it improved adoption rate overall?

ii. How operational and equipped are FTCs and FTC MCs in the management of FTCs? Is community based FTCs management system strengthened?

iii. How many of the project target FTCs transformed to or meets intermediate level of classification? (Oxfam will provide a document which shows criteria for the levels of classification)

iv. Has DAs/SMSs, and FTC MCs technical, managerial, and operational ability increased due to capacity building activities?

v. Are local NGOs capacitated on FTC MC Establishment, project management, financial management; grant report preparation, value chain approaches, and Monitoring Evaluation and Learning (MEL) tools that ensure provision of demand-based extension services?

vi. Have target FTCs established sustainable business schemes and generating revenue on a sustainable basis to sustain the provision of improved and farmer-driven extension services?

vii. Has the project reached to both women and youth equally in provision of modern extension services?

47

B. Has the project contributed and improved soil fertility of targeted smallholder farmers’ plots through the use of integrated soil fertility management (ISFM) technologies and best farming practices?

i. Has the relevant type and amount of inputs (improved seeds, fertilizer, etc.) been procured and delivered to FTCs for timely demonstrations? How many famers received inputs for demonstrations in their plot? disaggregated by gender

ii. Have target smallholder farmers (both male and female) increased access to ISFM technology demonstrations at FTCs that has led to adoption of such technologies and practices? a. What % of such demonstrations has led to adoption of such technology and

practices? Disaggregated by gender b. Has the project achieved 50% participation/attendance in ISFM technology

demonstrations? c. How was the gender composition in training and demonstration activities of

ISFM technologies? iii. Are targeted smallholders farmers (both male and female) practicing/using

ISFM technologies & best farming practices in their own plot? Which ones are adopted and which ones are not? And why?

C. Has increased access to improved market information and linkages led to increased productivity and/or income?

i. Have target smallholder farmers (both male and female) got access to market information through Market Information System MIS?

ii. Are the established market information centers at each of target FTCs increased access to and are providing market information that is timely, informative, and leads to improved marketing and increased linkages?

iii. Are DAs, SMSs and farmers/ farmer organizations/ are effectively and sustainably utilizing mobile-based market information, market information and post-harvest management technologies to advise farmers? disaggregated with gender

D. Has income of youth & women groups in target FTCs increased through business and/or saving schemes designed by the project?

i. Have capacity building trainings for DAs and SMSs enabled effective business plan development and value chain approach of youth & women groups?

ii. How many of the groups are functioning and generating earnings from the schemes sustainably (at least 10 of established groups)? How does the project impact group members’ life in terms of income increase and linking with FTC extension and advisory services?

iii. Does this groups enhanced economic and social empowerment of women’s in the community?

E. Has the project documented best practices and lessons during implementation period and shared with relevant stakeholders (MoA and others) for future and wider scale-up efforts?

F. What are the reasons behind achieving or not achieving the objectives of the project/ and or the learning questions?

G. Whether the project is contributing to Oxfam’s mission?

H. What is Oxfam’s value added?

48

I. Is the intervention in line with the priorities and policies of the people and communities it is working with?

J. Have the funds been used economically, effectively and efficiently? What is the Sustainability and prospects for scaling the project?

4. Scope of the evaluation, activities and methods Scope: This evaluation will be delimited to answering the above major and detailed questions to understand and draw lessons on the implementation performance and impact of the project in four of the regions (Amhara, Oromia, Tigray and SNNPR) of Ethiopia. The project started in 2014 and runs through 2016; this evaluation hence studies the changes in the impact, outcome and output indicators the project aimed to achieve between these periods. The evaluation approach taken will be gender-sensitive, inclusive of all stakeholders including people with a disability, culturally-sensitive and participatory; and will obtain informed consent from all evaluation participants Activities:

Major activities that are part and parcel of the evaluation are: preparing technical and financial proposal, reviewing project documents (project proposals, theory of change, MEL frameworks, baseline documents), drafting an inception report with clearly defined methods and data collection tools, piloting tools on the field and data collection, data clearance and analysis, preparation of first draft evaluation report, review of comments and production of final report, and participation in the validation workshop.

Methods:

To carry out this study the consultant will employ:

· Quantitative and qualitative methods to generate end line data on the project’s outcome or outputs indicators (see the annex below).

· Quantitative data will be collected (using the questionnaire to be developed in consultation with the OA-HARO program and AGRA M & E officer responsible for Ethiopia), edited, processed and will be analyzed using appropriate statistical packages while qualitative data will be collected through qualitative tools (such as focus group discussions, key informant interview, observations etc.,) and will be analyzed systematically.

· The data should be collected from the same sampled FTCs and targeted small-holders that were selected for baseline study and benefiting from ISAP project so as to see the progresses towards outcome.( see annex 1)

· In order to determine appropriate sample size at the household level power calculation needs to be employed. Simple random sampling methods needs to be employed while selecting target small-holders including Female Household (FH) who were the potential users of the FTCs and other project interventions. · The study should come up with sex disaggregated data and the conduct of gender analysis ( using OA gender analysis toolkit ) of the current situations around the project interventions 5. Time frame, deliverables and budget disbursement

Deliverable and time frame:

a) Finalized methodological approach for project evaluation, including schedule, deliverables and technical proposal: October 30, 2016

49

b) Review project documents, ToC, MEL frameworks and plans, financial and narrative reports and produce inception report: November 5, 2016.

c) Evaluation learning workshop (to further refine evaluation approach, discuss on anticipated results and research tools): November 8, 2016.

d) Data collection and preliminary finding briefing: November 23, 2016.

e) Produce the first draft version of the report: November 30, 2016

f) Second draft report after OA and partners review: December 5, 2016

g) Final report and verbal presentation on validation workshop (incorporating comments provided from OA and partners based on the feedback on the draft versions): November 10, 2016.

h) Submission of final reports, after comments from all stakeholders (within five days of receiving the comments)

i) Final report should follow the Oxfam evaluation template

j) Handover of all raw qualitative and quantitative data in electronic form (cleaned version), with hard copies of the report

The consulting firm should submit the technical and financial proposals by October 25, 2016 both electronically via address provided at contact person section below, and hard copy in separate sealed envelopes during office hours to Oxfam America Horn of Africa Regional Office/ Ethiopia, Addis Ababa, Bole Road, DH Geda Tower, 5th floor

As part of the technical proposal, the consulting firm shall attach copies of

· Renewed business license;

· VAT registration certificate and other legal documents;

· Certification from clients for whom a consultancy assignment has been carried out;

· Curriculum vitae of the proposed professionals 6. Logistics

Oxfam will arrange transport service during the fieldwork for the study outside of Addis Ababa. Other consultancy costs have to be clearly indicated and included in the financial proposal.

7. Evaluation responsibilities and management arrangements The study will be accompanied by a Steering Committee which is comprised of the project manager, Country Director, Oxfam HARO MEL Officer, OUS’s Monitoring, Evaluation, and Learning Specialist, Finance specialist and Human Resource manager whose role will be to establish the ToR, select best candidate bider based on agreed selection criteria, provide input to the Study Team and help ensure the quality of the study by reacting to work-plans, methodology, initial findings and final reports.

The lead consultant will be accountable/report to the Agricultural extension and Monitoring, Evaluation, and Learning coordinator, and Project officer . The consultant is expected to regularly update the mentioned staff on progress of evaluation and seek feedback and direction as required.

50

During implementation the project officer/coordinator will coordinate with relevant project and Oxfam staff to provide inputs, feedback and sign-off within agreed timeframes. Oxfam will effect payment to the CONSULTANT upon completion of consultancy deliverables stated in this Scope of Work. Oxfam is also responsible for quality assurance of this final evaluation: refining the TOR, commenting on the inception report, refining the tools, methodology of the study together with the CONSULTANT and then commenting on the draft research reports as well as signing off the final one. Oxfam will participate in and contribute to setting the overall framework and methods and tools of data collection, and supervision of the research assignment. Oxfam will be responsible for facilitating administrative issues in relation to contacting various offices through letters to simplify data collection.

8. Dissemination and use of evaluation findings

o Preferably the full evaluation report and at least the executive summary and management response will be posted on Oxfam websites and made available for publishing to institutional donors. This is consistent with Oxfam’s commitment to transparency and accountability

o The findings will be shared to communities, partners, and key stakeholders (internal and external) as part of Evaluation validation workshops. The findings will inform program development, organizational learning, accountability and advocacy. Following the completion of the evaluation, a management response to the findings and recommendations will be developed.

Annex 1: Number of project target Woredas, FTCs and sample households that were covered by the Baseline Study Number of project targeted FTCs per region per project year Region Woreda FTCs Sample

Woredas Sample FTCs/ Woreda

Sample Household /FTC

No. of Woredas

Woredas

Oromia 3 Wuchale 8 2

4

338 Sibu Sire 7

Honkolo 7 Amhara 3 Machakel 7

2 2

324 Gozamin 4

Aneded 5 Total 6 38 4 6 662 Annex 2: Geographical location and distribution of project targeted FTCs per regions Number of project targeted FTCs per region per project year Region Number of Woredas Number of FTC Oromia 3 22 Amhara 4 16 Tigray 3 9 SNNPRS 3 13 Total 13 60

51

Job Requirements

9. Required Profiles of the Evaluation Team

The consulting firm should have experience in conducting baseline studies, project/program evaluations (especially around agricultural related programs/projects), and should have multidisciplinary team including gender expert. The consultant team should fulfill the following requirements:

Post graduate degree in agricultural economics, agriculture development studies & other relevant social sciences from a recognized institution,

Extensive experience in conducting evaluations, baseline studies, and researches in the areas of agricultural growth initiatives and agricultural extension in particular,

Experience in program design, and monitoring and in producing high quality study or evaluations report,

Experience in conducting quantitative and qualitative analysis, gender sensitive evaluations, baseline surveys and studies

Knowledge of and practical experience in the application of conceptual frameworks and analysis of gender

Excellent working knowledge of Monitoring, Evaluation, and Learning (MEL) frameworks,

Excellent reporting writing skills in English language How to Apply

Application Procedure: Interested & qualified consultancy firms can send their technical and financial proposals which should include: 1) Clear and precise methodology and approach to undertake the assignment 2) Latest curriculum vitae of the consultants that highlight their proven experience in the areas required 3) Evidence of relevant previous experience and / or names and contact details of organizations the firm worked for. 4) Sample reports of previous similar assignments done by team members proposed for this assignment 5) Schedules of consulting fees 6) Copy of Renewed License, TIN, &VAT registration certificate of the consultancy firm The Technical and financial proposals shall be submitted in separate sealed envelopes to our Office during Office hours (8:30-5:00PM) by October 3, 2016 via the following address: Oxfam America Horn of Africa Regional Office Bole Road, DH Geda Tower, 5th floor Addis Ababa, Ethiopia

Forty percent of the people on our planet—more than 2.5 billion—now live in poverty, struggling to survive on less than $2 a day. Oxfam America is an

international relief and development organization working to change that. Together with individuals and local groups in more than 90 countries, Oxfam

saves lives, helps people overcome poverty, and fights for social justice. To join our efforts or learn more, go to www.oxfamamerica.org.

www.oxfamamerica.org

© 2016 Oxfam America Inc. All Rights Reserved. Oxfam America is a registered trademark of Oxfam America Inc., and the Oxfam logo is a registered trademark of Stichting Oxfam International.

HEADQUARTERS

226 CAUSEWAY STREET, 5TH FLOOR BOSTON, MA 02114-2206

(800) 77-OXFAM

Horn of Africa Regional Office DH Geda Tower, Bole Sub-City, Kebele 02

Addis Ababa, Ethiopia +251 116 628 422

http://www.oxfamamerica.org/

http://www.oxfamamerica.org/

Documents

END LINE EVALUATION REPORT OF INCREASING SMALLHOLDER ... · Training Centers (ISAP) project was funded by Alliance for a Green Revolution in Africa (AGRA) and implemented by Oxfam