Summative Evaluation of the Nutrition Component of the EU ...€¦ · Summative evaluation of the Nutrition component EU-SHARE Program – Evaluation report 8 Purpose and objectives

Summative Evaluation of the Nutrition Component

of the EU-SHARE program (2015-2018)

Evaluation report

Submitted to:

UNICEF Ethiopia

Prepared by:

ACT for Performance BV, Netherlands

JaRco consulting, Ethiopia

December 07, 2018

www.actforperformance.com

ACT FOR PERFORMANCE BV

HNK Ede Horapark

Bennekomseweg 41

6717 LL, Ede, The Netherlands

+ 31 (0)615094443

+ 31 (0)8008098

ACT for Performance Office network: Canada, Netherlands, and Democratic Republic of

Congo

P.O. Box 43107

Addis Ababa

Ethiopia

www.jarrco.info

+251-115-577236 /59

+251-115-577276 /99

http://www.actforperformance.com/

Summative Evaluation of the Nutrition component EU-SHARE Program – Evaluation report 1

TABLE OF CONTENTS

LIST OF ACRONYMS ............................................................................................................................................................................ 3

ACKNOWLEDGEMENTS ..................................................................................................................................................................... 6

1 EXECUTIVE SUMMARY .............................................................................................................................................................. 7

2 INTRODUCTION ........................................................................................................................................................................ 15

2.1 CONTEXT ................................................................................................................................................................................................... 15

2.2 EVALUATION OBJECTIVES AND SCOPE. ............................................................................................................................................... 16

2.3 OBJECT OF EVALUATION ........................................................................................................................................................................ 18

2.4 PROJECT’S THEORY OF CHANGE ........................................................................................................................................................... 19

3 EVALUATION APPROACH AND METHODOLOGY .............................................................................................................. 22

3.1 EVALUATION APPROACH ....................................................................................................................................................................... 22

3.2 DATA COLLECTION AND ANALYSIS ...................................................................................................................................................... 23

3.2.1 Evaluation matrix ....................................................................................................................................................................... 23

3.2.2 Evaluation sample ...................................................................................................................................................................... 23

3.2.3 Data collection ............................................................................................................................................................................. 25

3.2.4 Analysis and reporting.............................................................................................................................................................. 26

3.2.5 Ethical considerations .............................................................................................................................................................. 26

3.3 QUALITY ASSURANCE ............................................................................................................................................................................. 27

3.3.1 Overall quality assurance ........................................................................................................................................................ 27

3.3.2 Recruitment of survey teams .................................................................................................................................................. 28

3.3.3 Training and pre-test ................................................................................................................................................................ 28

3.3.4 Field manual ................................................................................................................................................................................. 28

3.3.5 Data quality assurance during data collection ............................................................................................................... 29

3.4 CONSTRAINTS AND LIMITATIONS ........................................................................................................................................................ 30

4 FINDINGS .................................................................................................................................................................................... 30

4.1 RELEVANCE .............................................................................................................................................................................................. 32

4.2 EFFECTIVENESS ....................................................................................................................................................................................... 34

4.3 EFFICIENCY ............................................................................................................................................................................................... 48

4.4 IMPACT ...................................................................................................................................................................................................... 50

4.5 SUSTAINABILITY ..................................................................................................................................................................................... 56

5 CONCLUSIONS ........................................................................................................................................................................... 59

6 RECOMMENDATIONS .............................................................................................................................................................. 64

ANNEX A ACTIVITIES OF THE PROGRAM (LOG-FRAME) .................................................................................................... 66

ANNEX B EVALUATION MATRIX ................................................................................................................................................ 68

ANNEX C LIST OF DOCUMENTS AND ARTICLES CONSULTED ........................................................................................... 74

ANNEX D STAKEHOLDERS AND INFORMANTS MET ............................................................................................................ 76

ANNEX E TERMS OF REFERENCE .............................................................................................................................................. 79

Summative evaluation of the Nutrition component EU-SHARE Program – Evaluation report 2

TABLES AND FIGURES

TABLE 1 : SAMPLE SIZE DETERMINATION ...................................................................................................................................................... 23

TABLE 2 : DISTRIBUTION OF SAMPLED HHS WITH EAS PER WOREDA, ZONE AND REGION ................................................................... 24

TABLE 3: SOCIO-DEMOGRAPHIC CHARACTERISTICS OF HOUSEHOLDS IN THE THREE REGIONS ............................................................. 31

TABLE 4 : ACHIEVEMENTS IN OTHER BASIC SERVICES INTERVENTIONS .................................................................................................... 40

TABLE 5 : ACHIEVEMENTS CONCERNING PROJECT EXPECTED RESULT 3 ................................................................................................... 41

TABLE 6 : ACHIEVEMENTS CONCERNING PROJECT EXPECTED RESULT 2 ................................................................................................... 47

TABLE 7 : TRENDS IN WASTING PREVALENCE AT BASELINE AND ENDLINE IN THE THREE REGIONS .................................................... 51

TABLE 8 : TRENDS OF SEVERE AND MODERATE WASTING RATES AT BASELINE AND ENDLINE ............................................................. 51

TABLE 9 : PREVALENCE OF SEVERE AND MODERATE UNDERWEIGHT BY REGION AT BASELINE AND ENDLINE ................................... 52

TABLE 10 : TRENDS OF UNDERWEIGHT BY AGE GROUP AT BASELINE AND ENDLINE ................................................................................. 52

TABLE 11 : TRENDS IN STUNTING PREVALENCE AT BASELINE AND ENDLINE IN THE THREE REGIONS ................................................... 52

TABLE 12 : DISTRIBUTION OF STUNTING BY AGE GROUPS AT BASELINE AND ENDLINE ............................................................................ 52

TABLE 13 : TRENDS IN THE FOUR KEY PROJECT INDICATORS ASSESSING PROGRESS IN COMPLEMENTARY FEEDING PRACTICES ....... 54

TABLE 14 : PROGRESS ACHIEVED IN BEHAVIOURAL CHANGES AND PRACTICES DURING PREGNANCY AMONG WOMEN ........................ 55

TABLE 15 : PROGRESS ACHIEVED IN HOUSEHOLDS HUNGER SCALE DURING PROGRAM IMPLEMENTATION ......................................... 56

TABLE 16 : EVOLUTION IN FREQUENCY OF ANC FOLLOW-UPS OF WOMEN DURING THEIR LAST PREGNANCY BY REGION .................. 56

TABLE 17: PRACTICES ON HAND WASHING AT DIFFERENT CRITICAL TIMES AT BASELINE AND ENDLINE ............................................. 57

TABLE 18: PROGRESS IN OTHER WASH PRACTICES AT BASELINE AND ENDLINE .................................................................................... 58

FIGURE 1 : PROJECT’S THEORY OF CHANGE .................................................................................................................................................... 20

FIGURE 2: VITAMINE A SUPPLEMENTATION ACHIEVEMENTS AMONG CHILDREN .................................................................................... 35

FIGURE 3: DEWORMING ACTIVITIES AMONG CHILDREN 24 – 59 MONTHS ............................................................................................... 35

FIGURE 4: DEWORMING ACHIEVEMENTS AMONG ADOLESCENTS ................................................................................................................ 36

FIGURE 5: ACHIEVEMENTS IN IRON FOLIC ACID SUPPLEMENTATION AMONG PREGNANT WOMEN ....................................................... 37

FIGURE 6: SCREENING FOR ACUTE MALNUTRITION AMONG CHILDREN ..................................................................................................... 37

FIGURE 7: SCREENING FOR ACUTE MALNUTRITION AMONG PREGNANT AND LACTATING WOMEN ....................................................... 38

FIGURE 8: CHILDREN UNDER 2 PARTICIPATION IN GMP ............................................................................................................................. 38

FIGURE 9: TRENDS IN EXCLUSIVE BREASTFEEDING AT BASELINE AND ENDLINE IN THE THREE REGIONS ............................................ 53

FIGURE 10: TRENDS IN MINIMUM DIET DIVERSITY FOR WOMEN IN THE THREE REGIONS ........................................................................ 55


LIST OF ACRONYMS

ACF Action contre la faim

ADA

AMREF

agricultural development agent

AMREF Health Africa

ANC

BoANR

BoLF

BCC

Ante natal care

Bureau of Agriculture and National Resources

Bureau of Livestock and Fisheries

behaviour change and communication

CBN Community-based Nutrition Programme

CF complementary feeding

CHD

CMAM

community health days

community management of acute malnutrition

CPP Country Programming Paper

CSO

CWW

civil society organization

Concern Worldwide

DEVCO International Development Corporation of the EU

DFID Department for International development of the UK

DA Development Agent

DAC Development Assistance Committee

DHS Demography and Health Survey

DRM disaster risk management

EA enumeration area

EC European Commission

ECHO European Commission's Humanitarian Aid and Civil Protection department

EDHS Ethiopia Demography and Health Survey

ENA Emergency Nutrition Assessment

ENGINE Empowering New Generations to Improve Nutrition and Economic

Opportunities

EPHI

EU-SHARE

Ethiopian Public Health Institute

The European Union – Supporting Horn of Africa Resilience

EU European Union

FAO Food and Agriculture Organization

FGD focus group discussion

FMOH Federal Ministry of Health

FTC

GAIN

Farmer Training Centre

Global Alliance for Improved Nutrition

GDP Gross Domestic Product

GMP growth monitoring and promotion

GoE

H/A

Government of Ethiopia

height-for-age

HABP household asset building programme


HDA

HEW

health development army

health extension worker

HH household

HPN health population and nutrition

HQ headquarter

HSDP Health Sector Development Programme

IEC information, education and communication

IFHP integrated family health programme

IMC International Medical Corps

IMEP integrated monitoring and evaluation Plan

IYCF infant and young child feeding

KAP knowledge, attitudes and practices

KII key informant interview

M&E

MDD-W

monitoring and evaluation

minimum diet diversity for women

MIYCF maternal, infant and young child feeding

MoA

MoANR

MoLF

Ministry of Agriculture

Ministry of Agriculture and Natural Resources

Ministry of Livestock and Fisheries

MUAC mid-upper arm circumference

NGO non-governmental organization

NNCB National Nutrition Coordination Body

NNP National Nutrition Programme

OECD Organisation for Economic Co-operation and Development

OTP

PCA

outpatient therapeutic program

Project Cooperation Agreement

PDA personal digital assistant

PIF policy investment framework

PLW pregnant and lactating women

PSNP Productive Safety Net Programme

QA quality assurance

REACH Renewed Efforts Against Child Hunger

RHB

SAA

SLP

Regional Health Bureau

social analysis and action

System-wide Livestock Program

SUN Scaling Up Nutrition

ToC Theory of Change

ToR Terms of Reference

ToT

UNEG

Training of Trainers

United Nations Evaluation Group

UNFPA United Nations Population Fund

UNICEF United Nations Children’s Fund

USAID United States of America Assistance for International Development


USI universal salt iodization

W/A weight-for-age

W/H weight-for-height

WaSH water, sanitation and hygiene

WFP World Food Programme

WHO World Health Organization


ACKNOWLEDGEMENTS

The Evaluation Team would like to thank the many participants in this evaluation for their

contributions.

We highly appreciated working with Mr. Eric Ategbo, the head of UNICEF’s Nutrition department, and

his team in Addis Ababa, in particular Ms. Barbara Baille, the M&E officer and focal point for this

evaluation, and Ms. Nardos Birru, the coordinator of the project we evaluated, and who always

responded promptly and timely to our many requests for information and documents. We also would

like to thank the UNICEF regional teams for accompanying us in the field and deepening the team’s

understanding of the project’s implementing environment.

We thank the representatives from the EU, FAO and from the Ministries and Bureaus of Health and

Agriculture in Addis Ababa, as well as the representatives of the implementing organizations (Concern

Worldwide, ACF, CARE, IMC and AMREF) for their kind cooperation and sharing of information.

We would like to express our special gratitude to the Regional Offices of Health and Agriculture of

Oromia, Amhara and SNNPR regions for their assistance in coordinating the meetings and visits

during the case study field missions.

Last but not least we thank the health and agriculture extension workers and the beneficiaries in the

kebeles for the time they shared with us and their frankness. They are the actors who experienced

the effects of the nutrition and agricultural activities in the communities. Much of what is written in this

report is founded on their accounts.

7

1 EXECUTIVE SUMMARY

Overview

1. Ethiopia has experienced repeated environmental and external economic shocks for several years

that have eroded rural livelihoods, exacerbating vulnerability and reducing resilience. Children are

most affected, with an estimated 38% who are stunted. Stunting rates in Ethiopia are higher than the

average for the Eastern and Southern Africa regions. Childhood undernutrition is a complex issue

with many causes, including inadequate dietary intake, frequent illnesses, as well as poor feeding and

caring practices. Inappropriate Infant and Young Child Feeding (IYCF) practices are thought to be an

important determinant of malnutrition. On the other hand, breastfeeding is nearly universal in Ethiopia,

with a reported 97% of children 0-5 years who have ever been breastfed. However, according to the

2016 Ethiopia Demographic and Health Survey (EDHS), only 59% of children 6 to 8 months of age

had reportedly consumed solid or semi-solid foods, and just 7% had consumed the recommended

diversity of complementary foods.

2. With two out every five children who are stunted, the long-term economic implications for the country

are serious as this affected generation will be less healthy and less productive. To address these

nutritional challenges, Ethiopia developed various nutrition and health related policies and strategies,

including the 2013 and 2016 National Nutrition Programme (NNP), the Health Sector Development

Programme IV (HSDPIV - 2010 to 2015), the Agriculture Sector Policy Investment Framework (PIF -

2010 to 2020), and the National Disaster Risk Management (DRM) Policy. In 2011, the country joined

the renewed global commitment to end undernutrition and developed the accelerated stunting

reduction strategy, which calls for increased focus on multi-sectoral and integrated approaches to

address stunting. Currently, a number of nutrition programmes support the NNP with the objectives

of combating undernutrition among children and women in the country.

3. To support Ethiopia in enhancing its resilience capacity to withstand external shocks through a holistic

development approach, including water, livelihoods, basic services, conflict resolution and peace

building, since 2012 the European Union (EU) has funded an initiative called the “EU-SHARE project”

to sustain access to water, food security and nutrition and promote developmental actions for a

sustainable agricultural growth in agro-pastoral areas of the country. The goal of this project is to

strengthen the linkages between relief, recovery and development. The nutrition component of this

EU-SHARE project, formulated in 2014, started in 2015 and ran until the end of 2018. It was designed

to contribute to the ECHO-DEVCO resilience building programme, which has as its objective to build

on nutrition outcomes of the other SHARE components, and to enhance decentralized and multi-

sectoral implementation. A number of nutrition-specific and nutrition-sensitive actions have been

implemented by UNICEF and its partner FAO in 17 woredas of Amhara, Oromia, and SNPP regions,

targeting adolescent girls, pregnant women, and children under 5, with a focus on the first 1000 days

of a child’s life.


Purpose and objectives

4. The data collection of this summative evaluation, which included an extensive end-line survey on a

series of nutrition indicators, took place in the period of March to July 2018, a few months before

closure of the nutrition component of the EU-SHARE project in October 2018. The baseline survey of

the project was done in May 2017, two years after the start of the project. Only having one year

between both surveys is quite short to observe significant changes. The endline survey assessed the

coverage of activities implemented through this project, as well as global knowledge and practices

around nutrition for children under 5, adolescents and pregnant and lactating women.

5. The purpose of this summative evaluation was to assess the results of the nutrition component of the

EU-SHARE project and the performance and accountability of the main program partners (UNICEF

and FAO) with respect to the agreed programme results. The evaluation also will be used to promote

learning from the findings and conclusions about what works, what doesn’t and why. The evaluation

report includes lessons learned on the added value of the multi-sectoral approach to nutrition,

especially considering the move from emergency to resilience.

6. This evaluation report provides information to the key audiences, including the Government of

Ethiopia (GoE), UNICEF, FAO, and other UN organizations working in the nutrition field, implementing

agencies, CSOs, EU and other donors on (i) the current status of knowledge, attitudes, and practices

of appropriate Maternal, Infant and Young Children feeding (MIYCF) behaviours; and (ii) the impact,

effectiveness and scalability of the nutrition specific and sensitive (mostly agricultural) measures,

which will support policies, strategies and future programming.

The evaluation applied the OECD-DAC criteria – relevance, efficiency, effectiveness, impact, and

sustainability – and its specific objective was to answer the following questions:

What was the intervention logic (theory of change) of the program and is the program sensitive

to local and cultural contextual issues and relevant to host communities? Was the program in

line with government strategies and policies?

What (and why) changes/effects/impacts occurred in the short-term (knowledge, skills,

attitudes opinions), medium-term (behaviors, actions), and long-term (condition, status) as

result of the program? What changes occurred at the impact level indicators as a

result/contribution of the program?

What changes occurred at output and activity level of the program indicators compared to the

baseline? To what extent were the changes on-track or off-track compared to the targets?

Why?

To what extent was the program reaching (or has the program reached) the intended

beneficiaries? To what extent has the program impacted on their welfare (i.e. in terms of

improvements in health, nutrition status, access to resources)? Were program beneficiaries

sufficiently engaged in planning, implementation and review to feel the program was

sufficiently “downward” accountable?

Were nutrition status improvements, if any, commensurate with the investments made/ value

for money assessment (compared to similar programs)?

What was the quality of nutrition services?

To what extent was the program sustainable? If so, how and in what ways?


If the program was to be scaled up, which aspects of its operation must be modified or

strengthened for it to operate effectively?

Which good practices should remain the same and be replicated?

Was the program implemented in a gender and conflict sensitive manner that was appropriate

for the different and changing context in each of the communities?

Approach

7. With the premise that the evaluation should be judged by its utility and actual use, the evaluation team

adopted a balanced participatory approach, which is intended to ensure that existing stakeholder

knowledge is shared and conclusions are verified, but which on the other hand guarantees

independence of evaluation conclusions and recommendations. Those who have the most to benefit

from this evaluation, i.e. UNICEF, Federal Ministry of Health (FMOH) FAO, health facilities, donors,

and beneficiaries of the EU-SHARE project, played an active role throughout the process, particularly

in shaping the strong learning focus of this evaluation.

8. The evaluation team used a combination of secondary and primary qualitative and quantitative data

methods to generate the necessary information, basically organized in two parts: (1) Household and

community survey in the same 2695 households approached in the baseline survey, assessing a set

of nutrition performance indicators; and (2) a Qualitative evaluation, using documentary review, semi-

structured interviews with key stakeholders, and focus-group discussions with health facilities staff,

community workers, and beneficiaries. Data collection was conducted at national, sub-national

(regional), community, and household level.

9. Evidence and findings of the program evaluation embrace the views of all key stakeholders, including

the most vulnerable and poor people among the target communities in the 17 Woredas. Using the

survey approach, the evaluation team was able to measure the impact of the nutrition component of

the EU-SHARE program on their social and economic situation.

10. The evaluation also used a theory-of-change based “realist evaluation” approach, reconstructing the

project’s theory-of-change (see figure 1) and verifying the change hypotheses on factors influencing

the processes underlying the intervention logic (inputs, process/activities, outputs, intended and

unintended outcomes).

11. The evaluation was guided by the evaluation matrix, presented in annex B, which is organized

according to the evaluation criteria requested by the terms of reference. The evaluation questions in

this matrix were inspired by the terms of reference, the theory of change, and a first review of project

documentation. The matrix lists the key questions and indicators to measure, addresses the

assumptions underlying progress, and presents the data sources and the data collection techniques.

The findings in this evaluation report respond to the questions defined by the evaluation matrix.

12. The end-line survey (i) assessed the level of malnutrition using indicators and anthropometry

measurements, in the intervention zones and compared with the baseline; (ii) determined the current

level of knowledge, attitudes and practices (KAP) on key MIYCF behaviors and compared with the

baseline; (iii) assessed coverage of Vitamin A supplementation, deworming, iron supplementation

and Grow Monitoring and Promotion (GMP) participation and compared with baseline coverage; and


(iv) identified key barriers and opportunities to successful implementation and scalability of household

and community gardening support and other agricultural/nutrition linkages.

13. The quantitative survey data were collected using structured modular questionnaires through

interviews with male and female adults in each sampled household. The survey included modules

related to the indicators established during the baseline survey. The survey questionnaire was pre-

tested and refined accordingly, along with checking standardization on anthropometry measurements.

The data were collected by teams of supervisors and enumerators, who operated simultaneously in

their different regions, to increase the speed of the assignment. They used personal digital assistance

(PDA) devices with SMART Standardisation and ENA software, as adapted by ACF Canada for PDAs.

All enumerators and supervisors with quantitative data collection experience were trained on the

specific ENA tools, mobile data collection, and quantitative interview techniques. The evaluation team

created a SPSS database where all the survey data from the PDAs were stored and cleaned. ENA

software was used to analyze anthropometric measurements, while SPSS software was used to

analyse the other data from the household survey.

14. The qualitative part of the evaluation used documentary review, interview and focus group discussion.

The evaluation team developed the interview and focus group guides for the qualitative study after an

initial extensive document review and the stakeholder mapping, and following the end-line survey.

The interview guides were customised according to stakeholder groups and pre-tested and adjusted

accordingly. Documents and secondary data reviewed were obtained from UNICEF, FAO, the project

NGO partners and Government counterparts. Interviews and focus group discussion (FGD) sessions

were audio recorded, translated and transcribed. Two teams were formed for qualitative data

collection. Both teams worked together in Addis Ababa, then split for field visits in the regions. Data

triangulation and deduction were used to analyze the qualitative data.

15. Information was triangulated by comparing the multiple lines of evidence produced by collecting data

from the four data sources (household survey; document review; semi-structured interviews with

project stakeholders; and focus group discussions with caretakers). The data were sex-disaggregated

and coded under each evaluation criteria and key question.

16. The methodology and tools were developed in line with the procedure of UNICEF’s ethical standards

in research, evaluation, data collection and analysis. Within this procedure, the principles and

requirements for evidence generation are applied to four core ethical issues, namely; Harms and

Benefits, Informed Consent, Privacy and Confidentiality, and Compensation and Payment.

Key findings

17. Relevance: The project adopted a relevant targeting and location-specific approach as the activities

are woreda-specific and designed according to anutrition needs assessment in the targeted woreda.

Activities are strongly focused on the most vulnerable households, selected by a nutrition committee

at kebele level, and more specifically focused on pregnant and lactating women, on mothers with

children under 5, and on adolescents. The project empowered women groups through increased

income as a result of the sale of eggs and vegetables.


18. Targeted pregnant and lactating women, adolescents, and households with children under 5 became

very engaged in the process, as confirmed by focus group discussions with these groups. Women

and male farmers and Agriculture development agents (ADAs) were trained on homestead gardening

in the three regions. Almost half of vulnerable populations (44%) received agricultural inputs for

backyard gardening.

19. In line with the global guiding principles developed by the SUN movement, and with government

policies such as the Seqota Declaration, NNP II, and the Nutrition-sensitive Agricultural Strategy. the

project adopted a multi-sectoral approach, with nutrition-specific and nutrition-sensitive (e.g. home

gardens, school nutrition clubs) activities, which was highly appreciated by the beneficiaries and led

to tangible results. Nutrition is no longer seen as only a health issue but also as an agricultural and

education issue. Inspired by the project, the Ministry of Agriculture established a Nutrition department.

20. The project also fostered awareness on the importance of intervening during the first 1000 days of a

child’s life, when nutritional gains are critical for a child’s development. Target populations learned

about feeding practices, including breastfeeding and preparing complementary food. Several

activities that were organized through campaigns, such as the provision of vitamin A, deworming

tablets, and iron folate supplements, were mainstreamed into the routine health programs.

21. Effectiveness: The following project interventions achieved most of the targets: (i) Vitamin A

supplementation; (ii) Deworming (in children aged 23 to 59 months and adolescents); (iii) Iron Folic

Acid (IFA) supplementation among pregnant women; (iv) Screening of children and pregnant and

lactating women for acute malnutrition; (v) Growth Monitoring and Promotion (GMP) of children under

2 years of age; (vi) Promotion of complementary feeding practices.

However, there were considerable discrepancies between data in the progress reports and endline

survey findings for deworming in adolescents and screening of children and pregnant and lactating

women. According to the progress reports, targets were achieved, while the survey reported limited

or no achievements.

22. The quality of extension services improved in the project woredas. Health extension workers (HEWs)

and Agriculture development agents (ADAs worked very well together. They were trained to

understand the multiple benefits of eating a variety of foods and gained knowledge and skills to reduce

post-harvest losses and to improve food availability during times of emergency crises. Regular

supportive supervision visits were conducted by nutrition experts appointed by FAO and UNICEF to

monitor the implementation of the program at the beneficiary level and ensure the quality of the work

of the HEWs and ADAs, necessary given the high turnover of these agents. The project thus solidified

the multi-sectoral approach that UNICEF has promoted over the years.

23. Factors that positively contributed to the project outcomes included: (i) basic services implemented

as routine activities; (ii) linkage between nutrition and agricultural activities, including a strong

collaboration between HEWs and ADAs; (iii) economic empowerment of targeted women through

productive activities; (iv) the use of the social analysis and action (SAA) method to foster behavioral

change; (v) the cascade 1-30/1-5 model of community leadership; (vi) the establishment of school

nutrition clubs; (vii) coordination of the regional taskforce; and (viii) a good collaboration at the EU

level between ECHO and DEVCO.


24. Constraints identified during the evaluation included: (i) delayed implementation; (ii) weak

coordination between administrators at zonal and woreda levels; (iii) high workload of the HEWs; (iv)

environmental factors such as recurrent drought and thus water shortage for drinking and for

vegetable production, and heavy flooding and landslides; and (v) tense security situations in some

woredas of the country.

25. Efficiency: The project underspent its available budget. According to the latest financial expenditure

report at the disposal of the evaluation team (August 17, 2018), a few months before closure, the

project spent €7.12 million, 66.7% of the total programmable funds available (€10.67 million). The

interviews with the implementing organizations also confirmed that it seemed to be impossible to

execute the remaining of the programmable budget in the last few months of the project.

26. Underspending was caused by a slow start of the project and cost estimations that were too high, but

also were compounded by a suboptimal coordination architecture. The project did not have a project

coordination unit with representatives of the implementing agencies, which hampered structural

project adjustments (e.g. shifting between components) to deal with the underspending, and which

also led in a few cases to NGO activities overlapping with government activities.

27. Impact: The project proved to be successful in addressing severe and moderate acute malnutrition,

and severe underweight in children under five years of age in the 17 targeted woredas. Improvement

was also observed in older children (30 months and older) suffering from wasting and severe

underweight. On the other hand, the project was not successful in reducing moderate underweight

and stunting prevalence among children in the geographic targeted areas.

28. The project interventions were successful in improving exclusive breastfeeding. However, although

some improvement occurred in the minimum acceptable diet, none of the targets was achieved when

looking at the four key indicators defined in the project’s logframe to assess progress in

complementary feeding practices among children. With respect to pregnant and lactating women, the

interventions contributed to: positive behavioral changes and practices during pregnancy;

improvement in the minimum diet diversity; and reduction of hunger in the households. However, the

impact on antenatal care practices was not conclusive. Regarding WASH practices, the intervention

slightly improved hand washing at critical times, but did not foster any improvement in water treatment

practices and use of toilets.

29. Sustainability: Scaling up of the project to all woredas in the country is potentially possible because

the National Nutrition Plan (NNP) II and the National nutrition-sensitive strategy have fully integrated

the project’s approach. However, the financing of both plans has not yet been secured.

30. The project’s work on feeding practices, and the close collaboration between health and agricultural

extension workers fostered by the project, are important activities that can be replicated in other

woredas with limited investments.


Conclusions, lessons learned, and recommendations

31. The project activities responded to the specific needs of the targeted population groups in the 17

woredas where the project intervened, and were in line with the government nutrition policy as

expressed in NNP II and the Nutrition-sensitive Agricultural Strategy. The quality of extension services

improved, despite many challenges.

32. The project interventions contributed to a reduction of the level of acute malnutrition and severe form

of underweight in children, but were not successful in reducing moderate underweight and stunting.

The practice of exclusive breastfeeding improved, but appropriate complementary feeding practices

among children remained a challenge. Behavioral changes and practices during pregnancy were

observed, the minimum diet diversity among targeted women improved, and hunger among

households was reduced. Although the project slightly improved handwashing practices, it did not

have a significant impact on antenatal care (ANC) practices in pregnant women, in water treatment

practices, or in the use of toilets among the target population.

33. One of the key targets of the project - establishing complementary feeding units managed by women

groups – did not succeed because there was not enough time. In addition, the project had to adjust

to an emergency mode in certain woredas because of drought or flooding.

R3: During a second phase, special attention should be given to building the units and organizing

the supply of products (supplements, seeds, gardening equipment) to guarantee timely

distribution and avoid delays.

R1: We recommend FMOH, UNICEF, and the EU design, fund, and execute a second project

phase of 4 – 5 years. Changing feeding practices and empowering targeted women groups

through complementary feeding units is a long-term matter that needs more time than the project

effectively had, even more so when taking into account the start-up delay and adjustments due

to emergencies that the project experienced.

R2: We recommend UNICEF and FMOH put even more emphasis in a second phase of the

project on the pregnancy and pre-pregnancy period for better impact, focusing on adolescents,

pregnant and lactating women.

A continuation of the project should build on the strengths of the first phase, as described in this

evaluation report, continue the targeting approach and reinforcing the collaboration between

health and agricultural services. However, reinforcing activities to support the communities as a

whole and not only the vulnerable households, such as school farms, are likely to foster more

engagement and support at the community level for the targeted activities and avoid tensions.


34. At the moment the evaluation was carried out, the project had spent about half of its budget. The

decision to conclude agreements with implementation agencies (NGOs) to increase the project’s

implementation capacity did not have the intended effect; only one third of the available budget for

the PCAs was spent. On the other hand, the level of execution of the components executed directly

through the government were the highest of the project.

35. The project did not have a project coordination unit with representatives of all implementation actors,

nor a project manager fully designated to the project with authority over all actors and vested in

government ownership. The project had a logframe and regular supervision missions were made, but

a systematic results-based monitoring mechanism to monitor progress against annual targets and

linked to expenditures was lacking.

36. The Government of Ethiopia’s National Nutrition Plan and Nutrition-sensitive Agricultural Strategy,

developed with technical assistance from the project, have mainstreamed the project’s activities, but

are not yet fully funded. Nevertheless, several key activities can be replicated elsewhere with limited

costs.

R4: We recommend UNICEF, FMOH and the donor to develop and implement a second phase

of the project focused on government ownership, and channel more funds through government

services rather than through implementing organizations. Government ownership is a condition

for sustainability and replication.

R6: We recommend UNICEF, FAO, FMOH, and the FMoA reinforce the link with the safety net

program and with value chain projects (e.g. moringa, NAIP), specifically to work with the targeted

woredas and communities on coping strategies to deal with extreme weather conditions.

R5: If the project continues (new phase), we advise UNICEF and FMOH to set up a project

coordination unit in the Federal Ministry of Health, supported by UNICEF and FAO, with a steering

group co-managed by the Ministries of Health and Agriculture, and chaired by the Minister of

Health. The unit should be a light structure to keep the project’s centre of gravity in the woredas.

If relevant, this project coordination unit will conclude agreements with implementation agencies,

such as CARE, to further disseminate the successful social analysis and action approach, or with

ACF (for instance) to provide training to government services and community leaders on

community management of acute malnutrition.

The project experienced implementation problems due to emergencies, such as drought, flooding,

and security. Therefore we recommend more flexibility for the coordination unit and an

administrative rapid response mechanism to efficiently adjust the project’s plan if needed, for

instance by moving to different woredas or by integrating new activities. Contractual agreements,

based on results rather than on interventions, could provide this flexibility.


2 INTRODUCTION

2.1 Context

Ethiopia has experienced repeated environmental and external economic shocks for several years

that have eroded rural livelihoods, exacerbating vulnerability and reducing resilience. Eastern and

southern lowlands and pastoral livelihoods have been particularly affected by more frequent droughts,

which led to significant loss of livestock, representing pastoralists' most valuable asset. Despite

sustaining double digit economic growth rates during the last twelve years, poverty remains high, with

27% of rural Ethiopians continuing to live below the poverty line and more than 22 million living below

the national poverty line1. Children are most affected, with an estimated 38% who are stunted. At

regional levels, stunting prevalence rates are over 46% in Amhara, 38% in SNNPR and 36% in

Oromia2, higher than the average stunting rate for Eastern and Southern Africa regions. Childhood

undernutrition is a complex issue with many causes including: inadequate dietary intake, frequent

illnesses, as well as poor feeding and caring practices. Inappropriate Infant and Young Child Feeding

(IYCF) practices are thought to be an important determinant of malnutrition. Breastfeeding is nearly

universal in Ethiopia, with a reported 97% of children 0 to 5 years ever having been breastfed.

However, only 59% of children aged 6 to 8 months of age had reportedly consumed solid or semi-

solid foods the day prior to the 2016 Ethiopia Demographic and Health Survey (EDHS), and just 7%

had consumed the recommended diversity of complementary foods.

The Cost of Hunger Study estimates that the annual costs associated with child malnutrition alone

are equivalent to 16.5% of Gross Domestic Product (GDP). Observations showed that in the effort to

reach middle income status, Ethiopia is hampered by the fact that 67% of the adult population suffered

from stunting as children3. With two out every five children stunted, the long-term implications are

even greater as this affected generation will be less productive, and less healthy. To address these

nutritional challenges, Ethiopia developed various nutrition and health related policies and strategies,

including the 2013 and 2016 National Nutrition Programme (NNP), the Health Sector Development

Programme IV (HSDPIV - 2010 to 2015), the Agriculture Sector Policy Investment Framework (PIF -

2010 to 2020) and the National Disaster Risk Management (DRM) Policy. In 2011, the country joined

the renewed global commitment to end undernutrition and developed the accelerated stunting

reduction strategy, which calls for increased focus on multi-sectoral and integrated approaches to

addressing stunting. Currently, a number of nutrition programmes support the NNP with the objectives

to combat undernutrition among children and women in the country.

Meanwhile, Development partners have committed themselves through the "Global Alliance for

Drought Resilience and Growth" initiative to better coordinate, harmonize, and align their programmes

and policies to enhance resilience against chronic droughts and promote economic growth in the Horn

1 Ethiopia’s Progress Towards Eradicating Poverty. An Interim Report on 2015/16 Poverty Analysis Study.

file:///C:/Users/Client/AppData/Local/Packages/Microsoft.MicrosoftEdge_8wekyb3d8bbwe/TempState/Downloads/2016%20Poverty%20Interim%20Report%20(1)%20(1).pdf.. 2 Ethiopia Demographic and Health Survey, 2016. 3 WFP, ECA, African Union. Cost of Hunger in Ethiopia: Implications for the Growth and Transformation of Ethiopia. Addis Ababa, 2013.


of Africa region. In line with this, Ethiopia has produced the Country Programming Paper (CPP) with

a focus on arid and semi-arid lands, with an aim to improve the livelihoods of agro-pastoral

communities and enhance their resilience capacity to withstand external shocks through a holistic

development approach, covering water, livelihoods, basic services, conflict resolution and peace

building4. To support Ethiopia in this regard, since 2012, the European Union (EU) has funded an

initiative called the “EU-SHARE project” which aims to address natural resources and access to water,

food security and nutrition, and promotes developmental actions for a sustainable agricultural growth

in agro-pastoral areas of the country, with the goal being to strengthen the linkages between relief,

recovery and development. The EU-SHARE project has six components, including: (1) Risk Financing

Mechanism of the productive safety-net programme (PSNP), (2) Nutrition, (3) Livestock health and

marketing, (4) Livelihood support, (5) Watershed Management, and (6) Local level capacity building

on coordination/planning. The Nutrition component of the EU-SHARE project, formulated in 2014,

started in 2015 and ran until the end of 2018. It was designed to contribute to the ECHO-DEVCO

resilience building programme, of which the objective is to build on nutrition outcomes of the other

SHARE components, and to enhance decentralized joint planning for the implementation. Additional

focus was given to ensure enhanced integration and harmonization of relief (ECHO) and development

(EU) opportunities. Under this Nutrition component, a number of nutrition-specific and nutrition-

sensitive actions funded by the EU have been implemented by UNICEF and its partner FAO in 17

woredas of Amhara, Oromia, and SNPP regions5, targeting adolescent girls, pregnant women, and

children under 5, with a focus on the first 1000 days of a child’s life.

In May 2017, a baseline survey of the EU-SHARE project was conducted, 2 years after the start of

the program in 2015. The report assessed the coverage of activities implemented through this project,

as well as global knowledge and practices around nutrition for children under 5, adolescents and

pregnant and lactating women. At the request of UNICEF, our evaluation team reviewed the baseline

study.

2.2 Evaluation objectives and scope.

The nutrition component of the EU-SHARE concluded in October 2018 and time has thus come to

undertake this summative evaluation (see annex 5 for the terms of reference). The purpose is to

assess the results of the nutrition component of the project in order to support accountability of

UNICEF and its partner FAO with respect to the agreed programme results and to promote learning

from the findings and conclusions about what works, what does not and why. It also includes lessons

learned on the added value of the multi-sectoral approach to nutrition, especially considering the move

from emergency to resilience. The evaluation took place between March and September 2018.

This evaluation report provides key information to stakeholders, including the Government of Ethiopia

(GoE), UNICEF, FAO, and other UN organizations working in the nutrition field, implementing

4 Integrated Nutrition Services: Multisectoral interventions to improve nutrition security and strengthening resilience. Consolidated proposal 2014. 5 Sekota and Zukala Woredas in the Amhara region; Dawe Ketchen, Gura Dhamole, Raytu, Arero, Dhas, Dillo, Dire, Miyo, Moyale Woredas in the Oromia

region; and Dasenech, Hammer, Boloso Sore, Damot Pulasa, Deguna Fanigo, Kindo Koysha Woredas in the SNPP regions


organizations, civil service organizations (CSOs), EU and other donors on: (i) the current status of

knowledge, attitudes, and practices of appropriate Maternal, Infant and Young Children feeding

(MIYCF) behaviours; and (ii) the impact, effectiveness and scalability of the nutrition specific and

sensitive (mostly agricultural) measures, which will support policies, strategies and future

programming.

This evaluation serves two main purposes:

Support accountability of UNICEF and its partner FAO with respect to the agreed program

results.

Promote learning from the findings and conclusions about what works, what does not and why.

This includes lessons learned on the added value of the multi-sectoral approach to nutrition,

especially considering the move from emergency to resilience. The evaluation also looks into

the policy, social, cultural, and economic context of the System-wide Livestock Program (SLP)

program.

The evaluation addresses the OECD-DAC criteria – relevance, efficiency, effectiveness, impact, and

sustainability – and according to the terms of reference, its specific objective is to answer the following

questions:

What is the intervention logic (theory of change) of the program and is the program sensitive

to local and cultural contextual issues and relevant to host communities? Is the program in line

with government strategies and policies?

What (and why) changes/effects/impacts occur in the short-term (knowledge, skills, attitudes

opinions), medium-term (behaviours, actions), and long-term (condition, status) as result of

the program? What changes occurred at the impact level indicators as a result/contribution of

the program?

What changes occurred at output and activity level of the program indicators compared to the

baseline? To what extent are the changes on-track or off-track compared to the targets? Why?

To what extent is the program reaching or has the program reached the intended

beneficiaries? To what extent has the program impacted on their welfare (i.e. in terms of

improvements in health, nutrition status, access to resources? Were program beneficiaries

sufficiently engaged in planning, implementation and review to feel the program was

sufficiently “downward” accountable?

Have nutrition status improvements, if any, been commensurate with the investments made/

value for money assessment (compared to similar programs)?

What is the quality of nutrition services?

To what extent will the program be sustainable? If so, how, and in what ways?

If the program is to be scaled up, which aspects of its operation must be modified or

strengthened for it to operate effectively?

Which aspects of good practice should remain the same and be replicated?

Was the program implemented in a gender and conflict sensitive manner that has been

appropriate for the different and changing context in each of the communities?


Evidence and findings of the program evaluation embrace the views of all key stakeholders, including

the most vulnerable and poor people among the target communities in the 17 Woredas. The impact

of the nutrition component of the EU-SHARE program on their social and economic situation has been

measured. Data collection was conducted at national, subnational (regional), community, and

household level.

The evaluation comprised a quantitative part (a survey measuring specific nutrition indicators), and a

qualitative part (based on a desk review, semi-structured interviews and focus group discussions with

stakeholders). The survey had the same scope as the baseline survey: 17 Woredas in 5 zones; 2695

households; and listed indicators. The qualitative research part took place in Addis Ababa and in 2

zones and 3 Woredas: Sekota Woreda in Wagehimra zone (Amhara region), Kindo Koyisha and

Damot Pulassa Woreda in the Wolayta zone (SNNP region). The team planned to visit Miyo and Dawe

Kechen Woreda in Borena zone (Oromia region) as well but could not go there due to security issues

in the regions.

2.3 Object of evaluation

The nutrition component of the EU-SHARE project aimed to contribute to improved nutritional status

of adolescent girls, pregnant and lactating women (PLW) and children under 5. The specific objective

was to contribute to the improvement of nutrition and dietary diversification practices of these target

beneficiaries, with a focus on the child’s first “1000 days.” Broad intervention areas included capacity

building and technical support in the areas of nutrition, health, food security and WASH,

communication for behavioural and social change, research and knowledge development,

coordination between development partners, and procurement of supplies. A number of activities

were identified and implemented under these intervention areas. Annex A presents the list of these

activities, extracted from the project’s proposal and structured according to project’s results. Aside

from the improvement to nutrition and dietary diversification practices of targeted beneficiaries, the

implementation of these activities were also intended to improve knowledge and capacity of national

partners (regional and woreda level) for coordination, implementation and management of agro-

pastoral and pastoral nutrition interventions. Five main results, each linked to the NNP, were expected

to be achieved through the implementation of the program. The results were: (1) Adolescent girls and

boys, pregnant and lactating women, and children under 5 are reached by quality preventative

nutrition interventions6, (2) Households are made aware of, and have increased access to, nutritious

foods and practices7, (3) Improved capacity for NNP implementation in project areas8, (4) Evidence-

based knowledge is available for NNP implementation9, and (5) Technical support is provided to the

EU and Joint programming in Nutrition10.

6 Linkage to NNP Strategic Objective 1 - Results 1.1 and 1.2; Linkage to NNP Strategic Objective 2 - Results 2.1 and 2.2 7 Linkage to NNP Strategic Objective 4 – Result 4.1, 4.6 and 4.7 8 Linkage to NNP Strategic Objective 5 – Result 5.1 and 5.5 9 Linkage to NNP Strategic Objective 5 – Result 5 10 Linkage to NNP Strategic Objective 5 – Result 5.3


The total targeted beneficiaries included 78,295 children under 2 years of age, 146,992 children aged

24-59 months, 50,734 PLWs, and 575,900 adolescent girls/boys and school age children in the 17

Woredas of the EC/ECHO clusters in Amhara, Oromia and SNNPR regions. The program is

implemented concomitantly with a number of nutrition programmes currently ongoing that support the

NNP with the objectives to combat undernutrition among children and women in Ethiopia. These

programs include: the Community Based Nutrition Programme (CBN), the Productive Safety Net

Programme (PSNP), National Universal Salt Iodization Programme (National USI programme), the

Integrated Family Health Programme (IFHP), the Alive and Thrive programme, the nutrition

programme called ENGINE, as well as the IYCF being integrated into Community Based Management

of Acute Malnutrition (CMAM) to strengthen preventive and developmental components of

management of acute malnutrition.

The Nutrition component of the EU-SHARE project is being implemented by UNICEF and its partner

FAO. However, some of the interventions are supported by NGOs with gradual hand over to the

government. A programme cooperation agreement (PCA) was signed with five NGOs having known

experience and currently operating in 12 out of the 17 intervention woredas11. These NGOs are Action

Contre la Faim (ACF) operating in Amhara and Oromia regions, CARE Ethiopia operating in Amhara,

Oromia and SNNPR regions, Concern Worldwide (CWW), International Medical Corps UK (IMC UK)

and Amref Health Africa (were ) operating in SNNP region. The NGOs provide a harmonized support

to the government systems in order to build and strengthen the support. Government counterparts

are mainly represented by the Ministry of Health and most Regional Health Bureaus (RHBs) through

the Health Extension Programme. The project was implemented in two main phases: (1) an inception

phase of 4 months during which the funds transferred from EU, through UNICEF Headquarters (HQ),

UNICEF Ethiopia and to the GoE and partners was processed, key activities necessary for program

implementation were defined, and a community dialogue was established; and (2) the implementation

phase, consisting of 42 months of activity implementation and monitoring in line with the program log-

frame and workplan. The total budget of the project was 10 million Euros, split between technical

support (€942,072.31), contract/PCA (€4,623,205.37) with implementing partners; capacity building

and mobilization for government (€1,049,866.32), supply (€1,261,403.43), M&E and operational

research (€777,270.45), general operating expenses (€691,977.07), and indirect operating costs

(€654,205.62).

2.4 Project’s theory of change

Assumptions and hypotheses on actors and factors influencing the processes underlying the

intervention logic (inputs, process/activities, outputs, intended and unintended outcomes) are central

to the process of a theory of change (ToC), since they are the ‘theories’ in the theory of change

thinking12. The evaluation team reviewed the intervention logic of the nutrition component of EU-

11 Integrated Nutrition Services: Multisectoral interventions to improve nutrition security and strengthening resilience. Consolidated proposal. 12 According to Vogel (2012), “Theory of change (ToC) for evaluation purposes tend to drill down into the detail of theories about cause-effect, the different pathways, actors and mechanisms the programme has influenced, as well as significant contextual conditions that had an influence.” The aim is to understand the underlying factors that determine issues of effectiveness and efficiency, and that foster positive change for target groups intended


SHARE project, and the underlying hypothesis or assumptions in order to develop the ToC (figure 1

hereafter). The ToC is a model linking program investment (inputs) to processes and outputs, and to

anticipated changes (outcomes) among the target population.

Figure 1: Project’s Theory of Change

The inputs consist of different international and national initiatives and policies aiming to fight against

chronic malnutrition. At international level, this includes the "Global Alliance for Drought Resilience

and Growth" initiative, designed in order to better coordinate, harmonize, and align their programmes

and policies to enhance resilience against chronic droughts and promote economic growth in the Horn

of Africa region. In line with this global initiative, the European Commission Communication on

Resilience developed the concept of resilience funding and integrated multiple levels of interventions

in the present project, addressing the causes of vulnerability and enhancing capacities to better

to benefit, and to contextualize and explain other issues raised by the stakeholders. Vogel, Isabelle (2012). Review of the use of ‘Theory of Change’ in international Development, UK Department of International Development.

“Global Alliance for Drought Resilience and Growth" initiative National policies and strategies Country Programming Paper Leadership, Coordination and Sectoral accountability Funding Available human resources, infrastructure and equipment

En

vir

on

me

nta

l a

nd

Pro

gra

mm

ati

c C

on

tex

t So

cio

-dem

ogr

ap

hic

, po

liti

cal,

eco

no

mic

, geo

grap

hic

al, c

ult

ura

l

Adolescent girls, pregnant and lactating women, and children under 5 are reached by quality preventative nutrition interventions

Households are made aware of, and have increased access to, nutritious foods and practices

Improved capacity for NNP implementation in project areas Evidence-based knowledge is available for NNP implementation Technical support is provided to the EU and Joint programming in Nutrition

Enhancement of drought resilience and food and nutrition security of vulnerable populations in Southern and Eastern Ethiopia

Improve nutrition and dietary diversification practices for adolescent girls, PLW, and children under 5, with a focus on the first “1000 days” in 17 woredas in Ethiopia

Ownership of the programme by national authorities

Capacity building and technical support Communication for behavioural and social change Research and knowledge development Coordination between development partners Procurement of supplies

INP

UT

S

AC

TIV

ITIE

S

OU

TP

UT

S

OU

TC

OM

ES


manage future uncertainty and change13. At the national level, different policies were also developed,

including the 2013 and 2016 National Nutrition Programmes, seeking to transform the economic and

development trajectory of millions of children and their mothers by addressing food and nutrition

insecurity in the country, as well as the Health Sector Development Programme IV (2010 to 2015)

aiming to ensure all Ethiopians benefit from a secure and adequate nutritional status in a sustainable

manner.Additional policies include the Agriculture Sector Policy Investment Framework (2010 to

2020), where the objectives embody the concepts of ‘producing more, selling more, nurturing the

environment, eliminating hunger and protecting the vulnerable against shocks’14. The National

Disaster Risk Management (DRM) Policy aims to reduce the risks and impacts of disasters through

the establishment of a comprehensive and integrated DRM system within the context of sustainable

development. UNICEF’s Country Programming Paper focuses on arid and semi-arid lands. All these

different policies influenced the development of the nutrition component of the EU-SHARE project as

an integrated approach that combines nutrition, agriculture, household food security, social protection,

health and education sectors. A linkage exists with the project on Leadership, Coordination and

Sectoral accountability, chaired by the State Minister of Health, and funding ensured by the EU. Apart

from policies, other inputs include human resources, infrastructures and equipment.

The project’s activity areas included capacity building and technical support (training, supervision

missions, and quarterly review meetings, among others), communication for behavioural and social

change, research and knowledge development, coordination between development partners, and

procurement of supplies.

The project’s main outputs were structured around five key project results: (1) Adolescent girls, PLWs,

and children under 5 are reached by quality preventative nutrition interventions; (2) Households are

made aware of, and have increased access to, nutritious foods and practices; (3) Improved capacity

for NNP implementation in project areas; (4) Evidence-based knowledge is available for NNP

implementation; and (5) Technical support is provided to the EU and Joint programming in Nutrition.

The outcomes included: (1) the Enhancement of drought resilience and food and nutrition security of

vulnerable populations in Southern and Eastern Ethiopia; (2) Improved nutrition and dietary

diversification practices for adolescent girls, pregnant and lactating women (PLWs), and children

under 5, with a focus on the first “1000 days” in the 17 targeted woredas; and (3) Ownership of the

program by national authorities.

The program was implemented in a country context of socio-demographic, political, economic,

geographical, and cultural diversity among the targeted woredas, which may have influenced the

expected outputs and outcomes. Overall, the investment (inputs) tried to: foster an improved

nutritional status of children and pregnant and lactating women and better dietary practices; improved

management capacity of national partners; awareness and engagement of participating communities;

and ultimately ownership of the authorities. The ToC guided the development of our evaluation matrix.

13 Vogel, Isabelle (2012). Review of the use of ‘Theory of Change’ in international Development, UK Department of International Development. 14 Ministry of Agriculture and Rural Development. Ethiopia’s Agricultural Sector Policy Investment Framework (PIF - 2010 to 2020), September 2010.


Different indicators, extracted from the project’s log-frame, were identified to measure the changes

occurred in terms of outputs and outcomes.

3 EVALUATION APPROACH AND METHODOLOGY

3.1 Evaluation approach

With the premise that the evaluation should be judged by its utility and actual use, our team adopted

a balanced participatory approach, which helps to ensure that existing stakeholder knowledge is

shared and conclusions are verified, while on the other hand guaranteeing independence of

evaluation conclusions and recommendations. Those who have the most to profit from this evaluation

(i.e. UNICEF, FMoH, FAO, health facilities, donors, and beneficiaries of the EU-SHARE project)

played an active role throughout the process, particularly in shaping the strong learning focus of this

evaluation. As requested by the ToR, our team used a combination of secondary and primary

qualitative and quantitative data methods to generate the necessary information, basically organized

in two parts: (1) Household and community survey in the 2695 households approached in the baseline

survey, according to a set of nutrition performance indicators provided by the ToR, and (2) a

Qualitative evaluation, using documentary review,

semi-structured interviews with key stakeholders,

and focus group discussions with health facilities

staff, community workers, and beneficiaries.

Our team applied a realist evaluation framework to

inform and guide the development and conduct of all

stages of work. Realist evaluation is a theory of

change driven approach that goes beyond the

simplistic focus of input-output evaluation models by

also focusing on the underlying policy, social and

economic mechanisms and contextual factors that

lead to the success (or not) of a policy, or a

project15,16. Indeed, realist evaluation is rooted in the

understanding that not all interventions work in the

same way, in all circumstances and for everyone. In

other words, context matters.

15 Westhorp, G. (2014) Realist Impact Evaluation: An Introduction. Accessed at:: http://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-

opinion-files/9138.pdf 16 Marchal, B. et al (2012). Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research.

Evaluation; 18:192


3.2 Data collection and analysis

3.2.1 Evaluation matrix

The evaluation was guided by the evaluation matrix, presented in annex B, which is organized

according to the evaluation criteria requested by the terms of reference. The evaluation questions in

this matrix were inspired by the ToR, by the theory of change and by a first review of project

documentation. The matrix lists the key questions and indicators to measure, addresses the

assumptions underlying progress, and presents the data sources and the data collection techniques.

The findings in this evaluation report respond to the questions defined by the evaluation matrix.

3.2.2 Evaluation sample

The appropriate study design for this survey was cross-sectional, where the cause and the effect were

traced at the time of the survey. To determine the sample size, the team calculated the necessary

sample size for different combination of levels of precision, confidence, and variability. Because the

survey was an endline evaluation, the appropriate computational sample size determination formula

used was the following change detection formula:

In order to keep homogeny at the cluster level and to obtain a sufficient sample size for reporting at

the zonal or regional level, 2,695 households were sampled. In determining the final sample size, the

rounding-off of household numbers when dividing the number of household among the enumeration

areas was considered.

Table 1: Sample size determination

95% level of

confidence

(Zα/2)

80% power

of a test P1 1-P2 D P1-P2

(P1-P2)*

(P1-P2)

10% non-

Response

Final sample

size

1.96 0.84 0.5 0.5 1.6 0.05 0.0025 260 2695

n = [(D)( Zα/2+.84)2 (P1(1 - P1) + P2(1 - 2)] / [(P1-P2)]2

n is the required minimum sample size,

Zα/2 is a factor to achieve the 95% level of confidence (corresponding tabular value of 1.96),

The 80% power of a test whose tabular value 0.84 is used as an input for sample size determination

P1 is the anticipated proportion for the key indicator (proportion of households with good knowledge of nutrition) 50% at the onset of the project

P2 is the anticipated proportion for the key indicator (proportion of households with improved quality of life targeted 55% after onset of the project) i.e. expecting minimum of 10% change due to program implementation

D is the square root of design effect of 1.6, with assumption of 7-10% intra class correlation (ICC) of two respondents from within the same enumeration area

(P1-P2) is the acceptable margin of error between actual value and estimate from the survey (5%)


The number of clusters/kebeles per woreda was assigned based on the population proportion as

shown in table 2.

Table 2: Distribution of sampled households (HHs) with enumeration areas (EAs) per woreda, zone

and region

Region Zone Woreda # Kebeles # EAs HH/EA HH/Region

Amhara Waghimra Sekota 5 10

30 390 Ziquala 2 3

SNNPR

South Omo Dasenech 3 4

35 1,505

Hammer 3 4

Wolaita

Boloso Sore 6 12

Kindo Koyisha 4 8

Degunaanigo 4 7

Damot Pulasa 4 8

Oromia

Bale

Guradamole 2 2

32 800

Dawe Kachen 2 2

Rayitu 2 3

Borena

Dehas 2 2

Arero 2 3

Dillo 2 2

Moyale 2 3

Miyo 3 4

Dire 3 4

Total Sample Size 51 81 2695

The survey team visited the enumeration areas where baseline data collection was carried out and

the eligible households were then selected by simple random sampling techniques from updated

household list obtained from local administrative records.

With respect to the qualitative part of the evaluation, the following criteria were used to select the

sample of woredas to visit: (i) level of achievement of key activities (in 2017); (ii) implementation of

nutrition-sensitive, along with nutrition-specific, activities; (iii) presence of pastoralist and sedentary

populations; (iv) NGO supporting the project in the specific woreda (the team wanted to see field

activities of at least 3 out of the 5 NGO partners); and (v) accessibility.

The team planned to visit: Sekota woreda in the Amhara region; Dawe Kechen and Miyo woredas in

Oromia region; Damot Pulassa, Kindo Koysha, and Hammer woredas in SNNP region. However, due


to security issues, the team could not visit Oromia region and had also to forego the Hammer woreda

because of a time constraint (travel distance was too long).

The team mapped the stakeholders in the regions and woredas to identify key informants to be

contacted for interviews, such as government and non-government staff involved in program planning

and implementation from national, Woreda to Kebele levels. Focus group discussions were conducted

with caregivers (woman and men) of children under 5 years of age admitted to the program, as well

as pregnant and lactating women of different age groups. A purposive sampling approach was used.

A total of 30 stakeholders were interviewed and 3 FGDs were conducted. The list of participants is

presented in annex D.

3.2.3 Data collection

The data collection and analysis tools were designed to generate the evidence on the evaluation

issues and questions as set out in the evaluation matrix (annex B). The lines of evidence were

developed in sequence in order to gradually build up the knowledge base of the evaluation team and

facilitate the refining of the data collection instruments for each subsequent data collection task.

The findings of the household survey helped to determine the changes observed in project outputs

and outcomes (target indicators compared to the baseline survey). Given that the objective of the

qualitative component of the evaluation was to assess the “how” and the “why” of the achievements,

the interview and FGD questions were formulated accordingly.

The household survey: (i) assessed the level of malnutrition using indicators and anthropometric

measurements in the intervention zones and compared with the baseline; (ii) determined the current

level of knowledge, attitudes and practices (KAP) on key Maternal, Infant and Young Children feeding

(MIYCF) behaviours and compared with the baseline; (iii) assessed coverage of Vitamin A

supplementation, deworming, iron supplementation and Grow Monitoring and Promotion (GMP)

participation and compared with the baseline coverage; and (iv) identified key barriers and

opportunities to successful implementation and scalability of household and community gardening

support and other agricultural-nutrition linkages.

The quantitative survey data were collected using structured modular questionnaires through

interviews with male and female adults in each sampled household. The survey included modules

related to the indicators established during the baseline survey. The survey questionnaire was pre-

tested and refined accordingly, along with checking standardization on anthropometric

measurements. The data were collected by teams of supervisors and enumerators, who operated

simultaneously in their different regions, to increase the speed of the assignment.

Household data were collected through personal digital assistant (PDA) devices using the SMART

Standardisation and emergency nutrition assessment (ENA) software, as adapted by ACF Canada

for PDAs. All enumerators and supervisors with quantitative data collection experience were trained

on the specific ENA tools, mobile data collection, and quantitative interview techniques.

The qualitative part of the evaluation used documentary review, interviews and focus group

discussions. The evaluation team developed the interview and focus group guides for the qualitative

study after an initial extensive document review, the stakeholder mapping, and after the household


survey. The interview guides were customised according to stakeholder groups and pre-tested and

adjusted accordingly. Documents and secondary data reviewed were obtained from UNICEF, FAO,

the project NGO partners and Government counterparts. Interviews and FGDs sessions were audio

recorded and transcribed. Two teams were formed for qualitative data collection. Both teams worked

together in Addis Ababa, then split for field visits in the regions.

3.2.4 Analysis and reporting

The evaluation team created a SPSS database where all the survey data from the PDAs were stored

and cleaned. ENA software was used to analyze anthropometric measurements while SPSS software

was used to analyse the other data from the household survey.

Data triangulation and deduction were used to analyze the qualitative data. Information was

triangulated by comparing the multiple lines of evidence produced by collecting data from the four

data sources (household survey; document review; semi-structured interviews with project

stakeholders; and focus group discussions with caretakers). The data were sex-disaggregated and

coded under each evaluation criteria and key question.

3.2.5 Ethical considerations

The inception report methodology and tools were developed in line with the procedure of UNICEF’s

ethical standards in research, evaluation, data collection and analysis. Within this procedure, the

principles and requirements for evidence generation are applied to four core ethical issues, namely;

Harms and Benefits, Informed Consent, Privacy and Confidentiality, and Compensation and

Payment17.

In terms of Harms and Benefits, the data collection methods (interviews, FGDs) and the tools

developed (questionnaires, interview and FGD guides) did not have any negative repercussions on

the health and well-being of participants. The supervisors and enumerators were trained on cultural

and ethical considerations for interviewing young girls and boys during the household surveys. During

the anthropometric measurements, children and women identified as malnourished were referred to

the health facilities for appropriate care.

Informed verbal consent was obtained from all participants prior to conducting interviews and FGDs.

They were made aware of the voluntary nature of their participation, and their decision on whether to

participate, including dissent or unwillingness to participate, was respected. Parental consent was

obtained to measure the nutrition indicators among young girls and boys.

The evaluation team also pledged confidentiality to interviewees and FGD participants and promised

to not quote them directly unless permission was given. Direct identifiers (e.g. personal information

such as names and addresses) were removed, and survey data were securely stored, protected and

disposed of, by limiting access through password protection and restricting the number of staff who

17 UNICEF. UNICEF procedure for ethical standards in research, evaluation, data collection and analysis. Division of Data, Research and Policy (DRP),

April 2015. Document Number: CF/PD/DRP/2015-001. P8-12.


had access to the data18. As GPS data were collected during the survey, measures such as

scrambling of co-ordinates, de-linking of data or assignation of broader geographical references were

taken to ensure confidentiality.

Participants were clearly informed that there was no payment or compensation for participating to

survey, interviews or FGDs. This did not distort their decision to participate or the responses given.

3.3 Quality assurance

3.3.1 Overall quality assurance

The evaluation team used the United Nations Evaluation Group (UNEG) Norms and Standard for

evaluations (2016) and the UNEG ethical guidelines to ensure quality of end of project review

processes, as well as the norms established by the OECD-DAC and specified in its publications:

“Principles for The Evaluation of Development Assistance” (1991), and “Quality Standards for

Development Evaluation” (2010). In accordance with these, ACT for Performance has developed its

own quality assurance (QA) policy, which was applied throughout the delivery of this evaluation and

is based on: i) principles; ii) areas to be covered: and iii) process. The QA policy is realized in the first

place by hiring the best qualified consultants to conduct the work, and in the second place, on the full

involvement of its senior management.

The evaluation team ensured that the following four principles were followed throughout the delivery

of the evaluation: i) ensuring that ToRs are fully met and well understood by all participants to the

evaluation; ii) stakeholders’ full participation: inclusiveness, and ensuring that the evaluation is also a

learning exercise; iii) gender sensitive; and iv) efficiency and effectiveness of the process.

The team’s QA ensured that:

context was taken into account, based on appropriate data collected both within the project,

program and similar work performed elsewhere;

key stakeholders were involved in terms of quality, number, and sex;

stakeholders’ opinions were accurately and clearly captured;

the methodology used allowed for triangulation and incorporation of a wide variety of opinions

and was strictly followed during implementation.

international standards for evaluation procedures (UNEG, UNICEF) were applied throughout

the delivery of the evaluation.

the analysis is of high quality, based on evidence and addressing the theory of change;

the gender dimension was taken into account;

all reports are of a good quality and written clearly and concisely.

In terms of process, the QA was performed at first instance by the team leader, who is the CEO of

the leading firm, and who interacted directly with the Steering Committee. In addition, the Director of

JaRco Consulting and the Director of Evaluation of ACT for Performance reviewed all products before

18 Boddy, J., Neumann, T., Jennings, S., Morrow, V., Alderson, P., Rees, R., and W. Gibson (2014). The Research Ethics Guidebook: A resource for social

scientists, University of London, http://www.ethicsguidebook.ac.uk/.


they were sent to the client. A native English speaker did the editing of the reports and the PowerPoint

presentations.

3.3.2 Recruitment of survey teams

The data quality assurance for the household survey started with recruiting the right personnel for

each assignment. Recruitment was only required for the evaluation and survey experts and the

enumerators as JaRco already had the in-house capacity for logistics, drivers capable of reaching

across Ethiopia’s outer regions and refugee camps, and supervisors experienced in conducting

surveys using PDA technology.

To expedite the start of the assessment, and as a measure towards collecting quality data, the team

recruited enumerators that worked with JaRco before in the regions covered by the assessment and

who thus had a good knowledge of the locality.

Recruitment as a supervisor or enumerator was dependent on the aptitude demonstrated during the

training and the pilot survey (pre-test). JaRco developed detailed job descriptions for the data

collection teams based on the requirements of the rapid assessment activities to depict their day-to-

day duties and responsibilities during the fieldwork. These job descriptions were discussed at length

during the training.

3.3.3 Training and pre-test

Data collectors and supervisors were thus recruited based on having experience with household

surveys, anthropometric surveys and electronic data collection as well as knowledge of the local

languages and cultures. They underwent intensive training prior to endline data collection activities.

A five-day training workshop was facilitated by JaRco with support from ACT for Performance, and

covered relevant background on: EU-SHARE nutrition activities and topic areas; sampling methods

and data collection procedures; skills development in interviewing techniques and anthropometric

measurements; extensive review and discussion of study questionnaires with practice on PDAs; and

research ethics. Importantly, it included practical skills development regarding operation of PDAs and

anthropometric measurements. The training included a one day standardization test of anthropometric

measurements for enumerators and supervisors and a two-day field pre-test of the questionnaires

and the data-gathering templates (using the PDAs). These activities provided opportunities for

detailed feedback to enhance data collectors’ skills prior to actual data collection. An additional one

fifth (20%) of the required number of supervisors and enumerators took part in the training to provide

for possible replacements.

3.3.4 Field manual

The persons selected as supervisors and enumerators in the assessment were given a field manual

covering the details of all aspects of their role. The manual contained the following general

information:

Introduction: a guide to the objectives of the survey, the instruments to be used, the team

composition, sampling methods used, and a general guideline for visits;


Role of the supervisors: how to prepare the data collectors/enumerators at the start of the day,

guide them during the survey, and the checking that is required at the end of each day;

Lines of communication among the survey team;

How to conduct an interview: advice for the data collectors on how to get the best out of the

interview, including instructions for ensuring the interview is fair and accurate and how to re-

interest a respondent that appears to have lost enthusiasm;

Cultural and ethical considerations for interviewing young girls and boys;

Guide to the use of the PDA, anthropometric measurements, and ENA.

3.3.5 Data quality assurance during data collection

The team ensured that data were accurate, timely, complete, and had integrity. This hierarchy of

quality assurance is connected through a chain of communication and specific checks following

standard operating procedures, which comprised back-checking19; spot-checking20; check before

sending21; and “live” assessment of returned data22.

Household survey data were collected on PDAs using the SMART Standardisation and ENA software,

as adapted by ACF Canada for PDAs. Each team of enumerators and one supervisor with quantitative

data collection experience was trained on the specific ENA tools, mobile data collection, and

quantitative interview techniques. Data were collected on the PDAs, reviewed by supervisors, and

then submitted to the server for immediate review and feedback by the JaRco core team. JaRco

performed in-house verification of data, and communicated directly with the team supervisor if any

significant errors were detected.

A daily quality check of the anthropometric data was performed during field work. ENA software was

used, ensuring that outliers are flagged, low performing enumerators were identified, and corrective

action was taken. The ENA methodology developed was applied to PDA/tablets for data collection

and management.

All field interviewers recorded interviews verbatim and made notes of their observations including

non-verbal cues. Supervisors and monitors listened in on some of the interviews and provided

feedback on interview techniques, completion, probing, etc. At the end of each day, members from

the research team went through the transcript to ensure that the notes were accurate and complete.

19 Check during fieldwork. Once an enumerator has completed a survey, they handed their tablet to the supervisor, who then gave them an empty

device so the Enumerator could perform another survey while the supervisor checked their work, to ensure that the questions were filled in properly, that there are no unintended gaps, and that the information was stored and time-stamped properly.

20 The supervisor performed spot checks on the enumerators’ work during the first three days of their deployment. During the survey, the supervisor

sat in and monitored to ensure that the process of questioning was being conducted properly. The supervisor gave feedback to the enumerator at the end of the session. Any common errors were highlighted to the whole team.

21 Before sending the digital forms to the Data Manager in Addis Ababa at the end of each day or week, the supervisor performed an overview check

on the data files to ensure that the information was complete, formatted correctly and there were no unaddressed issues. The supervisor wrote a short summary report to describe where the data had been collected, how the checks had proceeded and any anomalies of which the Data Manager needed to be aware.

22 The Evaluation Data Manager checked the quality of the returned information (be it on a weekly or daily basis) using MS Excel, and was able to relay

messages to the Supervisors and the Assessment Team about the quality of the data quickly and efficiently. This prompt feedback mechanism allowed for the field teams to return to households where any errors occurred and rectify them.


3.4 Constraints and limitations

The main constraints were logistics and security. In two survey areas in Wolaita zone, a tribal conflict

occurred, which made it difficult for the enumerators to go house to house for the data collection. The

selected eligible household members were instead brought to the health post for the interview and

the anthropometric measurement.

During the household survey, some minor security issues occurred on the roads accessing certain

Kebeles in Oromia region, which were managed by smooth communication of the supervisors with

community members. However, the security situation in the region worsened just after the survey,

which ultimately caused cancellation of the qualitative study in the Oromia region.

Timely planning prevented delays to obtain government research authorizations.

4 Findings

After a socio-demographic description of the population surveyed, we have structured the findings

according to the questions in the evaluation matrix (annex B), and based on data extracted from

different sources such as the 2018 UNICEF consolidated progress report, the 2017 FAO progress

report, the regional health bureaus progress reports, the most recent NGO progress reports, the 2017

baseline and 2018 endline surveys, as well as individual key informant interviews and focus group

discussions conducted.

4.1 Socio-demographic characteristics of the population surveyed

The socio-demographic characteristics include sex, family size, marital status, religion, educational

status and occupation.

The majority (92%) of the houses visited for the interviews were headed by males. In Amhara, 17.9%

of the households were headed by females, which was relatively higher than in Oromia and SNNP,

where respectively 7.6% and 5.6% of the households were female-headed.

On average, 4 or more people lived per household in the three regions. The average family size was

4 persons in Amhara, 5 in Oromia and 5 in SNNP.

Most of the 2700 women (out of the 2695 participating households) interviewed during the survey in

the three regions were married (92.7%) and those who were separated or divorced were relatively

higher in Amhara (16.4%) than in Oromia (3.3%) and SNNP (2.3%).

The leading religion of the respondents in the three regions was Protestant (41.0%), followed by

Orthodox (26.7%) and Muslim (17.7%). Orthodox was the predominant religion in Amhara (99.5%)

while in Oromia, the main religion was Muslim (59.0%) followed by Wake Fetta (29.0%). In SNNP, the

main religion was Protestant (70.0%) followed by Orthodox (19.0%).

More than half of the respondents (mainly women) were illiterate in the three regions (59.1%). The

illiteracy rate was 68.6% in Oromia, followed by 65.7% in Amhara and 52.3% in SNNP. In Amhara

and Oromia, the proportion of women who completed grades 1-4 and grades 5-8 was relatively higher

than the other educational groups. In SNNP, the proportion of women who completed grades 5-8

(16.3%) was higher than those who have completed grade 1-4 (13.9%) and grade 9-10 (10.0%).


In the three regions the main occupation of the households was farming (60.7%), followed by

government employment (10.6%). From the regions, farming was the main occupation followed by

small scale trading (60.8% and 11% respectively) in Amhara while in Oromia farming was followed by

pastoralist (52.4% and 15.9%, respectively). In SNNP, farming was followed by government

employment (65.1% and 10.2%, respectively).

The respondents were also asked what the HH’s main source of income was during the last four

weeks. In the three regions, the income was mainly obtained from “sale of home-grown cereals/cash

crops” (26.4%) and “sale of home reared livestock” (26.0%), followed by “petty trade/small business”

(23.0%). The table 3 below summarizes these socio-demographic characteristics of households

interviewed in the three regions.

Table 3: Socio-demographic characteristics of households in the three regions

Religion of Respondents All (%) Amhara (%) Oromia (%) SNNP (%)

Orthodox 26.7 99.5 5.8 19.0

Protestant 41.0 0.3 6.3 70.0

Catholic 2.3 0.0 0.0 4.1

Muslim 17.7 0.3 59.0 0.4

Wake Fetta 8.6 0.0 29.0 0.1

Others 3.6 0.0 0.0 6.4

Education Status of Respondents All (%) Amhara (%) Oromia (%) SNNP (%)

Illiterate 59.1 65.7 68.6 52.3

Adult education (Church Education or

Basic Literacy Program) 0.3 0.0 0.4 0.4

Primary (grade 1-4) 12.5 11.3 10.5 13.9

Primary (grade 5-8) 12.9 6.9 9.4 16.3

Secondary (grade 9-10) 9.1 9.0 7.4 10.0

Preparatory (grade 11-12) 1.4 0.5 0.9 1.9

TVET 0.8 2.0 0.3 0.8

College/University 3.9 4.6 2.6 4.4

Household's Main Occupation All (%) Amhara (%) Oromia (%) SNNP (%)

Farmer 60.7 60.8 52.4 65.1

Merchant 5.6 4.4 5.8 5.8

Small scale trader 5.6 11.0 4.8 4.7

Labourer 7.5 5.4 6.0 8.9

Government employee 10.6 9.5 12.1 10.2

Student 0.2 0.3 0.0 0.3

Pastoralist 4.8 0.0 15.9 0.2

No occupation / housewife 2.4 3.6 2.3 2.3


Other 2.5 5.1 0.9 2.7

Household's source of income All (%) Amhara (%) Oromia (%) SNNP (%)

No income source 8.5 3.6 2.3 13.2

Sale of home-grown cereals/cash crops 26.4 40.5 6.4 33.4

Sale of home reared livestock 26.0 39.0 49.6 10.2

Sale of home reared livestock products 13.2 3.3 27.9 7.9

Sale of fattening animals 5.9 0.5 15.5 2.1

Sale of homemade crafts 1.2 1.3 1.0 1.3

Sale of firewood/charcoal 2.7 0.0 3.0 3.2

Waged Labour (not safety net) 15.0 11.8 12.0 17.3

Petty trade/small business 23.0 21.3 15.6 27.4

Loan 1.7 0.3 0.5 2.8

Remittance 0.9 0.8 1.0 0.9

Safety Net Labour 7.5 17.9 8.5 4.3

Employment/Salary 12.9 13.1 12.4 13.2

4.2 Relevance

Finding #1: The activities of the project are woreda-specific and designed according to

nutrition needs assessments in the targeted woreda.

According to the project documents and confirmed by the stakeholder interviews and the focus groups

with beneficiaries, activities are strongly focused on the most vulnerable households, selected by a

nutrition committee at kebele level after a screening, and more specifically on pregnant and lactating

women, on mothers with children under 5, and on adolescents. The project fostered awareness on

the importance of intervening during the 1000 first days of a child’s life, when nutritional gains are

critical for the child’s development.

The project adopted a multi-sectoral approach, with nutrition-specific and nutrition-sensitive (e.g.

home gardens, school nutrition clubs) activities, which was highly appreciated by the beneficiaries

and led to tangible results.

Awareness was created regarding feeding practices, including breastfeeding and preparing

complementary food. Several activities that were organized through campaigns, such as the provision

of Vitamin A, deworming tablets, and iron folate supplements, were mainstreamed into the routine

health programs.

In woredas with a pastoralist population, agricultural training and provision of inputs were focused on

goat and poultry production, while the targeted most-in-need families in woredas with predominant

agricultural production also received vegetable seeds and training for home-gardening.

The project showed flexibility, and adjusted its activities to emergency situations when these occurred

(e.g. in the Amhara Region) and managed acute malnutrition by providing therapeutic food products.

Finding #2: The Nutrition component of the EU-SHARE program is fully aligned with

government policies such as the Seqota Declaration, NNP II, and the nutrition-sensitive

agricultural strategy.


According to our interviews with other donors and with government officials, UNICEF is well respected

for its nutrition expertise, and participates as the UN representative in the National Nutrition

Coordination Body (ministerial level) and in the Nutrition Donor Partner Forum. UNICEF is also a

member of the Nutrition Technical Committee (expert level) and its participation in these forums

fosters the alignment of its programs with the government priorities.

The multi-sectoral character of the project is in line with the guiding principles developed by the SUN

movement23. The focus group discussions with the beneficiaries confirmed that nutrition is not seen

anymore as only a health issue, but also as an agricultural and educational issue. Inspired by the

project, the Ministry of Agriculture established a Nutrition department.

The fourth expected result of the project contributed to this alignment, aiming at the following:

“Evidence-based knowledge is available for NNP implementation”. Four main activities were planned,

including: (i) the identification and development of an operational research topic assessing the

implementation of a multi-sectoral approach; (ii) the development and implementation of a baseline

KAP assessment; (iii) conducting a nutrition situation analysis through mapping, causal analysis and

intervention gap analysis, and (iv) developing and implementing a routine monitoring system. All

activities planned under expected result 4 were completed successfully, as confirmed by progress

reports and stakeholders interviewed.

UNICEF assigned a Nutrition Specialist and an M&E Officer to support the baseline KAP survey in 32

woredas of Amhara and Tigray regions. In Amhara, the two targeted woredas (Seqota and Ziquala)

were also project’s woredas. The support provided by the Nutrition Specialist and the M&E Officer

included designing the survey methodology, selecting survey indicators and developing survey

questionnaires. The survey was completed in the two regions and the report was under development

during the evaluation period.

The National Nutrition Situation Analysis was also completed, and the report finalised. For the

development and implementation of a routine monitoring system, over 500 review meeting sessions

were conducted in the 17 project woredas, involving programme implementers, woreda nutrition focal

persons, HWs and HEWs. Discussions were held with HEWs on how to simplify the method of

recording and reporting data.

Two (2) of the 17 targeted woredas of the EU-SHARE program are part of the 32 innovation phase

woredas that are targeted by the Seqota Plan (Seqota and Ziquala in the Amhara Region). According

to the documentation reviewed, 12 from the 17 project woredas integrated Vitamin A supplies in their

routine programme.

Finding #3: The project mostly targeted pregnant and lactating women from vulnerable

households and empowered them through increased income as a result of the sale of eggs

and vegetables.

Progress reports and all interviewed stakeholders at regional and woreda level mentioned the positive

impact that the revenues that pregnant and lactating women earned from the agricultural support.

These women received 6 chickens, vegetable seeds (tomato, carrot, onion, and chard) and

23 https://scalingupnutrition.org/about-sun/the-sun-movement-strategy/


agricultural equipment such as digging tools and a watering can. Across the board, men accepted this

women-oriented approach principally because of the opportunity that the household could increase

its revenues. Training was provided to women’s groups on complementary food production and

processing in which men participated as well.

In Faya Kebele (Seqota woreda, Amhara region) the provision of seed did not happen due to difficult

weather conditions. In some kebeles of the Oromia regions, seed and chicken were not distributed

because of the fragile security situation.

Finding #4: The targeting approach of the project is considered relevant by all stakeholders.

However, in some kebeles tensions arose because neighbouring households were not

targeted by the project, and wanted to receive the same services and inputs as the targeted

households. However, after explanation of the project’s approach, focusing on the families of

the women most in need, people generally accepted this.

Through the Health Development Army, people are quite aware of everyone’s socio-economic

situation, which enabled the community to understand why certain households are selected as project

beneficiaries.

4.2 Effectiveness

The evaluation matrix (annex B) formulates seven questions for effectiveness. Question 7 about

“disparities in access” has been addressed together with the first question concerning “access to basic

services for the population”.

Finding #5: The following project interventions achieved most of the targets: (i) Vitamin A

supplementation; (ii) Deworming in children aged 23 to 59 months and adolescents; (iii) Iron

Folic Acid (IFA) supplementation among pregnant women; (iv) Screening of children and

pregnant and lactating women for acute malnutrition; and (v) Growth Monitoring and

Promotion (GMP) of children under 2 years of age. However, there were discrepancies

between progress report data and survey findings for Vitamine A supplementation, deworming

in children, screening of children and pregnant and lactating women, and participation of

children in GMP. According to progress reports, targets were achieved, while the surveys

reported limited or no achievements.

The project’s first expected result stipulated that “Adolescent girls, PLW, and children under 5 are

reached by quality preventative nutrition interventions”. These preventative nutrition interventions,

also called “basic services”, included Vitamin A supplementation, deworming in children 23 to 59

months and adolescents, Iron Folic Acid supplementation in pregnant women, screening of children

and PLW for acute malnutrition, participation of children under two years of age in GMP and promoting

complementary feeding practices. The key achievements concerning the first expected result are

described in the various figures below.

Vitamin A supplementation in children. The Vitamin A supplementation was implemented as part of

the routine health extension programme in 12 woredas and by means of community health days

(CHD) in 5 woredas. According to the 2018 progress report, an orientation training was given to 1,435

leaders of health development armies (HDAs) by implementing NGOs in collaboration with woreda

health officials, to raise awareness about the benefits of Vitamin A and the availability of nutrition

services at health facilities, as well as the benefits of deworming and how to mobilise the community.

According to this progress report, Vitamin A coverage reached 91% of children under 5 in Amhara,


88% in Oromia, and 98% in SNNPR, exceeding the target of 71%.However, survey findings showed

that these achievements were in fact lower. At baseline, only Amhara and Oromia regions achieved

the targets, with proportions of 85% and 84,9% respectively. These achievements drastically

decreased at endline, none of the regions achieving the target, which is contradictory to the project’s

expectations (see figure 2).

Figure 2: Vitamin A supplementation achievements among children

Deworming in children 24 to 59 months. According to the 2018 UNICEF progress report, training was

provided to 147 HWs/HEWs from the 17 target woredas to enhance their technical skills in periodic

deworming, and orientation was provided on the transition to the routine modality. According to this

report, deworming coverage reached 76% of children 24-59 months old in Amhara, 86% in Oromia,

and 93% in SNNP regions, exceeding the project target of 80%.

Figure 3: Deworming achievements among children 24 - 59 months

85% 85%

64%

74%71%

35%

64%

45%50%

91%88%

98%92%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Amhara Oromia SNNP All Target

Baseline Endline Progress report

66%

55%49%

53%

80%

19%

38%

22%27%

76%

86%93%

85%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%




However, baseline and endline survey findings were far below progress report observations, none of

the regions achieving activity’s targets. Moreover, there was a dramatic decrease in achievements

from baseline to endline, which was also contradictory to project’s expectations (figure 3).

Adolescent Deworming. Regular deworming contributes to good health and nutrition for children of

school age and adolescents. It also helps children and adolescents to avoid the worst effects of

infection even if there is no improvement in sanitation. According to the administrative reports from

Regional Health Bureaus in Amhara, Oromia, and SNNP regions 2017 and 2018, 83% adolescents

were taking deworming medication in Amhara, 76% in Oromia, and 94% in SNNP, exceeding the

project target of 30% (figure 4). During the baseline and endline surveys conducted at the community

level, adolescents were asked if they were dewormed in the past six months before the survey. At

baseline, these proportions were 51%, 39% and 63% in Amhara, Oromia and SNNP respectively,

higher than the target of 30%.. However we measured lower figures at the endline in the three regions,

Oromia region performing even lower than the target.

Figure 4: Deworming achievements among adolescents

Iron Folic Acid Supplementation for pregnant women. Iron and folic acid (IFA) supplementation during

pregnancy helps in prevention of anemia and improves the overall birth outcomes. According to the

most recent project progress report (2018), the percentage of pregnant women who received IFA

during ANC through HW/HEW counselling sessions reached 67% in Amhara, 73% in Oromia, and

56% in SNNPR region, exceeding the project target of 30%. The baseline and endline surveys

assessed consumption of Iron and folic acid supplements during pregnancy among women with

children under 24 months and pregnant women (figure 5). Although the proportions exceeded the

targets at both surveys, we observed again a slight decrease from baseline to endline (74% to 61%

in Amhara, 69% to 68% in Oromia, and 73% to 71% in SNNP).

51%

39%

63%

55%

30%

41%

17%

55%

44%

83%76%

94%

84%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%




Figure 5: Achievements in Iron Folic Acid Supplementation among pregnant women

Screening for Acute Malnutrition in children and PLW. All the woredas have undertaken screening of

children and pregnant and lactating women using mid-upper arm circumference (MUAC) tapes.

According to the most recent project progress report, NGOs provided training on community

management of acute malnutrition (CMAM) and on infant and young child feeding (IYCF) in

emergency situations to over 580 HEWs at the woreda level. On-the-job training was given regarding

proper screening and identification of cases of malnutrition, and on supply management.

Figure 6: Screening for acute malnutrition among children

According to the administrative reports from Regional Health Bureaus in Amhara, Oromia, and SNNP

regions 2017 and 2018, screening coverage among children under five years of age reached 57% in

Amhara, 81% in Oromia and 92% in SNNP. Compared to Oromia and SNNP, the Amhara region did

not meet the project target of 80%.However, the household surveys conducted at the community level

74%69%

73% 72%

30%

61%

68%71%

69%

67%

73%

56%

65%

0%

10%

20%

30%

40%

50%

60%

70%

80%



33% 31%

13%

22%

80%

39% 37%

19%

28%

57%

81%

92%

77%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%




showed very different and lower performance data, with a sligh improvement from baseline to endline

(33% to 39% in Amhara, 31% to 37% in Oromia, and 13% to 19% in SNNP), none of the regions

achieving the project’s target of 80% (figure 6).

With respect to PLW, the administrative reports from Regional Health Bureaus showed that 41% were

screened in Amhara, 97% in Oromia and 94% in SNNP. Amhara region did not meet the target of

80%. However, the surveys showed that none of the regions achieved the target of 80%, although

there were slight improvements in screening coverages in Oromia and SNNP regions from baseline

to endline (figure 7).

Figure 7: Screening for acute malnutrition among pregnant and lactating women

Children under 2 participating in GMP. Children admitted in the Productive Safety Net Programme

underwent monthly measurements and received at the same time counselling on appropriate

complementary feeding practices.

Figure 8: Children under 2 participating in GMP

71%

54%

29%

43%

80%

63%60%

53%57%

41%

97% 94%

77%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%



26%

5%4%

7%

50%

9%13%

7%9%

61%57%

71%

63%

0%

10%

20%

30%

40%

50%

60%

70%

80%




According to the most recent project progress report, all regions achieved the project target of 50%,

children 0-23 months participating in Growth Monitoring and Promotion (GMP) being 61% in Amhara,

57% in Oromia and 71% in SNNP. However, our survey results showed quite different achievements,

far below the progress report observations. In Amhara, only 26 % of children attended GMP, and this

participation decreased to 9% a year later at enline. In Oromia and SNNP, although the participation

slightly increased from 5% to 13% and from 4% to 7% respectively, this was very far from the target

of 50% (figure 8).

Implementing IYCF through appropriate counselling. Complementary feeding approaches such as

counselling and cooking demonstrations were used to disseminate IYCF messages to the

communities, and IYCF spots were developed with a focus on issues including: the importance of

early initiation of breastfeeding, exclusive breastfeeding, dietary diversity and proper complementary

feeding, as well as key hygienic practices while feeding babies. According to the most recent progress

report, three IYCF spots have been developed in Amharic and Oromiffa languages and are being

transmitted via regional/zonal FM radio stations. The spots have reached approximately 27 million

people. IYCF counselling cards and posters were developed and distributed to health posts in all

target woredas. Around 48,000 mothers of children under 2 received counselling on exclusive

breastfeeding, preparation of complementary food for children above six months of age, personal and

environmental hygiene, the importance of Vitamin A, and available nutrition services for themselves

and their children. Sensitisation activities were conducted at 47 schools with 134 teachers and 532

students of school nutrition clubs. Trends of main IYCF indicators versus targets are presented in the

Impact chapter of this report.

Overall, according to the project’s progress reports, the interventions cited above achieved their

targets. Health, nutrition, and agricultural advisors, as well as HEWs and AEWs interviewed during

field visits confirmed that access to basic services has improved.

However, four cases of discrepancies between progress reports and survey findings were identified.

Vitamine A supplementation targets, deworming targets in children, screening targets in children and

PLW, as well as GMP participation targets were achieved in all regions according to the administrative

reports from the regional health bureaus and progress reports, but this was not confirmed by the

surveys. In addition, the trends in Vitamin A supplementation, deworming in children and adolescents,

IFA supplementation in pregnant women, and attendance to GMP decreased from baseline to endline,

which were contradictory to project’s expectations.

HEWs, AEWs, Nutrition and Agricultural advisers reported that there was no disparities in accessing

the services. Equal access was given to different groups. Potential beneficiaries that were displaced,

such as pastoralists, were accessed through mobile outreach teams.

Additional preventative nutrition interventions included: (i) Strengthening the promotion of

complementary feeding practices in all 17 woredas, (ii) training AEWs and HEWs on the Productive

Safety Net Programme (PSNP)/nutrition linkages and (iii) strengthening women groups. The social

analysis and action (SAA) training was provided to HEWs, community leaders, and community groups


and the discussions motivated members to change negative social norms affecting the nutrition status

of children and mothers in the community24.

The project supported the FMoH, Ministry of Agriculture and Natural Resources (MoANR), and

Ministry of Livestock and Fisheries (MoLF) in the development of the nutrition-sensitive agriculture

strategy and training manual as well as nutrition-sensitive agriculture messages. UNICEF nutrition

specialists and woreda health officers provided on-the-job training to HWs and HEWs on a regular

basis to enhance their technical capacity.

Women’s groups were identified and trained on topics such as food production, processing,

preparation and demonstration, as well as business management and marketing. However, at the

time of the evaluation, none of these production units were established and operational. Reasons

explained by agricultural advisors were that these women groups were formed and trained only

recently and setting up production units and shops for selling processed complementary foods takes

time.

Table 4: Achievements in other basic service interventions

Strengthen the promotion of complementary feeding practices in all 17 woredas

According to the most recent progress report, training on social analysis and action (SAA), developed and

piloted by CARE Ethiopia, was provided to 235 HEWs to help them develop the required facilitation skills to

initiate community dialogues to assist community members to identify prevailing social norms and initiate

feasible actions to improve the nutrition situation in their respective kebeles.

A total of 322 cooking demonstration sessions were held at all target woredas (at health posts by HEWs

and at village level by HDAs) to improve mothers’ knowledge about proper complementary feeding.

Cooking utensils were procured for 345 health posts to help them conduct the demonstration sessions.

School platforms have been supported where nutrition clubs were established in five schools. The clubs

were provided with small media equipment and IEC/BCC materials. Orientation sessions on main nutrition

issues to be promoted at school level were provided to teachers and student club members. These clubs

were transmitting key nutrition messages through mini media on a regular basis.

Training AEWs and HEWs on the Productive Safety Net Programme (PSNP)/nutrition linkages

138 experts from the health, agriculture, labour and social affairs sectors received Training of Trainers (ToT)

on PSNP and nutrition linkages. This training was cascaded down to frontline workers such as HEWs, DAs

and AEWs.

133 HEWs (versus 427 planned, thus 31%), 674 HDAs (versus 589 planned, thus 114%), and 147 AEWs

received the training.

About 1,100 AEWs were also trained on good nutrition practices.

Strengthening women groups

10 women’s groups were identified and established in 10 Woredas in the urban areas, trained on local

production of complementary food and provided with start-up capital for establishing the complementary

food processing units in each of these 10 woredas.

Beneficiaries included 885 women in Amhara, 800 in Oromia and 1,147 in SNNP regions.

24 AMREF – Health Africa (18th -20th March 2018), Program monitoring report, Integrated Nutrition Services: Multi-sectoral interventions to improve nutrition security and resilience.


These late activities were planned for implementation in the second half of 2018 and prior to the

project’s phase-out. As per the 2018 UNICEF progress report25, main achievements observed in these

additional interventions are presented in table 4 above.

Finding #6: The quality of extension services improved in the project woredas. Health

extension workers (HEWs) and Agriculture development agents (ADAs) were trained to

understand the multiple benefits of eating a variety of foods and to gain the knowledge and

skills to reduce post-harvest losses, and to improve food availability during times of

emergency crises. Regular supportive supervision visits were conducted by nutrition experts

appointed by FAO and UNICEF to monitor the implementation of the program at the beneficiary

level and ensure the quality of the work of the HEWs and ADAs, in spite of a high turnover of

these agents. Thus, the project solidified the multi-sectoral approach that UNICEF promoted

over the years.

The project’s third expected result stipulated that “Capacity for NNP implementation in project areas

is improved”. The seven activities implemented to achieve this expected result included: (i) the

development of materials for the agriculture sector; (ii) training of HEWs and ADAs agents on nutrition

and agriculture linkages; (iii) training of the target population in post-harvest management, food

preparation and processing techniques; (iv) technical support to MoA and BoA to strengthen multi-

sectoral coordination; (v) integrated supportive supervision (ISS) and review meetings on NNP

implementation; (vi) mobilization of women development army team leaders to support nutrition

interventions; and (vii) nutrition coordination meetings at zonal

and woreda level, with active participation of agriculture

representatives.

Thus, the project put a lot of emphasis on training to achieve this

result. To effectively mainstream nutrition into agricultural

programmes at the grassroots level, HEWs and ADAs were

made aware of the importance of collaborating to achieve

optimal results. Information from the documents and the

interviews indeed confirmed the good multisectoral

collaboration between these agents as one of the key outcomes

of the project. Given the high level of post-harvest losses in

Ethiopia (up to 30%), and given that project woredas are

regularly affected by drought and food shortages, HEWs and ADAs were equipped with the skills to

reduce post-harvest loss.

Main achievements observed are described in table 5 below.

Table 5: Achievements concerning project expected result 3

Development of

materials for

agriculture sector

According to the FAO progress assessment report (2017), FAO conducted three consultative capacity needs assessment workshops in 2016 to identify the capacity needs and priorities of the GoE and other institutions to mainstream nutrition into agriculture policies and programmes.

During the process of developing the NSA training manual, training materials from different

25 UNICEF (2018), Consolidated annual progress report to the European Union: Integrated Nutrition Services Multi-sectoral Interventions to Improve Nutrition Security and Strengthen Resilience — Joint Programme with FAO. CRIS 2014/348-501

“We ensure quality by making sure that the people who deliver the training have the skills to do so. We conduct pre- and post-tests. And we also ensure that those who have difficulties get the necessary support for improving their skills”.

NGO Program Coordinator, Wolayta Zone.

“We support the HEWs to maintain standards when performing various activities. For example, with supportive supervision we have elaborated a checklist. We use it to identify gaps and limitations, then we address these problems during review meetings”.

Health worker, Sekota Woreda, Amhara Region.

.


partners were collected. The technical working group agreed on the content and drafted an outline, ensuring that it will take into consideration the different categories of development agents. The draft training manual was shared with the working group for comments. It is expected to be finalised after the last pre-test and validation workshop to be held November 8, 2018.

By the end of 2017, following a formal request from the GoE, FAO and EPHI supported the first steps for the elaboration and implementation of food-based dietary guidelines (FBDG), which include a regional capacity-building component aligned with the nutrition-sensitive agriculture strategy.

Training HEWs

and HDAs on

nutrition and

agricultural

linkages

According to the 2018 project progress report, a training was conducted to introduce the principles of nutrition-sensitive agriculture and show how those principles can be applied to existing agriculture and food security programmes. The training also emphasised a close working relationship between HEWs and ADAs.

A total of 680 HEWs and 1020 HDAs were trained on nutrition and agricultural linkages. Training was provided to HEWs and ADAs on NSA-IYCN (infant and young child nutrition) in

all three regions.

Training of

farmers in post-

harvest

management,

food preparation

and processing

techniques

According to the 2018 project progress report, training on improved production and post-harvest management (crop production, methods for improved livestock rearing, milk production) was provided to 326 farmers in Amhara, 881 in SNNP, and 555 in Oromia.

During endline survey, 15.4% of HHs in SNNP region confirmed they received such training. These proportions were2.3% in Oromia and 1.8% in Amhara region.

Regarding the application of post-harvest loss reduction training, the endline survey showed that 87.5% of HHs that received the training in the three regions confirmed that they applied it, with 88.4% in SNNP, 85.7% in Amhara and 77.8% in Oromia.

Support MoA and

BoA with

technical

assistance

According to the FAO progress assessment report (2017), FAO nutrition experts (one at the national level and one for each of the three regions) have been supporting the sector in the development of the NSA strategy and delivered on-the-job support to build the technical capacity of agriculture experts at woreda agriculture offices and AEWs.

They assisted the Bureaus in identifying the appropriate variety of seeds, seedlings, and small ruminants to be distributed to beneficiaries.

Integrated

supportive

supervision (ISS)

and review

meetings on NNP

implementation

As per the 2018 project progress report, UNICEF, through its five implementing NGOs, put in place a monitoring mechanism to strengthen the implementation of the project at the grassroots level.

FAO delegated the overall supervision of regional project activities to its field officers (based in the BoANRs).

In the non-NGO supported woredas, through agreements with the BoANRs and BoLFs, the regional, zonal and woreda health and agriculture offices received financial support to conduct supportive supervision visits from zone to woreda level, and from the health centre to health post level.

Joint monitoring and supervision visits were conducted by UNICEF, FAO, implementing NGOs and woreda health and agriculture experts.

Implement

nutrition

coordination

meetings

According to the 2018 project progress report, seven (7) coordination meetings were held in each woreda, with a 50% participation rate of agriculture representatives (versus a participation target of 80%).

Overall, activities implemented under expected result 3 did not have specific targets. They were

performed as planned, and managers and beneficiaries interviewed were generally satisfied with their

implementation.


Finding #7: Factors that contributed to the project outcomes included: basic services

implemented as routine activities; linkage between nutrition and agricultural activities,

including a strong collaboration between HEW and ADA; economic empowerment of targeted

women through productive activities; the use of the SAA method to foster behavioural change;

the cascade 1-30/1-5 model of community leadership; the establishment of school nutrition

clubs; the regional taskforce; and a good collaboration at EU level between ECHO and DEVCO.

UNICEF seconded a nutrition consultant to support the EU’s engagement with Ethiopia’s

nutrition sector, who actively participated in all national nutrition coordination platforms,

resulting in a better understanding of the nutrition situation in the country by member states

and increasing their interest in engaging in nutrition interventions.

According to the document review and confirmed by the

stakeholders interviewed, several factors, described below,

contributed to the successful achievements of the interventions:

Basic services implemented as routine activities. Previously,

basic services such as deworming and Vitamin A

supplementation were delivered through campaigns only.

During the implementation of this program, basic services were

implemented as routine activities in existing health services in

the 17 woredas, and thus aligned with government priorities

and targeting the right beneficiaries, which has also improved

access to services.

Multi-sectoral approach linking nutrition, health and food production interventions. At different levels,

there has been a shift in mentality within government staff and the community on the relevance of

agriculture in nutrition. HEW and ADA used to implement their activities separately, but they are

currently working together. Productive Safety Net Programme activities were implemented

concomitantly with antenatal care, deworming, and Vitamin A supplementation. Beneficiaries of health

services were also beneficiaries of seeds and equipment for homestead gardens and the distribution

of poultry (mainly chickens); the entry point to the program is a household with a malnourished child.

Strong collaboration between HEW and AEW in the Kebeles. The high commitment of these two

categories of workers enhanced integration of health and agriculture sectors in the three regions. For

instance, cooking demonstrations promoted the use of diversified food (available in local markets or

produced in homesteads) and was linked with information about good hygiene and sanitation.

According to our interviews with NGO nutrition coordinators, as well as government nutrition and

agricultural advisors, better integration was achieved in SNNP and Amhara than in Oromia region.

Economic empowerment of women. Targeted women learned how to use the products yielded from

their backyard gardens and how to consume and sell the eggs produced by chickens they received.

The extra production (beyond the needs for their own consumption) from their backyard gardens and

the extra eggs from their chickens were sold and several of them were able to save money, to

purchase sheep and goats, and to invest in corrugated iron roofs. The intervention economically

empowered these women and their households and enabled them to financially access services they

were not able to access before.

The use of the SAA method to foster behavioural change. Women leaders were trained on the

application of SAA norms. Along with HEWs, they got familiar with initiating community conversations,

“In the past there were lots of children and people who were malnourished. Now, not only are our children healthy, but we also have the knowledge on what to feed them and ourselves properly…We have been economically empowered. AEWs don’t only give us the seeds and leaves; they come to our house and show us what to do to get the best crops. If there is a drought, they help us to get water for our plants. We can sell and make profit from the extra yield….We know how to keep our families healthy, and we can see now that there are less children who are sick because of malnutrition”.

FGD with PLW in Damot Pulassa.


which made it easier to quickly start social norm discussions. They visited the households to work

with mothers of children aged 0-2 months26.This made communities more easily accept the

interventions, to apply what they have been taught (feeding practices), and to change their behaviour.

The cascade 1-30/1-5 model of community leadership. Leaders were trained, and they trained other

leaders in remote areas, a cascade model that guaranteed a wide dissemination of key messages

and enhanced community awareness on the existence of services. This also enabled establishment

of SAA groups and ran the discussions. Participation in health post activities increased.

Establishment of school nutrition clubs (composed of teachers and students). Sensitising school

teachers and students on IYCF and deworming promoted the dissemination of messages to

classmates and parents, along with the establishment of a school garden, and adjustment of the

curriculum.

Existence of a regional task force (Regional Disaster Risk Management supervised by the Oromia

Agriculture Bureau). This task force oversaw the implementation of activities from regional to woreda

levels, providing information to households through the HEWs and ADAs and mobile health teams.

Good collaboration at the EU level between ECHO (Emergency Department) and DEVCO

(Development Department), as well as good collaboration between UNICEF and FAO, made federal

engagement possible.

Engagement at the national level was also fostered by the project’s expected result 5 which stipulated

that “Technical support is provided to the EU and Joint programming in nutrition”. UNICEF seconded

a nutrition consultant to support the EU’s engagement in Ethiopia’s nutrition sector and to provide

technical support to EU and member states on strategic and technical nutrition issues in line with the

Joint Programming in Nutrition. The consultant actively participated in all national nutrition

coordination platforms, played a vital role in drafting the EU joint Nutrition Programming Framework

and organised roundtable discussions on nutrition issues. Her work fostered a better understanding

of the nutrition situation in the country by member states, and increased their interest in engaging in

nutrition interventions. For example, the Italian Development Cooperation set up a technical working

group on nutrition in pastoralist areas and made its value chain programme nutrition sensitive. The

Norwegian Development Fund requested technical support to incorporate school nutrition and a

nutrition education component into its existing agriculture interventions, and the NGO Farm Africa

decided to include nutrition interventions in its livestock project.

In the same line, FAO seconded a nutrition consultant to the MoANR to support the nutrition case

team – now called the Food and Nutrition Coordination unit – in leading the implementation of the

2016 Nutrition-Sensitive Agricultural Strategy (NSA). The consultant participated in national nutrition

fora, supported the finalisation of the NSA training materials, and participated to ensure proper

implementation of FAO agreements with MoANR and MoLF. The activity planned under expected

result 5 was completed successfully.

Finding #8: Constraints in accessing basic services identified during the evaluation included

(i) delayed implementation; (ii) weak coordination between administrators at zonal and woreda

levels; (iii) recurrent drought; (iv) water shortage for drinking and for vegetable production; (v)

26 CARE. Trip report, Program monitoring for CARE supported activities of the EU-SHARE project in SNNPR, 2018, page 3.


heavy flooding and landslides; (vi) tense security situation in some woredas of the country;

and (vii) high workload of the HEWs.

The project faced several challenges during its implementation:

Late implementation because the planning did not sufficiently consider the time required for preparing

project implementation. The project took around 1.5 year to effectively start due to long administrative

procedures regarding agreements, budgets, contract

processes with implementing agencies and recruitment of

staff within the organizations. For example, NGO managers

reported that getting the agreement from the Regional Health

Bureaus to implement the project took more time than

expected, given long time periods dedicated to the different

focal points of the Regional Health Bureaus to read and

amend the project document. As per the UNICEF and FAO

managers interviewed, it also took time to UNICEF and FAO

to define the division of labour and to identify and hire

appropriate technical staff to manage the project (some

planned staff were not hired at all).

In addition, the unforeseen emergency occurrence in 2015,

for which more than 80% of the woredas were affected,

resulted in a considerable adjustment of the project’s content

in order to respond to the humanitarian situation, whilst at the

same time ensuring that the activities continued contributing

to the overall objective and expected results of the project. As

a result, the implementation had to be interrupted for almost

a year until the amendment was prepared and approved.

The lower than expected financial absorptive capacity of 3 out of the 5 implementing partners (in

terms of ensuring the timely implementation of activities and utilisation of the resources) led to a

revision of the budget. The funds transferred to the IPs were reduced by UNICEF. FAO underspent

as well and had to enter into an agreement with additional partners such as the Ethiopian Institute of

Agriculture Research and national NGOs specialised in livelihoods and resilience to accelerate the

implementation of delayed activities. At the moment the evaluation took place, just after these

adjustments occurred, the team was not able to assess the effects of these measures.

Weak coordination between administrators. Coordinating all EU-SHARE activities with zonal and

woreda officials involved in the multi-sectoral approach (EU-SHARE covers 15 sectors) was perceived

as challenging by interviewees. Woreda administrators had difficulties in organising monthly

coordination meetings and quarterly review meetings as recommended, because of competing

commitments among the participants.

Recurrent drought experienced in Amhara and Oromia regions caused programmatic adjustments.

The productive activities were reconsidered, and addressing the emergency became the priority.

HEWs and ADAs had to shift their priorities as well, which caused delays in the complementary food

production activities. No complementary feeding unit was established.

“Getting appropriate staff and maintaining them in rural areas is challenging. Since the 2016 drought there has been competition for hiring qualified people. There are several organizations looking for people to fill the same roles, and it proved challenging to be competitive… The locations of the woredas are very remote, with no electricity and other basic services… Many people don’t want to live in those areas. It is difficult to retain the staff”.

NGO Nutrition coordinator, Addis Ababa.

“Working with different government agencies through a multi-sectoral approach is difficult to coordinate. Different agencies have different priorities. After active awareness and (a) series of workshops, they were now getting (a) better understanding of their roles and they were progressively attending meetings and forums”.

Woreda administrator, SNNPR.

“Because of the drought, we moved away from the routine activities. A big part of the budget went into the emergency activities, although project’s objectives remained the same”.

Program Coordinator, Addis Ababa.


Water shortage for drinking and for vegetable production. For example, seven out of nine villages

faced serious water shortage in one Kebele of the Sekota woreda, Amhara region.

Heavy flooding and landslides caused a displacement of communities

for some time in regions such as SNNP. Cars could not access the

areas, and donkeys had to be used instead to transport the supplies

when possible. HEWs could not visit the households.

Tense security situation in some woredas of the country. In Borena

zone (Oromia region), conflicts between woredas made it difficult to

reach the beneficiaries. Project interventions stopped for two months

as internal displacement of the population in the area prevented the

government from conducting routine nutrition activities, and UNICEF

and FAO from undertaking field missions. There were similar situations

in Wolayta zone of SNNP region.

High workload of the HEWs made it difficult to implement all activities properly. These agents

appeared to be overloaded and could not therefore perform regular screening. At the time of the

evaluation, the government started hiring additional HEWs (second generation HEW program) to

reduce their workload.

Finding #9: The project had a few unanticipated effects: (i) the large quantity of chickens in

the targeted woredas as a result of the distribution of chickens combined with the training on

raising poultry; (ii) identifying “project goats” with yellow tags distributed to households

stigmatised the families receiving livestock support; (iii) several HEWs mentioned being

pressured in producing monthly reports with false data on screening activities that did not

reflect the reality of woredas in terms of cases of malnutrition.

FAO focused more on poultry production than on other livestock, which led to the large number of

chickens raised in the targeted woredas. Many households exchanged part of their chicken population

for goats and sheep. During interviews and in focus group discussions, informants reported that goats

distributed to vulnerable families by the project had large yellow tags on their body, indicating that the

family was under support. This sometimes brought tension and discrimination felt by children as those

from wealthier families were able to easily identify the ones that were receiving support.

The top priorities for care in the woreda are pregnant and lactating women, and children under 5.

Reports on screening of these two categories of the population should be produced monthly by HEWs

and sent to the MoH. The more cases they screen, the more support they receive. However, it is not

feasible for HEWs to perform screening every month in the woreda because of high workload and the

large area to cover. Given that the food supports the woreda get depend on the production of the

monthly report, HEWs are under pressure and inclined to put random numbers in the report. These

false data indicate high rates of malnutrition in the woreda, which do not reflect reality.

Finding #10: Targeted pregnant and lactating women, adolescents, and households with

children under 5 became very engaged in the process, as confirmed by the focus groups

discussions with them. Women and male farmers and ADAs were trained on homestead

gardening in the three regions. More than two out of five (44%) of vulnerable households

received agricultural inputs for backyard gardening.

“Because the frequency of drought is very high, the community needs to have different coping mechanisms... We should teach them on the importance of savings, which is the major coping strategy, next to behavioural change in feeding practices…Training needs to be scaled up, along with effective counselling campaigns, communications and workshops.”

Nutrition Officer, Oromia regional health bureau.


Messages to promote dietary diversity were developed for ADAs and HEWs for use in their

sensitization activities. They transferred their knowledge to farmers on new technology and

skills, and provided market-oriented information and advisory services.

Demonstrations on how to grow vegetables and fruits were conducted in schools in the three

regions. FAO teamed up with the Ethiopian Agricultural Institute to enhance the effectiveness

of the seeds used for complementary foods.

According to the UNICEF consolidated progress report of May 201827, 4 jobless youth groups were

organised in cooperatives for poultry, fishery, and horticulture activities, and half of these groups were

provided with start-up capital and agricultural inputs. However, in the financial report for March 2018,

no expenditures were mentioned for the youth group component.

The second expected result stipulated that “Households are made aware of, and have increased

access to nutritious foods and practices”. Key activities included: (i) the identification and support of

vulnerable households with specific interventions such as homestead gardens and livestock; (ii)

procurement and distribution of nutritious seeds and farm tools and livestock/poultry for backyard

production; (iii) supporting jobless youth groups (organised in cooperatives) in 8 woredas on fishery,

poultry production, community gardening and fruit production; (iv) developing messages to promote

dietary diversity and demand creation appropriate to media, schools and farmer training centers; (v)

conducting social mobilisation for dietary diversity and homestead gardens through demonstrations

at the farmer training centers and schools; and (vi) promoting and providing quality seed for production

of nutritious foods (fruits, vegetables and root crops) at farmer training centers. Main achievements

of these different activities are described in table 6 below and are mostly taken from the 2018 UNICEF

consolidated progress report.

Table 6: Achievements concerning project expected result 2 Identify and

support

vulnerable

households with

specific

interventions

In Amhara region, 515 women were trained on homestead gardening while 60 AEWs and 266 farmers were trained on forage production, locally available feed resource utilisation, and feed conservation methods.

279 farmers (mostly women) were given a refresher training on homestead gardening and 92 kebele experts were trained on food processing and preservation.

In SNNP region, 1,147 AEWs and farmers were trained on homestead gardening, and 881 AEWs and farmers were trained on proper handling and feeding of poultry and milk-producing animals.

In Oromia, 800 women were trained on home gardening.

Procurement

and distribution

of nutritious

seeds and farm

tools and

livestock/poultry

FAO supported 13,000 vulnerable households through agricultural inputs provision for backyard gardening, which represented 44.8% of the targeted HHs and achieved the expected target of 20%. Among these 13,000 HHs, 4,000 (13.8%) received cereals, vegetable seeds, fruit seedlings, pullets, and tools in Oromia region while 9,000 (31%) received orange fleshed sweet potato cuttings for their household consumption as well as forage cuttings for their livestock in SNNP region.

One specificity for Damot Pulasa and Boloso Sore woreda in the SNNP region was that all beneficiaries were from female-headed HHs.

Support jobless

youth groups

14 jobless youth groups were organised in cooperatives for poultry, fishery, and horticulture activities. The project target was achieved in this regard.

27 UNICEF, (2018), Consolidated annual progress report to the European Union: Integrated Nutrition Services Multi-sectoral Interventions to Improve Nutrition Security and Strengthen Resilience — Joint Programme with FAO. CRIS 2014/348-501


Half of these groups were provided with start-up capital, agricultural inputs and trainings, which improved their business skills and their knowledge about local food production.

Development of

messages to

promote dietary

diversity

Messages to improve the knowledge and practice of communities on dietary diversity were developed and used by AEWs at farmer training centres (FTCs) and by HEWs at school clubs in all the target woredas.

Nutrition-sensitive agriculture messages were developed and broadcasted via regional TV and radio stations, in collaboration with FMoH, MoANR and MoLF.

Conduct social

mobilisation for

dietary diversity

and homestead

gardens

Demonstrations on how to grow vegetables and fruits were conducted in 4 schools and 10 FTCs in Amhara region, 2 schools and 2 FTCs in SNNP, and in 6 schools and 10 FTCs in Oromia region.

Orientation workshops on school gardening were organised, with two teachers and 10 students participating from each school. The schools started preparing vegetable gardens, using the technical inputs received from the workshops.

Promote and

provide quality

seed for

production of

nutritious foods

The FTCs (which had developed germination centres) were followed up and supported by AEWs to ensure that the demonstration on vegetables and fruits were good models for farmers in the catchment areas.

AEWs transferred knowledge to farmers on new technology and skills, and to provide market-oriented information and advisory services.

As noted, interventions planned under project expected result 2 were mostly activity- (process) based

and descriptive, without specific targets. Most of the activities planned were implemented, and

nutrition and agricultural advisors, HEWs, ADAs and

beneficiaries interviewed were generally satisfied with their

quality. The exception was seed supply, which was

criticized in several woredas due to delayed supply or no

supply at all.

Two activities planned were not yet implemented at the

time of the evaluation: the promotion of milk- based

complementary food for children in pastoral areas of

project targeted woredas, and promoting and improving

access to quality milk for pregnant and lactating mothers.

Moreover, quantitative data were not available to assess

five indicators designed in the project’s logframe under the

expected result 2: (i) households with vulnerable children

to malnutrition producing at least 2 new types of

vegetables; (ii) households with vulnerable children to

malnutrition that increased their production and

consumption of milk and eggs; (iii) number of months per

year in which foods from own production are available in

the household; (iv) health workers trained on blended

materials; (v) children in the semi-urban target areas that received processed complementary food

from the processing units.

4.3 Efficiency

Finding #11: The project seriously underspent its available budget. According to the latest

financial report at the disposal of the evaluation team (August 17, 2018), a few months before

closure, the project spent €7.12 million, 66.7 % of the total programmable funds available

(€10.67 million). The interviews with the implementing organizations also confirmed that it will

“The major difference that we have seen now is a change in community awareness activities. The difference is that before farmers sold in the market all vegetables and fruits they produced... Now, due to the community awareness activities they are encouraged to consume them first, before selling the extra yield”.

Office Head, Kindo Koisha Woreda Agriculture Office.

“This program has reduced the number of families with issues of malnutrition. Each year we can see the levels decreasing due to the community awareness activities, household cooking demonstrations and the encouragement to use the crops they produce”.

Nutrition focal person, Oromia regional health bureau.

“Chickens have been distributed to families. Eggs are expensive in this kebele. Once produced these families first feed themselves and then sell the extra to buy other foods they do not grow at home… Now there are less malnourished children because of this community awareness on what to eat and the economic empowerment through livestock given to them”.

HEWs, Damot Pulassa.


be impossible to execute the remaining of the programmable budget in the last few months of

the project.

74.5% of the budget available for technical support (€0.94 million), was used, with a shift in TA from

FAO to UNICEF. FAO didn’t hire the planned expert in food security and nutrition to be seconded to

the Ministry of Agriculture and only used a small part of the budget available for short-term technical

expertise to regional bureaus of agriculture. UNICEF spent considerably more funds than planned on

the technical support to the EU-delegation and on their own Nutrition specialist.

Of the budget available for contracts (PCAs) with implementing organizations (€5.64 million), only

39.9% was spent. All planned activities, including the training components executed by the partner

NGOs, underspent, or were not executed at all at the moment of the evaluation (e.g. support to jobless

youth on milk and poultry production). The NGO partners CARE and ACF were able to fully execute

their available budget.

The activities with the government, such as training of HEWs and DAs, the quarterly review meetings,

and more general mobilization of government, did expend 90% of the available budget of €1.30

million. The expenditure data show a shift from training (for instance of HEWs on Nutrition-Agriculture

linkages) and the build-up of information systems to supervision, quarterly review meetings, and iron

folic acid supplementation.

The supply component (deworming tablets, seeds, farm tools, chickens) used 55.0% of the available

budget, (€1.68 million). This gap was principally caused because seeds, poultry and tools for back-

yard production, seed for farmes-training centres and seed for vulnerable households (emergency

response with early-maturing and drough-resistant seeds), were not procured, and local production

units were not set-up. On the other hand the project did provide, especially in 2018, more than two

times more in-kind start up capital to women groups than planned, in order to start complementary

feeding units.

A large part (84.5%) of the budget available for monitoring and evaluation and studies, €0.78 million,

was used at the moment of the evaluation, and the project overspent (116%) the budget available for

general operational expenses, because of additional overhead costs (compared to original budget)

such as office rent, vehicle maintenance, and administrative personnel.

The overhead costs were 18% of the total program expenditures (11% general operating costs and

7% indirect operating costs which is a contribution to UNICEF’s and FAO’s HQs). It is important to

note however that the PCAs with implementing organizations also include overhead costs. For

instance, the overhead of IMC is 23% in the PCA signed with UNICEF.

Several reasons were mentioned by the implementing partners for the underspending:

A long period to prepare the project documents and sign the contracts with the implementing

organizations, who had only 2 years left to implement the activities. Regional and woreda

governments had a say in the preparation of these contracts and their reviews took time.

A readjustment of activities to address emergency situations (i.e., drought in 2016; security

issues in Oromia in 2018) in several woredas.

Lack of flexibility of UNICEF’s PCA framework: funds can be moved from one budget

component to another, but it is not possible to initiate new activities, according to the

implementing partners. On the other hand, UNICEF mentioned that no formal request was

received from the partner NGOs to initiate new activities. The PCA framework has room for


new activities within the objectives of the PCA. The approval of budget adjustment requests

took some time.

A long delay (several months) between the request for payment sent to UNICEF and the actual

payment of funds, which caused delays in project execution, especially in 2017. This situation

has since been improved.

Difficulties in recruiting long-term staff and short-term experts, because the demand for

nutrition expertise was actually higher than the availability, in particular from the 2016 drought

period.

Delays in supplies, especially of seed and agricultural equipment, particularly in SNNPR and

Amhara regions.

Finding #12: The project did not have a project coordination unit with representatives from the

implementing agencies, which in a few cases (i.e., training of HEWs and DAs) led to NGO

activities overlapping with government activities, according to the stakeholder interviews. On

the other hand, the NGOs met on several specific occasions and their communication and

behavioural change approach was harmonized (using the SAA approach). The project

provided continuous technical backstopping to the agencies.

Effective coordination took place in the PCA preparation process; several coordination meetings were

organized which led to a division of woredas and specific activities, based on comparative

advantages. During the SAA training sessions provided by CARE, UNICEF organized coordination

meetings with the five implementing organizations, government staff, and FAO. PSNP-Nutrition

linkage training was provided by Concern and UNICEF. IMC worked on the production of ANC-

nutrition counselling materials, and several consultative workshops were organized with the

implementing partners.

Notwithstanding these encounters between the implementing agencies, there was hardly any

operational exchange between them, including FAO, on such areas as effective approaches, common

challenges and how to deal with these, unit cost benchmarking, economy of costs, etc. All five PCAs

applied the same perdiem policy. However, the difference between this policy and the perdiem policy

of the NGO itself, led to questions within the implementing organizations. AMREF added a 40% desert

allowance (allowed by MoFEC policy) to get more in line with its own per diem rates.

The project had a logframe but not a theory of change. Monitoring was done at the level of the specific

organization involved, but not at the level of the project (e.g. joint monitoring missions) using the

logframe, except for the supervision missions executed by the UNICEF staff involved. The progress

reports to the EU do refer to the logframe though.

Government stakeholders, in particular the regional health offices, mentioned the difficulty for them to

execute monitoring missions partly because of a lack of fuel and partly because of the inaccessibility

of certain woredas.

4.4 Impact

The evaluation matrix poses two key questions to assess the impact of the project, linked to the

objectives of the project: (1) What was the contribution of the program in reducing malnutrition in

geographic targeted areas? and (2) What was the contribution of the program in improving nutrition

and dietary diversification practices in geographic targeted areas? Targets were set in the project

document for dietary diversification practices, but not for nutritional status among children. The data


used to assess the impact criterion principally originated from the baseline and endline surveys

conducted in 2017 and 2018 respectively.

Finding #13: The project proved to be successful in

addressing severe and moderate acute malnutrition,

and severe underweight in children under five years of

age in the 17 targeted woredas. Improvement was also

observed in older children (30 months and older)

suffering from wasting and severe underweight. On the

other hand, the project was not successful in reducing

moderate underweight and stunting prevalence among

children in the geographic targeted areas.

Table 7 below shows that during program implementation,

the overall GAM prevalence decreased from 18.1% to

7.8%. Severe acute malnutrition (SAM) decreased from

7.1% to 1.8%, and the decrease was observed in all the

three regions. Moderate acute malnutrition (MAM)

decreased from 11% to 6%, but this decrease occurred

only in Oromia (from 10.8% to 5.7%) and SNNP (from

10.3% to 3.7%). The decrease in SAM and MAM occurred

in all age categories from baseline to endline, with greater improvement in the age category 30 months

to above (table 8).

Table 7: Trends in wasting prevalence at baseline and endline in the three regions

All Amhara Oromia SNNP

Baseline Endline Baseline Endline Baseline Endline Baseline Endline

SAM 7.1% 1.8% 14.6% 3.2% 10.8% 2.4% 5.1% 1.1%

MAM 11.0% 6.0% 11% 12.2% 10.8% 5.7% 10.3% 3.7%

GAM 18.1% 7.8% 25.6% 15.4% 21.6% 8.1% 15.4% 4.8%

Table 8: Trends of severe and moderate wasting rates at baseline and endline

Severe wasting

(<-3 z-score) Moderate wasting

(>= -3 and <-2 z-score)

Baseline Endline Baseline Endline

6-17 months 5.3% 2.9% 12.7% 7.9%

18-29 months 5.2% 2.5% 11.0% 7.2%

30-41 months 8.0% 1.0% 11.3% 4.1%

42-53 months 9.3% 0.8% 10.3% 5.3%

54-59 months 9.1% 1.3% 6.1% 2.0%

Total 7.1% 1.8% 11.0% 6.0%

As observed through survey findings, people interviewed also felt that the number of malnourished

children generally decreased in the intervention areas, particularly a reduction in acute malnutrition.

With respect to underweight, the overall prevalence decreased from 26.1% to 21.4%. Severe

underweight decreased from 10.3% to 6.1%, but there was no change in moderate underweight

“In 2017, about 600 children were admitted in the OTP (outpatient therapeutic program).. Now, this month (July) there were only 20 children listed as OTP beneficiaries”. Emergency Preparedness Officer, Agriculture Office, Damot Pulassa.

“In our woreda there are 29 kebeles (…) Compared to the past there are fewer cases of acute malnutrition. The number of admissions to OTP has decreased from 2,000 to about 20-30 now”. Emergency preparedness Officer, Kindo Koisha Woreda Health Office

“Malnutrition in children has significantly reduced (...) In the past there were at least 4 children admitted to OTP per week. In the last three months no child has been admitted. There are no longer children in our kebele that are fed by ready to use therapeutic foods”.

HEWs, Damot Pulassa.

“Less children have been admitted in the OTP. Children are healthier, because they are consuming healthy foods”.

HEW, Kindo Koisha.


prevalence between the endline and the baseline surveys (15.3% and 15.8% respectively). In Oromia,

underweight rates decreased from 24.9% to 20.3% while it decreased from 24.6% to 15.9% in SNNP.

Such a decrease did not happen in Amhara, where underweight rates (severe and moderate) actually

increased, from 33.5% to 40.3% (table 9). Thus, the project interventions achieved better outcomes

in severe than moderate underweight. As for acute malnutrition, greater improvement was observed

in older children (30 to 59 months) suffering from severe underweight (table 10).

Table 9: Prevalence of severe and moderate underweight by region at baseline and endline


Baseline Endline Baseline Endline Baseline Endline Baseline End-ine

Severe 10.3% 6.1% 13.4% 14.8% 11.7% 6% 8.2% 3.8%

Moderate 15.8% 15.3% 20.1% 25.5% 13.2% 14.3% 16.4% 12.1%

Overall 26.1% 21.4% 33.5% 40.3% 24.9% 20.3% 24.6% 15.9%

Table 10: Trends of underweight by age group at baseline and endline

Severe

(<-3 z-score) Moderate

(>= -3 and <-2 z-score)

Baseline Endline Baseline Endline

6-17 months 5.0% 6.8% 12.1% 12.9%

18-29 months 7.1% 6.8% 16.9% 13.8%

30-41 months 14.4% 6.1% 19.4% 17.8%

42-53 months 12.4% 5.5% 14.1% 14.8%

54-59 months 15.9% 2.7% 17.8% 23.3%

Total 10.3% 6.1% 15.8% 15.3%

Table 11: Trends in stunting prevalence at baseline and endline in the three regions



Severe 14.4% 16.0% 20.1% 25.2% 16.2% 14.2% 13.5% 13.6%

Moderate 17.2% 22.5% 13.4% 30.7% 16.3% 20.3% 18.5% 21%

Overall 31.6% 38.5% 33.5% 55.9% 32.5% 34.5% 32% 34.6%

Table 12: Distribution of stunting by age groups at baseline and endline

Severe stunting

(<-3 z-score) Moderate stunting

(>= -3 and <-2 z-score)

Baseline (%) Endline (%) Baseline (%) Endline (%)

6-17 months 8.8 9.7 13.0 19.1

18-29 months 18.9 17.3 15.5 21.9

30-41 months 18.4 20.3 24.4 25.8

42-53 months 12.4 16.7 16.0 23.0

54-59 months 11.5 19.3 13.5 24.0

Total 14.4 16.0 17.2 22.5


With respect to stunting, the overall prevalence increased from 31.7% to 38.5%, an increase observed

in both severe (14.4% to 16%) and moderate (17.2% to 22.5%) forms (table 11), and in all age groups

in the three regions (table 12). Therefore, the intervention was not successful in reducing stunting

prevalence.

Finding #14: With respect to children, the project interventions were successful in improving

exclusive breastfeeding according to the endline survey. However, although some

improvement occurred in the minimum acceptable diet, none of the targets was achieved

concerning the four key indicators defined in the project’s logframe to assess progress in

complementary feeding practices among children. With respect to pregnant and lactating

women, the interventions contributed to positive behavioural changes and practices during

pregnancy, improvement in the minimum diet diversity, and reduction of hunger in the

households, but the impact on antenatal care practices was not conclusive. Regarding WASH

practices, the intervention slightly improved at critical times but did not foster any

improvement in water treatment practices and use of toilets.

Early initiation of breastfeeding is recommended for the benefit of the newborn and the mother. The

first breast milk (the colostrum) contains antibodies which provide natural immunity. It is

recommended that children be exclusively breastfed during the first 6 months of life and to continue

with breastfeeding until 24 months or more.

Figure 9: Trends in exclusive breastfeeding at baseline and endline in the three regions

Figure 9 above shows that the exclusive breastfeeding rate was 89.5% at endline, higher than the

baseline (80.1%). In the regions, the exclusive breastfeeding rate improved in Amhara (from 82.6%

to 98.7%) and in SNNP (from 74.8% to 90%). In Oromia, the rate remained fairly stable at baseline

and endline (86.2% and 84.6%, respectively).

Four indicators were defined in the project’s logframe for assessing progress in complementary

feeding practices among children. Table 13 presents the achievements with regard to these indicators

82.6%86.2%

74.8%80.1%

98.7%

84.6%90.0% 89.5%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Amhara Oromia SNNP All

Baseline Endline


against the targets set. Children who received the minimum acceptable diet reached 14.2% at endline

which was slightly better than baseline (12.0%).

Table 13: Trends in the four key project indicators assessing progress in complementary feeding practices

Targets All Amhara Oromia SNNP

Base End Base End Base End Base End

Children 6-23 months old with a minimum acceptable diet

4% improvement from baseline

12.0% 14.2% 4.8% 9.9% 13.6% 8.9% 13.4% 18.4%

Continued breastfeeding up to 2 years

90% 30.8% 31.3% 55.8% 47.8% 16.1% 22.7% 31.6% 33.8%

Children who receive foods from 4 or more food groups


66.0% 56.2% 43.7% 62.7% 64.3% 61.8% 71.7% 51.1%

Children who receive solid, semi-solid or soft foods the minimum number of times or more a day


75.4% 74.9% 84.2% 71.9% 72.9% 78.7% 74.8% 73.3%

Improvements were observed in Amhara (from 4.8% to 9.9%) and in SNNP (from 13.4% to 18.4%),

while there was a decline in Oromia (from 13.6% to 8.9%). The

target (4% improvement compared to the baseline) was

achieved in the SNNP region for this indicator.

Regarding continued breastfeeding up to 2 years, no significant

progress was observed, and the target of 90% was not

achieved in any of the three regions.

The proportion of children receiving foods from 4 or more food

groups decreased from 66.0% at baseline to 56.2% at endline.

Only the region of Amhara achieved the target for this indicator

(43.7% at baseline versus 62.7% at endline). No change was observed in the proportion of children

6-8 months receiving solid, semi-solid or soft foods (75.4% at baseline versus 74.9% at endline). None

of the regions achieved the target of 10% improvement, although some improvements were observed

in Oromia region (78.7% at endline versus 72.9% at baseline.

Among women, taking diversified food during pregnancy improved from baseline (61.6%) to endline

(71.8%), with the regions of Amhara and Oromia achieving greater improvements (table 14 below).

“Now in our households we are eating foods that we didn’t eat before…we know what healthy food we can use to feed our children with”.

FGD with PLW, Kindo Koisha Kebele.

“In the past our children used to get sick and some would die. Now they are healthier… we see that there are lower number of malnourished children… now we know what kind of foods we need to feed ourselves and to our children”.

FGD with PLW, Damot Pulassa Kebele.


Table 14: Progress achieved in behavioural changes and practises during pregnancy among women

All (%) Amhara (%) Oromia (%) SNNP (%)

Baseline Endline Baseline Endline Baseline Endline Baseline Endine

Diversified food intake during pregnancy

61.6% 71.8% 42.5% 70.7% 53.0% 73.1% 71.7% 71.4%

Take deworming tablets during pregnancy

7.2% 16.8% 8.9% 24.0% 5.6% 23.7% 7.6% 10.6%

Giving colostrum 12.7% 18.9% 10.3% 12.7% 15.5% 24.0% 11.9% 17.3%

Take adequate rest 42.3% 43.2% 49.3% 36.7% 49.8% 51.7% 36.3% 39.6%

Take additional meals especially snacks during pregnancy

51.5% 41.8% 39.7% 56.7% 47.0% 39.6% 57.4% 39.6%

Taking deworming tablets during pregnancy improved from baseline (7.2%) to endline (16.8%), and

in all regions. Practice of giving colostrum also improved, with 18.9% of women at endline versus

12.7% at baseline, all regions achieving positive outcomes. The practice of taking adequate rest

remained more or less at the same level (42.3% to 43.2% overall), and in all regions. The proportion

of women taking additional meals especially snacks during pregnancy decreased from 51.5% to

41.8%, with a greater decrease observed in Oromia and SNNP regions.

Figure 10: Trends in minimum diet diversity for women in the three regions

Minimum diet diversity for women (MDD-W) assesses whether a woman has consumed at least 5 out

of 10 of the defined food groups the day or night before the survey. It can be used as a proxy indicator

for higher micronutrient adequacy which reflects the diet quality. As shown in figure 10, the proportion

of pregnant and lactating who consumed a minimum diet diversity at endline was 23.7%, higher than

7.4% 7.3%

17.1%

12.5%

9.2%

15.2%

34.2%

23.7%

0%

5%

10%

15%

20%

25%

30%

35%

40%

Amhara Oromia SNNP All

Baseline Endline


the baseline MMD of 12.5%. This improvement was observed across the three regions, the SNNP

region achieving greater improvement (34.2%).

The household hunger scale (HHS) is a household level indicator which measures the food quantity

dimension of food access. It is used to monitor the prevalence of hunger over time and to assess the

food security situation in specific areas. Households with little to no hunger improved from baseline

(86.3%) to endline (90.1%) in the three regions (table 15), with Amhara region achieving greater

improvement (92.1% to 97.2%). Moderate hunger also decreased from baseline (13.4%) to endline

(9.2%). Severe hunger was observed in 0.7% of the households in the three regions and it was higher

in Oromia (2%).

Table 15: Progress achieved in Households Hunger Scale during program implementation

Household hunger

score All (%) Amhara (%) Oromia (%) SNNP (%)


Little to no hunger in the household

86.3% 90.1% 92.1% 97.2% 84.2% 88.9% 85.9% 88.9%

Moderate hunger in the household

13.4% 9.2% 7.6% 2.8% 15.5% 9.1% 13.9% 10.8%

Severe hunger in the household

0.2% 0.7% 0.3% 0.0% 0.3% 2.0% 0.2% 0.3%

Support for antenatal care (ANC) was implemented as one of the project’s components in the 17

Woredas. According to the focused ANC guideline of Ethiopia, each pregnant woman should undergo

at least four ANC follow-ups at health facilities during which medical check-ups and counselling

messages are provided for application during pregnancy, delivery and after birth. At baseline, 37.1%

of women reported they had 4 or more ANC follow-ups in their last pregnancy, but this decreased to

33.2% at endline (table 16). At regional level, a high decline was reported in Oromia (42% at baseline

to 28.6% at endline), while there was slight improvement in Amhara (from 13.4% to 19.7%) and no

change in SNNP (40.7% versus 40.3%). The impact of the intervention on ANC practices was not

conclusive.

Table 16: Evolution in frequency of ANC follow-ups of women during their last pregnancy by region

Frequency of ANC

visit All (%) Amhara (%) Oromia (%) SNNP (%)


0 16.0% 15.4% 17.3% 25.9% 12.2% 12.9% 18.0% 13.9%

1 4.0% 5.9% 4.5% 10.2% 6.0% 6.2% 2.6% 4.5%

2 16.7% 15.0% 24.8% 17.2% 15.5% 18.3% 15.1% 12.2%

3 26.2% 30.4% 40.1% 27.0% 24.3% 34.0% 23.5% 29.2%

4 or More 37.1% 33.2% 13.4% 19.7% 42.0% 28.6% 40.7% 40.3%

The evaluation also captured progress achieved in WASH practices, particularly in hand washing,

water treatment and use of the toilet. Hand washing with soap at critical times reduces the


transmission of diseases, such as diarrhea which has a direct link with malnutrition and other

diseases. The project recommended practicing hand washing with soap in five critical times: after

defecation, after handling child feces or cleaning a child's anus, before preparing food, before feeding

a child, and before eating.

A proportion of 71% of endline survey respondents confirmed they have received hand washing

information, which was better than the baseline (67.3%). At the regional level, higher proportions were

observed in SNNP (77.3% versus 36.9%) and Oromia (71.9% versus 18.4%) than in Amhara (48.7%

versus 12.1%). From the respondents who received hand washing information, 99% claimed to

practice washing hands with water and soap or ash at critical times, which was better than the baseline

(97.6%). Across the three regions, these proportions were 98.5% in Amhara, 99.3% in Oromia and

98.8% in SNNP.

Practice of hand washing after defecation was mentioned by 61% of the respondents and it was better

than the baseline (57%). Hand washing practice after cleaning the bottom of a child following

defecation was confirmed by 34% of the respondents, which was also better than the baseline

(15.7%). Similarly, hand washing practice before preparing food improved from baseline (59.5%) to

endline (75%), along with the practice of washing hands before feeding a child (from 9.4% at baseline

to 28% at endline). Only the practice of hand washing before eating among responding women did

not improve (71% at endline versus 83.6% at baseline). Table 17 below summarizes the

achievements in washing hand practices among survey respondents.

Table 17: Practices on hand washing at different critical times at baseline and endline

Progress in WASH

practices All (%) Amhara (%) Oromia (%) SNNP (%)


Information on hand

washing 67.3% 71.0% 12.1% 48.7% 18.4% 71.9% 36.9% 77.3%

Washing hands with

water and soap at

critical times

97.6% 99.0% 100.0% 98.5% 98.0% 99.3% 96.7% 98.8%

Hand washing after

defecation 57.0% 61% 66.6% 59.3% 58.5% 64.5% 53.7% 60.3%

Hand washing after

cleaning child’s anus 15.7% 34% 2.5% 35.8% 20.2% 35.6% 16.8% 32.7%

Hand washing before

preparing food 59.5% 75% 46.9% 83.9% 76.0% 81.8% 54.1% 69.6%

Hand washing before

feeding a child 9.4% 28% 5.6% 33.5% 19.0% 26.9% 23.1% 26.4%

Hand washing before

eating 83.6% 71% 80.9% 80.8% 90.2% 87.0% 80.9% 60.8%

Households were also asked whether they treated water for their consumption. In total, 17.9% of the

respondents reported that they used water treatment, hence no improvement compared to the


baseline (19.1%). At the regional level, only Oromia showed a slight improvement in this practice

(34.4% at endline versus 30.8% at baseline). In terms of types of toilets used, almost all households

(98.5%) had unimproved sanitation facilities across the three regions from baseline to endline (table

18 below).

Table 18: Progress in other WASH practices at baseline and endline

All (%) Amhara (%) Oromia (%) SNNP (%)


HHs that used water

treatment 19.1% 17.9% 4.5% 9.2% 30.8% 34.4% 16.8% 11.4%

HHs with

unimproved

sanitation facilities

98.9% 98.5% 97.8% 99.2% 100% 97.8% 98.7% 98.7%

4.5 Sustainability

Finding #15: Scaling up of the project to all woredas in the country is potentially possible

because the National Nutrition Plan II and the National Nutrition-Sensitive Strategy have fully

integrated the project’s approach. However, the financing of both plans has not been secured

yet.

The project managed to integrate the provision of Vitamin A and deworming tablets into the routine

activities of the health centres. However, procurement and supply of the products are not

mainstreamed yet into the government system, and they are still organized by UNICEF with money

from external donors.

Although the government is a key partner in the project, sustainability was not a specific objective

built in the project from the onset. Moreover, the project’s four year duration, with a long start-up

period, was not enough to build the local capacity needed to sustain the results. The complementary

feeding units for instance, which should have produced sustainable revenues for the targeted women,

were not created. On the other hand, the project had a strong link with the Productive Safety Net

Programme (PSNP), which increased the sustainability of the provision of seeds and chickens to

vulnerable families enabling them to produce or purchase complementary food.

The project’s activities to alleviate severe malnutrition in emergency situations are funded by external

sources and it is not likely that these activities can be sustained by the government in the near future.

Finding #16 : The project’s work on feeding practices and the close collaboration between

health and agricultural extension workers fostered by the project are important activities that

can be replicated in other woredas with limited investments.

Community based infant and young child feeding is the focus of the project and the work on feeding

practices was met positively raccording to the evaluation team’s focus group discussions with

beneficiaries. The number of children with severe acute malnutrition who needed to be treated in

health clinics considerably decreased according to health and nutrition officers the team met in the

field.


The use of the health development armies, and the 1-30/1-5 cascade model of community leaders

involved, proved to be a cost effective and sustainable model,

which can be replicated elsewhere. According to various

sources, including the beneficiaries, the social analysis and

action approach (SAA) promoted by CARE and picked up by

the other implementing organizations and by the government

health service, was quite effective in initiating behavioural

change leading to better nutrition diets and feeding practices,

and is being replicated elsewhere in the country.

Across the board, the capacity of health and agricultural

extension officers to work together and to deal with nutrition

challenges greatly improved, and is one of the key successes

of the project. On the other hand, a rapid turnover of

government staff negatively affects the sustainability of the training investments.

5 CONCLUSIONS

This chapter summarizes the findings and identifies the lessons learned and forms the basis for the

recommendations presented in chapter 6.

Conclusion #1: The project activities respond to the specific needs of the targeted population

groups in the 17 woredas where the project intervenes, and are in line with the government

nutrition policy as expressed in NNP II and the Nutrition-Sensitive Agricultural Strategy. 28

The project has a targeting approach and its main focus is on the feeding practices of the households

in the kebeles and builds on the work done by health and agricultural extension workers. The project

targets vulnerable households and in particular pregnant and lactating women, adolescents, and

children under 5, and the activities are adapted to the specific conditions of the kebele. The integration

of female and male community leaders as trainers and sensitizers of target populations reinforced the

credibility of the messages and mitigated tensions with villagers who did not receive the services

provided by the project.

The multi-sectoral character of the project, in particular the link between feeding practices and

agricultural production, proved to be relevant and fostered innovative collaboration between health

extension workers and agricultural development agents, who coordinated their actions and converged

their messages.

The social analysis and action approach, promoted by CARE, proved to be a relevant and efficient

tool to change behaviour in feeding practices. The distribution of chickens resulted in the selling of

eggs and thus economic empowerment for the targeted women groups.

Conclusion #2: Part of the targets related to the implementation of basic services were

achieved according to the survey data, and considerable discrepancies between progress

report data and survey findings were observed. The quality of extension services improved

28 Linked to findings 1 – 4 on relevance.

“Twice a month we come to the kebele centre to discuss about the importance of dietary diversification and attend cooking demonstrations; 1-to-30 leaders go house-to-house to teach about the importance of exclusive breastfeeding and better diets.”

FGD with PLW, Faya Kebele

“The health and agricultural extension workers come together and teach us about what we need to eat, like vegetables, and how to grow these. We now have these backyard gardens that changed the way we and our children eat.”

FGD with PLW, Damot Pulassa Kebele.


despite many challenges. Targeted pregnant and lactating women, adolescents, and

households with children under 5 became very engaged in the process.29

The use of different databases with different samples probably explains the discrepancy observed

between the reports of the health administration and the surveys. The sample of children and pregnant

and lactating women applied to develop the regional health database, used by the administrative

reports, is much smaller than the sample applied by the surveys.

Conducting the baseline survey a year after the beginning of the intervention, a period where the

project came to steam, has jeopardised the trend between baseline and endline. The baseline data

are not really baseline data, because the project attained already some results. Moreover,

environmental hazards such as droughts and floods diverted the project’s implementation from its

initial trajectory, limiting access to services and potentially weakening the already vulnerable children,

adolescents and PLW. This may explain the decrease, rather than an expected improvement, of the

results in basic service delivery (Vitamine A supplementation, deworming, IFA supplementation and

participation to GMP) from baseline to endline; and the improvement of indicators related to

emergency response services such as screening in children and PLW.

Given that many activities were implemented without specific targets, it was difficult to appraise the

achievements. Nevertheless, we can conclude that the targeted approach focusing on specific

population groups and using specific messages on feeding practices; multisectoral engagement;

capacity development through a cascade training model with involvement from local leaders; women

empowerment through vegetable and poultry production; and community sensitization using SAA,

fostered the results.

As expressed by the focus groups, supporting households on vegetable and animal production

enabled them to improve their diet and generate income to access other services. We observed a

strong engagement among the target populations, which is a critical success factor. Unfortunately,

seed supply did not go as smoothly as planned.

Although the collaboration between health and agricultural extension workers was mentioned by all

informants as one of the core successes of the project, the absence of a project coordination unit,

along with the high workload of HEWs, hampered an even better integration of both sectors.

Visual identification of households receiving goats stigmatized the beneficiary families. Better

anticipation of these challenges, and more flexibility in the project design (results-based rather than

activity-based) would have probably lead to better project’s achievements.

Conclusion #3: At the moment of the evaluation, the project had spent about two third of its

budget. The decision to have agreements with NGOs to increase the project’s implementation

capacity did not have the intended effect; only one third of the available budget for the PCAs

was spent. On the other hand, the level of execution of the components executed directly

through the government were the highest of the project.30

The effective project duration was too short to execute the available budget, and more funds could

have been directly channelled through the government’s health and agricultural services and

29 Linked to findings 5 – 10 on effectiveness 30 Linked to finding 11 on efficiency


institutions. The preparation and approval of the program documents underlying the PCAs took a long

time, as did the hiring of additional staff by the implementing organizations, who collaborated anyway

with the local government services in the execution of their interventions. With the exception of CARE,

who had the specific task to promote and disseminate the SAA method, and of ACF who fully executed

the available budget by aligning the activities with their ongoing programs, the added value of the use

of intermediate implementing organizations is difficult to see in this project.

The project had to make readjustments to respond to emergency situations (climate and security) and

had to deal with sub-optimal supply systems (e.g. seed and iron folid acid). The activity-based

architecture of the project (compared to a more flexible results-based set-up) did not provide sufficient

flexibility to efficiently deal with the required adjustments.

Conclusion #4: The project did not have a project coordination unit with representatives of all

implementation actors nor a fully designated project manager with authority over all actors

and vested in government ownership. The project did not have a systematic monitoring and

reporting mechanism either at the level of the project.31

The coordination was done by UNICEF’s CBN & Micronutrient unit head and one program officer, who

did an excellent job in moving the project forward and getting all parties doing their part, but who did

not have the authority to make operational decisions on all project components, such as the one

executed by FAO, which underperformed; no complementary feeding units were set up, yet these

were a core element of the project. A project manager with full budget authority, a project team and

effective and formal coordination and monitoring mechanisms could have examined alternative

implementation options.

Accountability was not well organized (who is ultimately accountable for the project’s results; UNICEF,

FAO, the implementing organizations?). UNICEF was ultimately accountable for nutrition-specific

activities and FAO for nutrition-sensitive activities, but the implementation agencies executed both

nutrition-specific and nutrition-sensitive activities (e.g. behavioural change) and they had a contract

with UNICEF, which makes UNICEF ultimately accountable for the performance of these NGOs.

Moreover, and more importantly, the government did not feel accountable for the results. At the federal

level, the government was not involved, and at subnational levels the government services were more

engaged as partners than as owners. Regional and woreda governments participated right from the

inception stage, taking part for instance in consultative workshops to identify the context-specific

underlying causes of malnutrition and design the activities. In addition, UNICEF and FAO transferred

funds on a quarterly basis directly to the target woreda health and agriculture offices to support for

instance their monitoring of routine nutrition programs. However, the governments did not have

authority on activity and budget decisions and hence did not own the project.

The donor took notice of the annual reports but did not actively monitor the project. A more active and

regular monitoring by the donor would have facilitated timely identification of challenges and solutions.

It would also have fostered a better understanding of this project (and the lessons learned) in the

nutrition donor group at the federal level.

The project underspent and did not realize all the outputs and outcomes expected, but the results that

were attained are interesting and innovative and should have been better known in the nutrition

31 Linked to finding 12 on efficiency


dialogue at the federal level. The lack of structure, the absence of government ownership, and the

limited engagement of the donor are the main reasons that the project results are relatively unknown

at the national level.

Conclusion #5: The project interventions contributed to a reduction of the level of acute

malnutrition and severe form of underweight in children, but were not successful in reducing

moderate underweight and stunting. The practice of exclusive breastfeeding improved, but

appropriate complementary feeding practices among children remained a challenge.

Behavioural changes and practices during pregnancy were observed and the minimum diet

diversity among targeted women improved, and hunger among households reduced. Although

the project slightly improved hand washing practices, it did not have a significant impact on

ANC practices in pregnant women, in water treatment practices and use of toilets among the

target population.32

Immediate causes of childhood undernutrition include inadequate dietary intake and frequent

illnesses. Intermediate causes include insufficient access to food, inappropriate infant and young child

feeding and low quality care practices, along with limited access to water, sanitation, and health

services. Underlying causes comprise, among others, the political and economical enabling

environment, and environmental hazards33.

Wasting is a sign of acute or recent nutritional deficit. It represents the failure to receive adequate

nutrition in the period immediately preceding the survey and may be the result of inadequate dietary

intake or a recent episode of illness causing loss of weight and the onset of malnutrition. Underweight

can be a measure of acute or chronic malnutrition and it results from inadequate feeding, poor

absorption or excess loss of nutrients. A child can be underweight for his/her age because s/he is

wasted, stunted, or both. Stunting mainly reflects the long-term effects of malnutrition in a population.

Weight improvement can be achieved within a short period of time with consumption of ready-to-use

foods, which explains the imrpovements in wasting and severe underweight outcomes during the

project implementation period. Addressing immediate causes of malnutrition through regular free of

charge provision of ready-to-use foods, Vitamin A tablets and other supplements, as well as

vaccination of malnourished children admitted to the program, improved their weights. However,

addressing stunting requires more emphasis on the intermediate and basic causes of malnutrition

because it takes more time to a child to increase his/her height. The period between baseline and

endline survey was one year, which is enough to influence positive weight change (thus wasting and

severe underweight prevalence), but too short to foster significant change in height of children. This

can explain why we did not observe an improvement of stunting prevalence rates during the program

implementation period.

From baseline to endline survey, greater improvement in acute malnutrition and severe underweight

occurred in the age category 30 months and above. Although a positive outcome, this result was

beyond the window of opportunity of 0-24 months (first 1000 days) for optimal infant and child growth,

thus too late to catch up growth retardation. The lack of improvement in stunting prevalence occurred

32 Linked to finding 13 and 14 on impact

33 UNICEF (1997). Causes of Child Malnutrition. https://www.unicef.org/sowc98/fig5.htm

https://www.unicef.org/sowc98/fig5.htm


in all age groups, indicating exposure of these children to difficult conditions in their households from

their very first days of life. High emphasis should therefore be put on the pregnancy period (ANC,

targeted IYCF, micronutrient supplementation), post-delivery period (GMP, IYCF, deworming,

complementary food production and consumption), and pre-pregnancy period (nutrition counselling,

deworming, micronutrient supplementation, food fortification, water treatment, improved toilets etc.)

for better effect.

With respect to behavioural change, it usually takes time for people to absorb and apply new

knowledge. Many sensitisation activities, along with home gardening and livestock distribution, took

place in the targeted woreda. These preventative measures for addressing intermediate and basic

causes of malnutrition empowered the households and reduced hunger, and although target

populations gained knowledge according to the focus groups discussions, we did not observe a

significant change in complementary feeding practices among children and women. The endline

survey did not confirm a sustained effect on the critical nutritional situation of children that are still

within the window of opportunities (first 1000 days). This was probably due to the length of the project,

which was not long enough to foster sustained behavioural change with respect to feeding practices

in the community.

Conclusion #6: The GoE National Nutrition Plan and Nutrition-Sensitive Agricultural Strategy,

developed with technical assistance by the project, do mainstream the project’s activities, but

are not yet funded. Nevertheless, several key activities can be replicated elsewhere with

limited costs.34

Promising activities to be replicated are:

Nutrition-sensitive agriculture initiatives and successful collaboration between health and

agricultural extension workers;

Mainstreaming of Nutrition into PSNP;

Use of the SAA method to foster behavioural change;

Distribution of chicken and inputs for backyard gardening that empower women’s groups;

Use of the 1-to-30/1-to-5 leader groups to sensitize the target populations on improved feeding

practices and better diets;

Integration of Vitamin A and deworming in routine

activities of health services;

Notwithstanding the fact that the project did not succeed yet in

terms of the establishment of complementary feeding units at

the time of the evaluation, the CFU model can potentially

sustain and institutionalize the promising agricultural activities

the project fostered among targeted women, and should not be

abandoned but integrated in other projects, for instance the

Productive Safety Net Programme.

34 Linked to findings 15 and 16 on sustainability

“Because the frequency of drought is very high, the community needs to have different coping mechanisms... We should teach them on the importance of savings, which is the major coping strategy, next to behavioural change in feeding practices…Training needs to be scaled up, along with effective counselling campaigns, communications and workshops.”

Nutrition Officer, Oromia regional health bureau.


6 RECOMMENDATIONS

Recommendation #1: We recommend to design and fund a second project phase of 4 – 5 years.

Changing feeding practices and empowering targeted women groups through complementary

feeding units is a long-term matter that needs more time than the project effectively had, even

more so when taking into account the start-up delay and adjustments due to emergencies that

the project experienced.

The lack of improvement in stunting prevalence occurred in all age groups, indicating exposure of

these children to difficult conditions in their households from their very first days of life. A second

phase of the project should therefore put even more emphasis on the pregnancy and pre-pregnancy

period for better impact, focusing on adolescents, pregnant and lactating women.

A continuation of the project should build on the strengths of the first phase, as listed in the section

just before this chapter, continue the targeting approach and reinforcing the collaboration between

health and agricultural services. However, reinforcing activities (such as school farms) to support the

communities as a whole and not only the vulnerable households are likely to foster more engagement

and support at the community level that the targeting activities, and help to avoid tensions.

Although the project did not succeed yet in sustaining one of its key results – adjusted and better

feeding practices – the evidence from the stakeholder interviews show that the project’s focus on

complementary feeding practices to combat moderate malnutrition, in addition to supplements to fight

severe malnutrition, is the right one. Efforts should be doubled to establish successful complementary

feeding units.

UNICEF, the donor, and the GoE, should also consider extending the intervention area beyond the

most vulnerable woredas, in particular for the behavioural change and productive activities, to

increase the chance on success.

Recommendation #2: A second phase of the project should build on government ownership

and channel more funds through government services rather than through implementing

organizations.

Government ownership is a condition for sustainability and replication. We recommend to set up a

project coordination unit in either the Federal Ministry of Health or the Federal Ministry of Agriculture,

supported by UNICEF and FAO, with a steering group co-managed by both Ministries and chaired by

the Ministry that does not house the project coordination unit. The unit should be a light structure to

keep the project’s centre of gravity in the woredas.

If relevant, this project coordination unit will conclude agreements with implementing organizations,

such as CARE, to further disseminate the successful social analysis and action approach, or with

ACF, for instance, to provide training to government services and community leaders on community

management of acute malnutrition.

During a second phase, special attention should also be given to supply of products (supplements,

seeds, gardening equipment) to guarantee timely distribution and avoid delays.

The project experienced implementation problems due to emergencies, such as drought, flooding,

and security. Therefore, we recommend more flexibility for the coordination unit and an administrative

rapid response mechanism to efficiently adjust the project’s plan if needed, for instance by moving to


different woredas or by integrating new activities. Contractual agreements based on results rather

than on interventions provide this flexibility.

We also recommend to reinforce the link with the Safety Net program and with value chain projects

(e.g. moringa, NAIP), specifically to work with the targeted woredas and communities on coping

strategies to deal with extreme weather conditions.

66

ANNEX A ACTIVITIES OF THE PROGRAM (LOG-FRAME)

Activities for result 1

1. Procure deworming tablets (nearly 26,782) and support the supplementation for Children 23 to 59 months, pregnant women and adolescents.

2. Support and strengthen continued iron folic acid supplementation of PLW. 3. Printing and distribution of family health cards and IEC materials. 4. Train 850 HWs on blended training materials for PHCU in target woredas. 5. Implementation of IYCF through appropriate counselling addressing the barriers to exclusive

breastfeeding (through HEWs with support of HDAs). 6. Train 1179 DAs and 6800 HDAs, on good nutrition practices including dietary diversity and

promotion (strengthen the promotion of complementary feeding practices in all the 17 woredas) .

7. Train 589 DAs and 427 HEWs on the Productive Safety Net Programme (PSNP)/nutrition linkages.

8. Conduct rapid assessment on the available food commodities to design the best nutrient quality for complementary food production and existing practices on CF preparation in the project Woredas.

9. Strengthen the promotion of complementary feeding practices in all 17 woredas. 10. In the 10 Woredas, 10 women’s groups in the urban areas will be identified, organized and

trained on local production of complementary food including processing, preparation and demonstration.

11. Establish 10 urban CF processing units in target areas in semi urban towns through business model.

12. Train the 10 CF producing women groups (1 per urban model) on business management and marketing.

13. Provide start-up capital (in kind) to women’s groups to establish the complementary food processing units in the 10 CF project Woredas.


1. Identify and support vulnerable households (female headed households with children) with specific interventions such as homestead garden and livestock.

2. Procurement and distribution of nutritious seeds and farm tools and livestock/poultry for backyard production (provision of drought resistant, early maturing seeds and root crops suitable to target sites).

3. Provide small ruminants/livestock/poultry to HHs and conduct trainings associated with livestock/poultry to HHs.

4. Promote the application and use of skills gained by HHs in post-harvest loss reduction techniques.

5. Support jobless youth (organized in cooperatives) in the 8 Woredas on fishery, poultry production, community gardening and fruit production to create access for nearby households.

6. Development of messages to promote dietary diversity (including milk and fish products) and demand creation appropriate to media, schools and farmer training centers.

7. Conduct social mobilization for dietary diversity and homestead gardens, which will be done through demonstrations at the farmer training centers and schools.

8. Promote and provide quality seed for production of nutritious foods (fruits, vegetables and root crops) at Farmers Training Centers (FTC).

9. Promote milk-based complementary food for children in pastoral areas of project targeted Woredas.

10. Promote and improve access to quality milk by pregnant and lactating mothers. 11. Study Tours (in country and abroad) for MoA and BoA Experts.


1. Develop nutrition training material for agriculture sector. 2. Train 680 HEWs and 1020 DAs on nutrition and agricultural linkages. 3. Mobilize women development army team leaders to support nutrition interventions. 4. Implement nutrition coordination meetings with active participation of agriculture

representatives.


5. Mobilization and training of 35 of food security taskforce on nutrition-agriculture linkages in 17 Woredas.

6. Support MOA and BOA (SNNP, Amhara and Oromia) by providing TA to strengthen multi-sectoral coordination.

7. Training of extension workers and farmers in postharvest management, food preparation and processing techniques.

8. Capacity development of Agriculture sector through provision of training at Federal, Regional and Woreda level.

9. Integrated supportive supervision (ISS) and review meetings on NNP implementation.


1. Operational research topic identified and carried out which assess the implementation of multi-sectoral approaches to nutrition interventions, linking with agriculture.

2. Evaluation methodology developed and implemented through baseline/endline KAP assessments.

3. Nutrition situation analysis developed through mapping analysis, causal analysis, intervention gap analysis and drafting of synthesis report.

4. Routine monitoring system developed and implemented, inclusive of field travel.


1. Nutrition technical support provided to EUD and member states in line with the Joint Programming in Nutrition.

2. Technical support provided to the EUD on strategic and technical nutrition issues in line with the purpose of this proposal as well as EU's facilitation role in the Joint Programming on Nutrition.

35This activity will build on training and joint planning methods promoted through the FAO ECHO-funded project “Building Capacity for Better Food Security Planning” (OSRO/GLO/303/EU), which is implemented in collaboration with UNICEF and other partners and has provided training for a multi-disciplinary team for Ethiopia.

68

ANNEX B EVALUATION MATRIX

Evaluation criteria/Questions Indicators Data collection

method/analysis Source of information

Evaluation team

responsibility

1 Relevance

1.1

To what extent are the activities relevant to the needs of the population in the region?

Extent to which the activities respond to the needs of the population

Extent to which the activities respond to the needs of the targeted woreda/region/country

Content analysis Document review Individual interviews

UNICEF project’s proposal UNICEF Strategic Plan Programme Cooperation Agreement

(PCA) Key informants: partners, stakeholders,

beneficiaries, community members

Lead: ACT for Performance

1.2 Is the program in line with government strategies and policies?

Extent to which the selected initiatives were coherent with national priorities

Content analysis Document review

UNICEF Documentation Government health, food and nutrition

policies and strategies

1.3

Does the program adequately consider gender norms and tensions within the communities?

Extent to which gender equality issues have been factored into program design and implementation

Extent to which program outcomes promoted gender equality within the communities

Content analysis Document review FGDs

Project’s proposal Government health, food and nutrition

policies and strategies Program activity reports (documents and

quantitative data) from UNICEF, FAO and NGOs

Key informants: partners, stakeholders, beneficiaries, community members

1.4

Does the program effectively use a ‘do no harm’ approach sensitive to local contextual issues?

Extent to which the program outcomes benefited targeted beneficiaries in an equitable manner in both host communities and refugees


Program activity reports Key informants: partners, stakeholders,


1.5

Does the program take a context-specific approach within each region and woreda?

Extent to which the activities implemented were coherent with region and woreda specificities


Program activity reports Key informants: partners, stakeholders,





Evaluation team

responsibility

2 Effectiveness

2.1

How effective has the program been in improving access to basic services for the population?

Percentage of children 24-59 months old who are dewormed (indicative target 80%)

Percentage of adolescent boys and girls in target districts who take deworming medication (30%)

Percentage of PLW who have received iron folate during ANC (30%)

Percentage of children 0-23 months participating in GMP (50%)

Percentage of semi-urban towns in target areas that established a complementary food processing unit through business model (60%)

Number of children in the semi-urban target areas that received processed complementary food from the processing units (60%)

Number of PLW or caregivers of children under 2 among PSNP clients engage in BCC counselling (70%)

Number of health workers trained on blended materials (80%)

Number of health workers trained on emergency IYCF (80%)

Content analysis Document review Individual interviews FGDs

Baseline and endline surveys Administrative data Administrative reports Project progress report from partners

and Woredas Minutes of project and surveillance

meetings Projects M&E system and database Key informants: partners, stakeholders,


2.2 Has the quality and quantity of the services improved?

Proportion of PLWs, young mothers and adolescents attending ANC and PNC in health facilities

Proportion of children under 5 attending regular growth monitoring and promotion (GMP) activities

Number of woredas where health and agriculture staff implement joint activities (17 woredas)

Number of nutrition coordination meetings attended by agriculture representative (80%)

Content analysis Document review Individual interviews FGDs

National nutrition situation analysis (mapping, gap analysis and causal analysis, synthesis report)

Operational research findings report Report on technical support Key informants: partners, stakeholders,





Evaluation team

responsibility

Number of HW and DAs trained on agriculture nutrition linkages (80%)

Lead: ACT for Performance JaRco Consulting for survey findings

2.3 What were the major constraints in accessing basic services? How can these be improved?

Limiting factors Enabling factors

Content analysis Document review Survey Individual interviews FGDs

Woreda Reports Project progress report Baseline and endline surveys Minutes of the meetings Key informants: partners, stakeholders,


2.4 Were there external factors which affected the effectiveness of the activities? How can these be addressed in future programmes?

Enabling environment (legal, institutional, socio-economic, cultural)

Supply factors (access to services) Demand factors (perception of beneficiaries)


Administrative data Administrative and training reports Project progress reports Key informants: partners, stakeholders,


2.5 Were there any unanticipated effects (positive or negative)?

Nutrition and Health indicators Document review Individual interviews Survey

Baseline and endline surveys Minutes of the meetings Key informants: partners, stakeholders,


2.6 Were the targeted populations and their sufficiently admitted and engaged in the process?

Number of households with children vulnerable to malnutrition producing at least 2 new types of vegetables (Indicative target 29,000HHs)

Number of households with children vulnerable to malnutrition that increased their production and consumption of milk and eggs (Indicative target 29,000HHs)

Number of HHs trained and applied post-harvest loss reduction (80% of eligible HHs)

Number of months per year in which foods from own production are available in household (increased from 9 to 12 months for 29,000HHs)

Document review FGDs Survey

Baseline and endline surveys Project progress reports Key informants: partners, stakeholders,





Evaluation team

responsibility

Number of jobless youth groups members that improved skills and knowledge on local food production (14 groups)

Number of youth groups with increased average income per year from local food production (14 groups)

Number of households receiving fruit and vegetable seeds, drought resistant and early maturing seeds (20% of targeted HHs)

Households that received small livestock / poultry and associated training (30% of target HHs)

Number of targeted women receiving training on food processing (150)

2.7 Are there any disparities in access that occur in certain groups (such as female headed households, individuals living with a disability) and what can be done to improve on these?

Distance to health facilities for female headed households

Distance to health facilities for individuals with a disability

Perception of beneficiaries on access to the programme for female headed households and individuals who are living with a disability

Document review Individual interviews

Program progress report Key informants: partners, stakeholders,


3 Efficiency

3.1

Have the activities been implemented efficiently?

Use of PAFs and PME mechanisms Budget spent according to plan Activities implemented according to plan Overhead costs Benchmarks for unit costs

Content analysis Document review FGDs

Program activity reports (documents and quantitative data) from UNICEF, FAO and NGOs



3.2 In what way could the nutrition activities of EU Share have been implemented more efficiently

Coordination with other nutrition programs Staff capacity (skill mix) Capacity building of implementing organizations

Document review Cost analysis FGDs

Program’s financial data base Annual financial statements, financial

reports




Evaluation team

responsibility


3.3

Have the results been achieved at the lowest cost? Could there be a greater effect at the same cost? If so, how?

Program’s costs in total and broad categories (staff/material/travel/etc.)

Rate of disbursement/real and planned (gap)

Document review Cost analysis Individual interviews

Program’s financial database/file UNICEF and partners’ annual financial

statements and reports Manual of procedures Relevant staff (Government, UNICEF,

FAO, NGOs)

4 Sustainability

4.1

What aspects of the program are good practice and should be replicated?

Scaling-up and replication potential Government mainstreaming the program

activities in its annual plan and budget


Program activity reports from UNICEF and partners



4.2

What aspects of the program would need to be modified or strengthened if this was to be scaled up?

Performance assessment framework




4.3 What aspects of the program are woreda-specific? How could the program be replicated in other woredas?

Workplans Content analysis Document review Individual interviews


Woreda Reports Key informants: partners, stakeholders,


4.4 What challenges are likely to be encountered in replicating the program in both similar and different woredas?

Enabling environment Supply factors Demand factors



Woreda Reports Key informants: partners, stakeholders,





Evaluation team

responsibility

4.5 To what extent and in what ways have capacities been enhanced in government, civil society and NGOs to sustain the results?

Governments policy adjustments Implementation capacity indicators (quality of

documents; quality of monitoring) Most significant change indicators


(MSC, perception)



5 Impact

5.1

What was the contribution of the program in reducing malnutrition in geographic targeted areas? (overall objective)

Wasting, stunting and underweight prevalence among children under five years of age

Wasting prevalence among PLW

Baseline and endline anthropometric surveys

Individual interviews FGDs

Baseline and endline evaluation assessment reports


Lead: JaRco consulting

5.2

What was the contribution of the program in improving nutrition and dietary diversification practices in geographic targeted areas? (specific objective)

Steps taken by the community to change their household eating practices and the results Percentage of children 6-23 months old with a

minimum acceptable diet (4% improvement from the baseline)

Percentage of children with continued breastfeeding up to two years of age (90%)

Percentage of children 6-23 months old who receive foods from 4 or more food groups (4% improvement from the baseline)

Percentage of children 6-23 months old who receive solid, semi-solid or soft foods the minimum number of times or more a day (10% improvement from the baseline)

Survey on nutrition and dietary diversification practices

Individual interviews FGDs

Baseline and endline evaluation assessment reports

Selected Partners and other Key informants: partners, stakeholders, beneficiaries, community members

74

ANNEX C LIST OF DOCUMENTS AND ARTICLES CONSULTED

Document Description Source Date

received Review format

1. Terms of Reference (TOR). “Endline Survey of the nutrition component of the EU SHARE program." December 2017

UNICEF March 2 Electronic

2. Integrated Nutrition Services: Multisectoral interventions to improve nutrition security and strengthening resilience. Consolidated proposal


3. Baseline Assessment of EU-SHARE Project in Amhara, Oromia and SNNPR. Inception Report. Submitted by IPE GLOBAL LIMITED and ABH Services PLC, December 2016


4. Baseline assessment of EUSHARE project in Amhara, Oromia and SNNPR. Draft report. IPE GLOBAL LIMITED & ABH services PLC. February 1, 2018


5. Programme Cooperation Agreement (PCA) signed with the NGO Action Contre la Faim, 26 July 2016


6. Programme Cooperation Agreement (PCA) signed with the NGO International Medical Corps UK; 31 August, 2016


7. Programme Cooperation Agreement (PCA) signed with the NGO CARE Ethiopia


8. Programme Cooperation Agreement (PCA) signed with the NGO Concern Worldwide; 01 September 2016


9. Programme Cooperation Agreement (PCA) signed with the NGO Amref Health Africa; 08 August 2016


10. WFP, ECA, African Union. Cost of Hunger in Ethiopia: Implications for the Growth and Transformation of Ethiopia. Addis Ababa, 2013.


11. UNICEF. UNICEF procedure for ethical standards in research, evaluation, data collection and analysis. Division of Data, Research and Policy (DRP), April 2015. Document Number: CF/PD/DRP/2015-001.

UNICEF April 24 Electronic

12. Government of Ethiopia. National Nutrition Program, 2016-2020 Internet March 14 Electronic

13. Ethiopia Demographic and Health Survey, 2011 Internet March 14 Electronic

14. Ethiopia Demographic and Health Survey, 2016 Internet March 14 Electronic

15. Health Sector Transformation Plan (2015/16 – 2019/20) Internet March 14 Electronic

16. FMoH/UNICEF/EU Situation Analysis of the Nutrition Sector in Ethiopia: 2000-2015. Ethiopian Federal Ministry of Health, UNICEF and European Commission Delegation. Addis Ababa, Ethiopia 2016

Internet March 14 Electronic

17. Ministry of Agriculture and Rural Development. Ethiopia’s Agricultural Sector Policy Investment Framework (PIF - 2010 to 2020), September 2010


18. National Disaster Risk Management Policy Internet March 14 Electronic

19. Household Asset Building Programme (HABP) document Internet March 14 Electronic

Articles consulted

20. Vogel, Isabelle (2012). Review of the use of ‘Theory of Change’ in international Development, UK Department of International Development


21. Westhorp, G. (2014) Realist Impact Evaluation: An Introduction. Accessed at: http://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/9138.pdf



Document Description Source Date

received Review format

22. Marchal, B. et al (2012). Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research. Evaluation; 18:192


23. Pawson, R. and Tilley, N. (1997) Realistic Evaluation. London: Sage Internet March 14 Electronic

24. Palinkas, L.A., et al., Purposeful Sampling for Qualitative Data Collection and Analysis in Mixed Method Implementation Research. Adm Policy Ment Health, 2013


25. Tongco, M., Purposive Sampling as a Tool for Informant Selection. Ethnobotany Research & Applications, 2007. 5: p. 147 - 158


26. ILO. Integrating gender equality in monitoring and evaluation of projects. March 2014


27. Boddy, J., Neumann, T., Jennings, S., Morrow, V., Alderson, P., Rees, R., and W. Gibson (2014). The Research Ethics Guidebook: A resource for social scientists, University of London, http://www.ethicsguidebook.ac.uk/

Internet April 24 Electronic

Other documents

28. Program progress reports UNICEF Electronic

29. Minutes of project and surveillance meetings UNICEF Electronic

30. UNICEF and partners’ annual financial statements and financial reports

UNICEF

Electronic

31. ECHO-DEVCO resilience building program reports UNICEF Electronic

32. Community Based Nutrition Programme (CBN) UNICEF Electronic

33. The Integrated Family Health Programme (IFHP) document UNICEF Electronic

34. The Alive and Thrive programme document UNICEF Electronic

35. Nutrition programme- ENGINE UNICEF Electronic


ANNEX D STAKEHOLDERS AND INFORMANTS MET

Location Name of contact persons

Title Organization Contact address

Addis Ababa

Birara Melese Nutrition case team coordinator

Ministry of Health 913291992

Florence Tonnoir Nutrition Advisor FAO 944026386

Workicho Jateno Programme coordinator FAO 911374862

Nazirawit Yohannes Project Manager International Medical corps (IMC)

946399016

Alem Agazi Programme coordinator CARE International 911882467

Leulseged Tolla Programme coordinator Concern Worldwide 911362191

Kassahun Negash Programme coordinator AMREF Health Africa 911408766

Jogie Health and Nutrition coordinator

Action Against Hunger 941616365

Nardos Birru Nutrition Specialist UNICEF [email protected]

Eric Alain Ategbo Chief of Nutrition section

UNICEF [email protected]

Barbara Baille Monitoring and Evaluation Officer

UNICEF [email protected]

Amal Tucker Brown Nutrition Specialist UNICEF [email protected]

Pierre-Luc Vanhaeverbeke

Program Manager Rural development and Food Security Section

EU

Tel : +251116612511 Ext 24 Email : [email protected]

Ursula Truebswasser Technical Advisor EU Tel: +251929034804 Email: [email protected]

Jan Willem Nibbering Food Security Policy Officer

Netherlands Embassy [email protected]

Amhara

Ato Simeneh Worku MNCH Head and Nutrition Focal Person

Amhara regional health bureau

0918705549

Tiguaded Fentahun Nutrition Specialist UNICEF in Amhara Tel:- 0975186390 Email:[email protected]

Derebie Nega AEW Sekota Woreda Tel : +251 966 38 87

Debisa Welde HEW Sekota Woreda Tel : +251 927 38 68 56

Asefu Ayalew HEW Sekota Woreda Tel : +251 931 62 06 72

Tsehaynesh Haile Tsehaynesh Welde Ambanesh Fantaye Zenebu Negaw

FGDs: Leaders and mothers of children under 5

Sekota Woreda – Faya Kebele.

mailto:[email protected]



tel:-




Birtukan Belay Atsede Alemu Weyzer Desaleg Awetu G/Medin Asefu Zenebe

Oromia Taye Wondimu Nutrition focal person Oromia regional health bureau

0916850112

SNNPR Shimeles MNCH Head SNNPR regional health bureau

0911052342

SNNPR Genet Legese Lassiso Nutrition focal person SNNPR regional health bureau

0912024145

SNNPR Tsedeke Desalegn Area Coordinator Concern World Wide In SNNPR

Concern World Wide

0932561430

SNNPR Sister Elfenesh Health and Nutrition Coordinator IMC Wolayta zone

IMC

SNNPR Birhanesh Yayina Kindo Koisha – HEW 0945834226

SNNPR Tesahun Muluneh Kindo Koisha -Woreda Agriculture Office: Office Head

0913137469

SNNPR

Mezgebe Meskele Amarech Ala Mulunesh Muta Marta Petros Yirgalem Dachi Abaynesh Lenchi Dinkinesh Ganta Zenebech Yohannes Azalech Gitore

FGD Kindo Koisha 9 participants interviewed

SNNPR Mesgana Dansa HEW – Damot Pulassa 0970128726

SNNPR Zewditu Elias HEW – Damot Pulassa 0932617485

SNNPR Zewidu Boltana

Damot Pulassa -Woreda Health Office MCH Nutrition Core Processor

0911066716

SNNPR Takele Yigezu

Damot Pulassa -Woreda Agriculture Office Emergency Preparedness Officer

0910006933

SNNPR Desalech Ashango Zenebech Chemso Alkase Arge

FGD – Damot Pulassa 7 participants interviewed




Deraritu Mega Demekech Silte Tigest Dawit Obayse Jorke

79

ANNEX 5 TERMS OF REFERENCE

Terms of Reference (TOR)

1. Background

Ethiopia has experienced repeated environmental and external economic shocks for several years,

which have eroded rural livelihoods, exacerbating vulnerability and reducing resilience. Eastern and

southern lowlands and pastoral livelihoods have been particularly affected by even more frequent

droughts which led to significant loss of livestock, representing pastoralists' most valuable asset.

The European Union (EU) is funding an initiative aiming at addressing natural resources and access to

water, food security and nutrition and promoting developmental actions for a sustainable agricultural

growth in agro-pastoral areas in order to strengthen the linkages between relief, recovery and

development.

Under the nutrition component, a number of preventative nutrition actions has been implemented across

17 Woredas in Amhara, Oromia and SNNP regions. This will contribute to the improvement of

nutrition and dietary diversification practices for adolescent girls, pregnant and lactating women, and

children under 5, with a focus on the first “1000 days.”

UNICEF and its partner FAO are responsible for the implementation of the nutrition specific and

sensitive aspects of the EU Share project. UNICEF is accountable for the evaluation of the impact of

this component of the project.

2. Rationale for the Research Activity/Justification

Recognizing the importance of monitoring and evaluation of programs to contribute to a larger nutrition

program evidence base, UNICEF is supporting an assessment of this project. An evaluation of the

nutrition component EU SHARE programme is necessary to monitor implementation of project

activities, as well as assess change in indicators of progress toward the expected results of the project.

The assessment can also yield information on implementation of project components to help guide the

development and implementation of the components throughout the project lifespan, as well as in

varying livelihood/geographical project areas.

In May 2017, a baseline survey of the EU-Share project was conducted. The report assessed the

coverage of activities implemented through this project, as well as global knowledge and practices

around nutrition for children under 5, adolescent and pregnant and lactating women. As the project is

ending in October 2018, time has come to undertake the end-line evaluation of the nutrition component

of the EU-Share. A detailed description of the project components can be found in Annex 1.

3. Owners of the research and use of the findings

A Steering Committee has been established consisting of representatives from the Federal Ministry of

Health (FMoH), UNICEF Ethiopia (Nutrition and Food Security Section and M&E Section), and the

EC Delegation. As the main implementing agency of the National Nutrition Programme, FMoH will

coordinate and co-chair the Steering Committee and oversee the general process, with technical advice

and support from UNICEF. UNICEF will be responsible for the managerial aspects of this consultancy

and liaise between the contracted institution and the Steering Committee. The Steering Committee will

clear the Inception Report. The inception report, survey tools, and draft and final reports will be

reviewed and endorsed by the Steering Committee. UNICEF will support the dissemination of the

results for wider stakeholders.

4. Scope of the Research Activity

The evaluation will 1) assess the current status of knowledge, attitudes and practices (KAP) of

appropriate Maternal, Infant and Young Children feeding (MIYCF) behaviors, 2) yield information on

implementation of programming to help guide the development and implementation of similar types of

programs (e.g. scaling up): identify key barriers and opportunities to successful implementation and

scalability of the nutrition interventions, household/community gardening support and

agricultural/nutrition linkages, 3) Compare the information collected with the baseline and measure

progress made. While addressing these 3 main objectives, the evaluation will follow the below

evaluation criteria: i) Relevance; ii) Effectiveness; iii) Efficiency and; iv) Sustainability.

5. Methods

The institution will develop an evaluation design to address key objectives and questions detailed

below, based on the methodology used in the baseline survey. The evaluation will take place within the

17 project woredas, grouped into 5 clusters across Amhara, Oromia and SNNPR. The contracted

institution should de visit the same EAs as during the baseline (81 Enumeration Areas (EAs) from 51

kebeles).

The methodology will use a mix method approach as the one used at baseline phase. In particular:

- Household level quantitative assessment, using the same baseline sampling design. The

sampling size proposed should include a minimum of 2,620 households. The exact size of the

sample should be large enough to capture significant change in indicators from the baseline.

- Given that no comparison group was constructed in the initial baseline, firms should propose a

matching or other design that can help estimate treatment impacts, and should discuss how they

will account for selection and differential “take-up” of the programme among households that

were exposed.

- Qualitative approach will include Key Informant Interviews (KIIs) and Focus Groups

Discussions (FGDs):

KIIs with health and agricultural community workers, authorities and

community leaders. The firm will propose the appropriate number of KIIs to

be conducted.

FGDs: a minimum of 3 sets of FGDs (as per baseline) segmented by mothers,

fathers and grand-mothers. The firm will propose eventual additional target

groups for FDGs and the exact number of total FDGs to be conducted, as

appropriate.

The minimum set of indicators to be collected during the end-line are listed in Annex 2.

When assessing the impact of the Nutrition component of the EU Share project, the evaluators will

design specific indicators, derived from the 4 criteria and related questions listed in the table below.

Amendments to this list can be proposed. The firm is required to use a mix methodology approach to

answer the questions and to propose the related collection tools.

Criteria Questions

Relevance

1. To what extent are the activities relevant to the needs of both host

communities and refugees and the region/country at large?

2. Is the programme in line with government strategies and policies?

3. Does the programme adequately consider gender norms and tensions

within the communities?

4. Does the programme effectively use a ‘do no harm’ approach sensitive

to local contextual issues?

5. Does the programme take an appropriate approach within each region

and woredas?

Effectiveness

1. How effective has the programme been in improving access to basic

services for the population

2. Has the quality and quantity of the services improved?

3. What were the major constraints in accessing basic services? How can

these be improved?

4. Were there external factors which affected the effectiveness of the

activities? How can these be addressed in future programmes?

5. Were there any unanticipated effects (positive or negative)?

6. Were the targeted populations sufficiently engaged in the process?

7. Are there any disparities in access that occur in certain groups, such as

female headed households, disabled individuals and what can be done

to improve on these?

Efficiency

1. Were there activities that were more or less efficient than others?

2. How could nutrition components of EU Share activities be made more

efficient?

Sustainability 1. What aspects of the programme are good practice and should be

replicated?

2. What aspects of the programme would need to be modified or

strengthened if this was to be scaled up?

3. What aspects of the programme are woreda-specific? How could the

programme be replicated in other woredas while taking these factors

into account?

4. What challenges are likely to be encountered in replicating the

programme in both similar and different woredas?

5. Are there any characteristics of the programme (positive and negative)

that are unique for this programme and would not apply to other

programmes?

6. To what extent and in what ways have capacities been enhanced in

government, civil society and NGOs to sustain the results?

6. Specific Tasks with timeline

1) An inception report should be developed outlining the methodology for each component/activity and

the timeline for completion. This includes a detailed description of the evaluation design, sampling

methodology, proposed team composition and size and a clear supervisory/monitoring and data quality

assurance plan. A methodology for data entry and/or data transcription should include numbers of data

clerks and a timeline for completion, or a methodology for data transfer should mobile phones or tablets

be used. Key persons involved in data cleaning and analysis should be identified. Prior to

commencement of data collection, this inception report and all survey tools must be reviewed and

approved by the Steering Committee.

2) The institution will select/train/supervise enumerators and interpreters with previous experience. If

mobile phones or tablets are to be used for data collection, these devices must be used during training

to familiarize enumerators and supervisors with the devices and to help identify any device or system

related issues prior to data collection. A pre-test must also be conducted (using mobile phones or tablets,

if they are to be used during actual data collection) prior to official data collection. A field manual

should be developed and distributed to enumerators and supervisors for reference on key data collection

activities (e.g. sampling methods, ethics, data quality measures, etc.).

3) Sufficient resources should be committed to ensure data collection is carried out in a timely manner.

This entails adequate team size and composition during data collection, supervisors, field monitoring

and logistical arrangements. At least one supervisor should be allocated per team of enumerators. The

institution will ensure high quality measurements, and organize a daily quality check of data collected.

ENA software is highly recommended for daily check of anthropometric data. Special attention must

be paid to ensure accurate age in months is captured for children. Area specific local calendars should

be produced to assist with this.

4) The institution will be responsible for developing a data entry template based on survey tools (if

paper based tools are used), or an appropriate data entry template on the mobile phones or tablets used

during data collection, and data transcription if qualitative data collection is included in the evaluation

design. Sufficient time should be spent testing the template before tools are printed or data entry

templates on mobile phones/tablets are finalized, including testing data entry (or transfer) and quality

of data outputs after the pre-test is conducted and prior to official data collection.

A timeframe for completion of data entry can be fine-tuned based on testing the data entry template,

with adjustments made as necessary to the protocol (numbers of clerks, etc.) in order to complete in a

timely manner. A report on data quality (missing cases, etc.) will need to be provided to UNICEF upon

completion of data entry/transfer, prior to data analysis and report writing.

No. Outputs/Deliverables Time frame

1 Preparation of survey tools and an inception report, including: sections

on evaluation design, sampling methodology/survey protocol, survey

tools, training, pre-test, field work, logistics and data entry/transfer.

This should include clear plans for ensuring accurate estimation of

child age and overall data quality (spot checking data entry in the field,

supervisory checklists, calendars, etc.).

February 2018

2 Submission of inception report (40 pages max) and survey tools to

UNICEF

March 2018

3 Training of enumerators and supervisors. Report on training (including

number of participants, agenda, issues encountered and how resolved,

etc.) should be produced and submitted, along with all training

materials, to UNICEF

April 2018

4 Pre-test April 2018

5 Testing of data entry templates/data transfer and checking quality of

data outputs after pre-test and prior to data collection. Re-adjustment

of tools and/or templates based on issues found during pre-test.

April 2018

6 Field Data collection. Must provide weekly report to UNICEF during

data collection.

May-June 2018

7 Submission and endorsement of data quality output report and field

report (detailing methods used and any issues encountered and how

resolved) to UNICEF

June 2018

8 Copies of cleaned and referenced electronic data sets in an agreed

format received (for quantitative and/or qualitative data, depending on

analysis plan)

July 2018

9 Data cleaning July 2018

10 Submission of draft report (40 pages max without annex) on

preliminary findings to UNICEF

August 2018

11 Update to draft report based on comments/review from Steering

Committee and UNICEF

September 2018

12 Submission of final report (40 pages max), a presentation and a short

‘policy brief’ on results for wider circulation, to UNICEF

October 2018

13 Presentation on findings from final report to Steering Committee October 2018

7. Estimated duration of contract

10 months (January 2018 – October 2018).

8. Expected Deliverables

No. Outputs/Deliverables

1 Submission of inception report and survey tools to UNICEF

2 Training of enumerators and supervisors. Report on training (including number of

participants, agenda, issues encountered and how resolved, etc.) should be produced and

submitted, along with all training materials, to UNICEF

3 Initial Data collection. Must provide weekly report to UNICEF during data collection.

4 Submission and endorsement of data quality output report and field report (detailing

methods used and any issues encountered and how resolved) to UNICEF

5 Copies of cleaned and referenced electronic data sets in an agreed format received (for

quantitative and/or qualitative data, depending on analysis plan)

6 Submission of draft report on preliminary findings to UNICEF – 40 pages max

7 Submission of final report (40 pages max), a presentation and short ‘policy brief’ on results

for wider circulation, to UNICEF

8 Presentation on findings from final report to Steering Committee.

9. Reporting and Supervisor

The Head of CBN/MN Unit from UNICEF Nutrition Section will provide supervision on this project.

The institution will report to or meet with the supervisor on a regular basis on the progress of the data

collection and analysis. The institution will also be responsible to report to the Steering Committee that

will be organized under FMoH to oversee the general process.

10. Expected background and Experience

The assignment is open only for consultancy firms or academic institutions (with current legal

documentation) that qualifies based on the following criteria:

Experience conducting large household surveys;

Expertise in computer assisted personal interviewing (CAPI)

Expertise in quantitative and qualitative data collection;

Previous experience in carrying out similar assignments in the fields of nutrition, food security

and health;

Strong knowledge of community-based development programmes;

Ready to engage individuals for the assignment with expertise and experience on quantitative

and qualitative analysis;

Strong institutional capacity to work in a cross-cultural and multidisciplinary environment and

deliver the assignment within the specified period of time as indicated in this terms of reference;

Knowledge of the Ethiopian socio-cultural and religious context is an advantage.

Obligations of the Research Company

The research company will assign experts all of who shall be adequately qualified and

experienced in conducting public health/nutrition surveys to deliver the expected outputs in a

satisfactory and timely manner;

The research company shall engage a local team to help with field work, facilitate data

collection and assist data analysis;

Research team will secure ethical approval from local ethical review committee;

The local team will have to speak the local language;

The questionnaire will have to be translated into the local language;

The research company will make provision for his own transport and related costs;

The research company will be fully responsible to deliver the described outputs.

11. General Conditions: Procedures and Logistics

- The firm should provide its own materials, i.e. computer, office supplies, anthropometric tools,

transports.

- 30% of the agreed cost will be paid after reception and endorsement of the inception report by

UNICEF.

20% will be paid after submission of field report and approval by UNICEF.

- The remaining 50% will be paid after reception and endorsement of final report and

presentation of result done for the Steering Committee.

12. Policy both parties should be aware of:

- No contract may commence unless the contract is signed by both UNICEF and the Contractor.

- Contractor will not have supervisory responsibilities or authority on UNICEF budget.

13. Intellectual property rights (Insert this text or modify it based on discussions with government

counterparts).

All intellectual property rights in the work to be performed under this agreement shall be vested in the

FMoH and UNICEF, including without limitations, the right to use, publish, translate, sell or distribute,

privately or publicly, any item or part thereof. The FMoH and UNICEF hereby grants to the Recipient

Organization a non-exclusive royalty-free license to use, publish, translate and distribute, privately or

publicly, any item or part of the work to be performed under this Agreement for non-commercial

purposes. Neither the Recipient Organization nor its personnel shall communicate to any other person

or entity any confidential information made known to it by the FMoH and UNICEF in the course of the

performance of its obligations under the terms of this Agreement nor shall it use this information to

private or company advantage. This provision shall survive the expiration or termination of this

Agreement.

The core reports will be issued by the steering committee for the research noting in the

acknowledgements sections institutions and persons who have made major contributions to their

authorship. Once the official report is cleared the institution will be free to work further on those papers

for publication in peer reviewed journals. The institution will provide the steering committee members

with raw data, corrected/verified data once cleaned and programming files that permit replication of

results from core evaluation reports.

Data collected for the research is the property of the Government of Ethiopia/UNICEF country

programme. Master versions of the data, coding protocols and programming code permitting replication

of results of core evaluation reports will be kept by the programme. Copies of the data will be distributed

to researchers with the permission of the evaluation Steering Committee with a view to helping to

disseminate learning derived from the data sets.

Annex 1. Proposal of EH-Share project – Nutrition component.

Consolidated

proposal_March 2016.docx

Annex 2. Minimum set of indicators to be collected during the end-line survey.

Overall objective of the project: Contribute to improved nutritional status of adolescent girls,

pregnant and lactating women and children under 5.

End-line evaluation will have to assess the level of malnutrition in the areas of intervention and compare

with baseline, in order to evaluate the impact of the project on malnutrition rate.

Indicators to collect: Percentage of malnutrition among children under 2 years of age in the project

area: Wasting rate (MUAC, W/H); Stunting rate (H/A); Underweight rate (W/A). Percentage of

malnutrition among PLW: Wasting rate (MUAC).

Specific objective of the project: Improve nutrition and dietary diversification practices for

adolescent girls, Pregnant and Lactating Women (PLW), and children under 5 with a focus on

the first “1000 days” in 17 woredas in Ethiopia.

End-line evaluation will have to: Determine levels of KAP on key MIYCF and child caring and

maternal practices within the different intervention areas and compare with baseline results.

Indicators to collect: Percentage of children 6-23 months old with a minimum acceptable diet. (target:

4% improvement from the baseline); Percentage of children with a continued breastfeeding up to two

years of age (target: 90%); Percentage of children 6-23 months old who receive foods from 4 or more

food groups. (target: 4% improvement from the baseline); Percentage of children 6-23 months old who

receive solid, semi-solid or soft foods the minimum number of times or more (target: 10% improvement

from the baseline)

Expected result 1: Adolescent girls, PLW, and children under 5 are reached by quality

preventative nutrition interventions.

End-line evaluation will have to: Assess coverage of Vitamin A supplementation, deworming, iron

supplementation and GMP participation in the surveyed woredas, and compare with the baseline

coverage.

Indicators to collect: Percentage of children 24-59 months old who are dewormed (target 80%);

Percentage of children 6-59 months old supplemented with Vitamin A; Percentage of adolescent boys

and girls in target districts who take deworming medication (target: 30%); Percentage of PLW who

have received iron folate during ANC (target: 30%); Percentage of children 0-23 months participating

in GMP (target: 50%).

Expected result 2: Households are made aware of, and have increased access to, nutritious foods

and practices.

End-line evaluation will have to: Identify key barriers and opportunities to successful implementation

and scalability of the household/community gardening support and agricultural/nutrition linkages.

Indicators to collect: Percentage of semi urban towns in target areas established complementary food

processing unit through business model (target: 60%); Percentage of children in the semi urban target

areas that received processed complementary food from the processing units (target: 60%); Percentage

of PLW or caregivers of children under 2 among PSNP clients engage in BCC counselling (target:

70%); Percentage of households with vulnerable children with minimum dietary diversity (Indicative

target 18,000HHs); Percentage of households with vulnerable children to malnutrition increased their

production and consumption of milk and eggs (Indicative target 18,000HHs); Percentage of HHs trained

and applied post-harvest loss reduction (target: 80% of eligible HHs); Number of months per year in

which foods from own production are available in household (target: increased from 9 to 12 months for

18,000HHs).