Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
Final Evaluation Report
Final Evaluation Report
Evaluation of UNICEF’s response to the Water,
Sanitation and Hygiene Needs in Jordan as a
result of the Syrian refugee crisis (July 2012 to
July 2017)
PREPARED BY
INTERNATIONAL SOLUTIONS
GROUP
An IMC Worldwide Company
PREPARED FOR
UNICEF Jordan
16 April 2019
1
Evaluation Timeframe
This evaluation covers UNICEF’s WASH programme activities between July 2012 and July
2017. The evaluation was conducted between November 2017 and September 2018. Data
collection activities took place during July and August 2018.
Geographic Scope
This evaluation covers UNICEF Jordan WASH activities in the four camps of Za’atari,
Azraq, Cyber City (closed in September 2016) and King Abdallah Park, as well as Rukban
and Hadalat (abandoned in September 2017) at the Syrian/Jordanian border. Although there
are Syrian refugees registered in all 12 Governorates of Jordan, the evaluation focused on the
refugees registered in Zarqa, Irbid, Mafraq and Amman Governorates, which represent 90
percent of registered refugees.
Organizations and Consultants conducting the evaluation
This evaluation was conducted and led by International Solutions Group, a division of IMC
Worldwide, and its partner, the Interdisciplinary Research Consultants, a research firm based
in Amman, Jordan.
UNICEF Jordan was the evaluation’s commissioning organization.
1
Table of Contents THE WASH PROGRAMME AND CONTEXT ...........................................................................6 DESCRIPTION OF THE WASH PROGRAMME ..................................................................................6 PROGRAMME HISTORY ..................................................................................................................... 6 WASH PROGRAMME PURPOSE, COMPONENTS, AND MANAGEMENT ........................................................ 8 WASH PROGRAMME LOGIC MODEL AND RESULTS FRAMEWORK ............................................................. 9 PROGRAMME CONTEXT......................................................................................................... 12 MANAGING AN INFLUX OF REFUGEES FROM A RELATIVELY WATER-SUFFICIENT AREA .................................... 12 PRE-EXISTING POLITICAL ATTITUDES TOWARD REFUGEES ........................................................................ 13 LEGAL AND INSTITUTIONAL FRAMEWORK OF THE WASH SECTOR IN JORDAN ............................................ 13 ECONOMIC TENSION ...................................................................................................................... 13 FUNDING ENVIRONMENT ................................................................................................................ 14 DESCRIPTION OF THE EVALUATION ................................................................................. 15 PURPOSE AND OBJECTIVE ...................................................................................................... 15 SCOPE .............................................................................................................................. 15 EVALUATION METHODOLOGY......................................................................................... 15 DESK REVIEW ..................................................................................................................... 15 QUANTITATIVE ANALYSIS....................................................................................................... 16 PRIMARY RESEARCH: QUALITATIVE DATA COLLECTION METHODS AND SAMPLING ................................ 16 COVERAGE IN INTERVIEWS AND FOCUS GROUPS .................................................................................. 16 RAPID ASSESSMENTS AND USER INTERVIEWS ...................................................................................... 17 WASH MESSAGING AND PRACTICES ................................................................................................ 17 PARTNER INTERVIEWS .................................................................................................................... 17 DUTY BEARERS AND STAFF .............................................................................................................. 17 STAKEHOLDER INTERVIEWS.............................................................................................................. 17 ANALYSIS METHOD........................................................................................................................ 17 LIMITATIONS TO THE EVALUATION............................................................................................ 18 ETHICAL PRINCIPLES IN CONDUCTING THE EVALUATION ................................................................. 18 FINDINGS ....................................................................................................................... 19 RELEVANCE ........................................................................................................................ 19 EVALUATION QUESTION 1 ............................................................................................................... 19 EVALUATION QUESTION 2 ............................................................................................................... 26 EVALUATION QUESTION 3 ............................................................................................................... 27 EFFECTIVENESS ................................................................................................................... 29 EVALUATION QUESTION 4 ............................................................................................................... 29 EVALUATION QUESTION 5 ............................................................................................................... 53 EFFICIENCY ........................................................................................................................ 63 EVALUATION QUESTIONS 6 AND 7 .................................................................................................... 63 EVALUATION QUESTION 8 ............................................................................................................... 69 EVALUATION QUESTION 9 ............................................................................................................... 70 EVALUATION QUESTION 10 ............................................................................................................. 72 SUSTAINABILITY .................................................................................................................. 72 EVALUATION QUESTION 11 ............................................................................................................. 72 EVALUATION QUESTION 12 ............................................................................................................. 74 EVALUATION QUESTION 13 ............................................................................................................. 74 EVALUATION QUESTION 14 ............................................................................................................. 75 EVALUATION QUESTION 15 ............................................................................................................. 75 EVALUATION QUESTION 16 ............................................................................................................. 76
2
COVERAGE ......................................................................................................................... 77 EVALUATION QUESTION 17 ............................................................................................................. 77 EVALUATION QUESTION 18 ............................................................................................................. 79 COORDINATION .................................................................................................................. 79 EVALUATION QUESTION 19 ............................................................................................................. 79 EVALUATION QUESTION 20 ............................................................................................................. 80 CONCLUSIONS ................................................................................................................ 80 LESSONS LEARNED AND RECOMMENDATIONS ................................................................ 84 LESSONS ........................................................................................................................... 84 RECOMMENDATIONS ............................................................................................................ 86 WORKS CITED ................................................................................................................. 89 ANNEX A: EVALUATION TERMS OF REFERENCE................................................................ 92 ANNEX B: ANONYMIZED LIST OF INTERVIEWS ............................................................... 104 ANNEX C: LIST OF SITES VISITED .................................................................................... 105 ANNEX D: DOCUMENTS CONSULTED ............................................................................. 106 ANNEX E: DATA COLLECTION TOOLS ............................................................................. 112 ANNEX F: TEAM DESCRIPTION....................................................................................... 119 ANNEX G: EVALUATION MATRIX ................................................................................... 120 ANNEX H: UNICEF WASH PROGRAMME RESULTS FRAMEWORK..................................... 149 ANNEX I: UNICEF PROVIDED FINANCIAL ESTIMATES ...................................................... 155 ANNEX J: ADDITIONAL FINANCIAL INFORMATION ......................................................... 162
3
Table of Figures Figure 1 WASH Programme Implied Logic Model ................................................................ 11 Figure 2 Funds budgeted vs mobilised by year (USD in millions).......................................... 14 Figure 3 Public Latrine at Azraq camp .................................................................................... 23 Figure 4 The stand-by power generation increases the ability of the water supply and
distribution system to function................................................................................................. 24 Figure 5: Za'atari Camp 2012 (Source:Times of Israel, 29 August 2012) ............................... 33 Figure 6 Very clean household latrine observed in Za'atari camp. .......................................... 54 Figure 7 Rubbish Bin in Azraq ................................................................................................ 55 Figure 8 Power generators and fuel storage at Rukban water system. Extra power generation
and fuel storage capacity minimizes the potential for water system down time. .................... 58 Figure 9 Unimproved pit latrine cover at an ITS near Amman ............................................... 61 Figure 10 Cost of providing a cubic meter of water by year based on partner budgets .......... 65 Figure 11 Water provision cost savings in Za'atari (Source: UNICEF WASH Programme) .. 66 Figure 12 Membrane bioreactor wastewater treatment system functioning properly at Za'atari
Camp. ....................................................................................................................................... 67 Figure 13 Division of labor among UNICEF Partner organizations. (Source: WaSH in
Za'atari Snapshop, Jan-Feb 2014) ............................................................................................ 69 Figure 14 Percent of UNICEF Total Expenditure by Beneficiary Type (August 2012 – July
2017) ........................................................................................................................................ 71 Figure 15 WASH Programme expenditure by activity category (August 2012 – July 2017) . 71
Acronyms and Abbreviations ACF Action Contre la Faim
ACTED Agency for Technical Cooperation and Development
BPRM The Bureau of Population, Refugees, and Migration
CEDAW Convention on the Elimination of All Forms of Discrimination against Women
CRC Convention on the Rights of the Child
DAC Development Assistance Committee
FGD Focus Group Discussion
GIS Geographic Information System
GTZ Gesellschaft für Internationale Zusammenarbeit
IEC Information, Education, and Communication
IMF International Monetary Fund
INGO International Non-Governmental Organization
ISG International Solutions Group
ITS Informal Tented Settlements
JEN Government of Japan supported development agency
JHCO Jordanian Hashemite Charity Organization
JRP Jordan Response Plan
JVA Jordan Valley Authority
KAP Knowledge, Attitudes, and Practices
4
KII Key Informant Interview
MBBR Moving Bed Biofilm Reactor
MOE Ministry of Education
MWI Ministry of Water and Irrigation
NGO Non-Governmental Organization
NRC Norwegian Refugee Council
NRW Non-Revenue Water
OECD Organization for Economic Cooperation and Development
RAM Results Assessment Module
TDS Total Dissolved Solids
TOR Terms of Reference
UNHCR Office of the United Nations High Commissioner for Refugees
UNOPS United Nations Office for Project Services
UNRWA United Nations Relief and Works Agency
USAID US Agency for International Development
VIP Ventilated Improved Pit
WAJ Water Authority of Jordan
WASH Water, Sanitation, and Hygiene
WIS WASH in Schools
1
Executive Summary In March 2012, when a relatively small number of Syrians were fleeing the civil war into
Jordan, UNICEF responded by setting up Water, Sanitation, and Health (WASH) services
and facilities at border crossings, called transit centers. The initial numbers of refugees were
in the thousands rather than the much higher numbers that UNCHR was predicting, and some
members of the international press described the international community’s response and
predictions of a massive refugee influx alarmist and misleading (Seely, 2012).
By August UNICEF’s and UNHCR’s preparations paid off. One thousand refugees per week
crossing the border quickly became over 10,000 per week (UNHCR, 2012). Some refugees
found sponsorship and entered host communities, where they had access to household and
government WASH services. Others, unable to return home and without other options,
entered the Za’atari refugee camp that opened on 30 July 2012. By December, over 66,000
refugees were living in the camp (UNICEF, December 2012).
In July 2012, UNICEF was the only organization in Jordan that had the resources, capacity,
and institutional commitment to take leadership of the WASH response related to the Syrian
refugee crisis. UNICEF accepted responsibility for providing WASH services in a country
that was exhausted by previous refugee crises, water scarcity, a complex political
environment, and funding uncertainty. From July 2012 through July 2017, UNICEF provided
life-saving water and sanitation resources under these difficult conditions for approximately
400,0001.
The programme that UNICEF developed as its WASH response to the crisis was to become
one of the largest programmes in UNICEF Jordan’s portfolio. The scope of the programme
was initially a mandate to provide services to refugees in camps. It expanded to include
services for water-deprived Jordanian host communities that were resource constrained
before the crisis and were further burdened by the crisis. The WASH programme’s scope also
grew to provide urgent WASH services for refugees living in the border settlements at
Hadalat and Rukban. The emergency response required quick and flexible decision making
and efficient utilization of scarce resources, which UNICEF successfully performed.
The WASH programme was not without missteps or shortcoming, which the evaluation team
discusses in the full report. Overall however, the programme provided an effective response
in line with its core mission, providing life-saving WASH services in response to the Syrian
refugee crisis. It has also increased efficiency and worked to ensure coverage of vulnerable
populations. Two difficult achievements in a dynamic and complex environment.
By 2017, The WASH programme had evolved in response to its changing environment,
adopting a framework that worked at four levels:
1. Provision of equitable WASH services and the dissemination of WASH messages in
the four camps (Za’atari, Azraq, King Abdullah Park and Cyber City) and two
settlements in Rukban and Hadalat, to the most vulnerable people namely children
and women.
1 Evaluation Terms of Reference, page 4.
2
2. Provision of gender equitable support to the most vulnerable people in host
communities (in all 12 Governorates) namely to children and women.
3. Support to the WASH sector through technical support to the Ministry of Water and
Irrigation.
4. Coordination of the sector at camps and national level.
The WASH programme’s objective, its highest-level target, is that2 “Quality WASH facilities
are sustained and utilised, and hygienic behaviours practiced, by the most vulnerable while
ensuring the protection of the environment.”
The programme measures the achievement of this objective through four outcome indicators:
Indicator 1: Existence of strategic planning capacity in MoWI and sector and other
relevant institutions in Emergency preparedness and response.
Indicator 2: Proportion of population at camps and host communities including schools
using improved drinking water as per Jordan standards.
Indicator 3: Proportion of population at camps and host communities including schools
using improved sanitation facilities.
Indicator 4: Proportion of population practicing hygiene promotion at targeted camps,
host communities and schools.
Background and Context The WASH programme has faced complex external social, environmental, political, and
institutional challenges from its inception in 2012. Among the first was managing
expectations among the Syrian and Jordanian populations, particularly regarding the
provision of water to refugees in view of a population that had to pay for the resource. Other
challenges included Jordan’s economic crisis in 2012, managing perception of the refugee
population, and making do with financial resources that fell short compared to need
throughout the programme. UNICEF dealt with these obstacles while implementing a
complex WASH programme and serving as a conscientious steward of its donors’ resources
and interests.
Purpose of the Evaluation The purpose of this evaluation was to assess the degree to which UNICEF’s WASH
programme had achieved its intended results from its beginning in July 2012 through July
2017. The audience for the evaluation is current programme stakeholders, and future WASH
programme implementers that may benefit from its lessons learned. The evaluation’s
objective was to independently assess the degree to which the WASH programme’s design
and implementation was relevant, effective, efficient, and will lead to some definition of
programme sustainability. The evaluation also sought to discover if the programme had
covered all relevant populations, especially the most vulnerable, and the degree to which the
programme coordinated well with other similar initiatives and government strategy. UNICEF
excluded the impact criterion from the evaluation’s scope, stating that “due to the relatively
short time period since the start of the interventions.3
This evaluation covers WASH programme activities from July 2012 to July 2017.
Geographically, it covers programming in four camps (Za’atari, Azraq, Cyber City, and King
2 Results frameworks from 2013 – 2017 are included as Annex H
3 Evaluation Terms of Reference – page 4.
3
Abdullah Park), host communities, informal tented settlements, and Rukban and Hadalat,
the settlements on the Syrian/Jordanian border. It also covers refugees and Jordanians living
in host communities with high numbers of refugees.
Methodology ISG conducted a summative evaluation of UNICEF’s WASH programme. As described in
the evaluation’s terms of reference, the team used a mixed method approach that included the
review of UNICEF’s data and documentation, and the collection of qualitative data through
interviews and Focus Group Discussions (FGD). In addition, the team reviewed WASH
programme initiatives through on-site assessments.
The document review established the programme’s implied logic model and delineated its
goals and objectives. The review was also used to create interview and focus group guides.
As described in the evaluation’s ToR, quantitative analysis relied on a review of the
programme’s documents and other secondary sources.
Primary, qualitative data collection was carried out through interviews and focus groups with
stakeholders, beneficiaries, programme partners, UNICEF staff, donors, government officials,
and other key informants. The evaluation team used a rapid assessment methodology to
evaluate specific WASH programme interventions intended to improve water and sanitation
for targeted populations. The evaluation team interviewed UNICEF partner organizations that
were involved in the construction or rehabilitation of WASH facilities, the provision of
WASH services, and promotion of hygiene initiatives. The purpose of these interviews was to
understand partner perspectives on the degree to which interventions met evaluation criteria
and to understand ways in which management of the programme excelled and would benefit
from improvement4.
Findings The full evaluation report organizes findings according to the Organization for Economic
Cooperation and Development – Development Assistance Committee (OECD-DAC) criteria
included in the evaluation’s ToR, which include relevance, efficiency, effectiveness, and
sustainability as well as relevant humanitarian criteria. Findings are extensive as they cover
five years of programme operations across a variety of geographic contexts and the range of
programme activities. Here we provide a summary of the important findings.
Relevance: In camps and settlements interventions were relevant and appropriate in terms of
meeting rapidly changing WASH needs of targeted populations under difficult circumstances.
The programme’s context was particularly difficult in its early phase. UNICEF maintained
programme relevance during the winter storms, outbreaks of illness, and other obstacles to
smooth management (such as trucker strikes) that arose throughout the 2012-2017 period.
During 2012 and 2013, UNICEF determined needs by estimating the quantity of water,
sanitation services capacity, and numbers of facilities that people displaced by the Syrian
civil war required at transit centers and in Za’atari, King Abdullah Park and Cyber City. In
this period UNICEF designed and implemented water and sanitation service options and level
of services based on its analysis of the dynamic emergency response context, opinions of
partners and other stakeholders, and the resources it had available. Beginning in 2014,
UNICEF used an assessment process to maintain the relevance of its interventions. Examples
of these assessments include a multi sector assessment of ITSs conducted by REACH in
2014, and assessments in the Za’atari and Azraq camps in 2015 and 2017. Other means of
4 A list of partners that the team interviewed is included in Annex B.
4
maintaining relevance were implemented throughout the programme and are discussed in the
full report.
UNICEF has made crucial improvements to water and wastewater infrastructure in host
communities and informal tented settlements. However, it is difficult to judge the relevance
of these interventions as many other international organizations provide services in these
communities and the degree to which the UNICEF programme addresses needs generated by
the refugee crisis is unclear.
Effectiveness: In the provision of equitable WASH services and dissemination of WASH
messages in the four camps, the WASH programme achieved high levels of effectiveness for
the targeted population across all years. The programme also very effectively supported the
WASH sector through technical support to the MoWI, as well as National WASH in School
Standards. Government officials at the MoWI and WAJ reported satisfaction with the WASH
programme’s contribution to planning documents, strategies, and policies. UNICEF’s
achievement included successful coordination of the WASH sector from the camp to the
national level. The programme met its targets under each of these activity areas. The WASH
Programme struggled to meet its targets in its activities focused on supporting host
communities, including WASH in schools.
Efficiency: The WASH programme worked continuously to improve the efficiency of its
operations. As an example, the evaluation team estimates that between 2013 and 2016, the
cost of delivering water to refugees in the camps dropped from 3.34 Jod/m3 to 0.24 Jod/m3.5
The programme made similar efficiency gains in sanitation services and solid waste
collection. Other Innovations led to more efficient coverage and management of services,
such as introducing a voucher system for contractors performing desludging that made them
accountable to camp residents and reduced potential corruption.
Sustainability: UNICEF has done everything possible to ensure that its interventions are
long-lasting and that mechanisms are in place for their operation. Examples include
organizing WASH committees to oversee water access and hygiene promotion, constructing
systems out of durable materials, and implementing messaging campaigns for hygiene
promotion and water conservation. However, the ultimate sustainability of the programme is
a complex issue. It includes determining if and when UNICEF will hand WASH
management off to another entity, managing the transition of refugees either back to their
place of origin or a permanent settlement elsewhere, the dismantling or integration of camp
WASH infrastructure into Jordan’s municipal systems, and dealing with Jordan’s medium-
term water scarcity issues.
Coverage: The degree to which the programme is focused on the most vulnerable people is
influenced by the WASH programme initial mandate, which is to maintain WASH service
levels and standards in the camps. UNICEF maintains initiatives that monitor coverage in the
camps and works with partners to ensure vulnerable populations receive necessary services.
These initiatives work specifically to monitor services to people with disabilities, identify
people who meet UNICEF’s vulnerability criteria, and ensure that interventions target
vulnerable people and households.
While people entering the camps may have been the most vulnerable at the programme’s
outset, the evaluation team believes that in later years occupants of ITSs were more
vulnerable and received less attention from the WASH programme. However, the necessity
5 Figures based on estimates for the cost of water delivery included in partner organization budgets from 2012 –
2017.
5
of maintaining service in the camps combined with budgetary shortfalls has prevented
UNICEF from identifying and reaching that population extensively.
Coordination: UNICEF greatly minimized duplication of services and service gaps in
camps. Its role as WASH sector lead allowed it to assign sectors of Za’atari to the
responsibility of partner organizations and to set standards among those organizations. It
created similar standards and management arrangements in other camps. As sector lead,
UNICEF implemented mechanisms for coordination in the face of emergencies, such as
winter storms, and established third party monitoring systems that allow for quality and
security incident reporting.
UNICEF partners and GoJ officials that the evaluation team interviewed stated satisfaction
with UNICEF as sector lead. One partner said that, “UNICEF … always (has) staff in
Azraq.” A donor said that they were very happy with UNICEF’s coordination and
partnership. A staff member of one partner said, “UNICEF does a very good job of
controlling the work.”
Lessons Learned and Recommendations The full report includes sixteen lessons for the reader’s consideration. Most important among
them are the following:
Quick decision making, and decisive action saves lives. The decisions and actions UNICEF
took in the early days of the programme were crucial for initial inhabitants of Za’atari. In the
settlements, UNICEF provided water and sanitation services to respond to the settlements’
exigencies. UNICEF moved quickly and effectively, with clear goals, such as ensuring that
each person had at least 15 liters of water per day. The WASH programme also recognized
that its intervention would have an impact on the larger Ruwaished host community. To
address that impact, UNICEF rehabilitated the community’s water and sanitation facilities
and provided support to identified vulnerable HHs.
Installing quality water treatment and delivery equipment is crucial for the long-term
sustainability and use. UNICEF’s installed higher cost and higher quality water infrastructure
at Za’atari, Rukban and Azraq, including the treatment plant for Zaatari. In each case, a
robust design was followed by very good installation and excellent post construction
operation and maintenance. This was borne out through the evaluation team’s on-site
investigations. In a key-informant interview (KII) with the contractor in Rukban, the
operators noted that UNICEF always purchases high quality equipment. The result is
operational efficiency, equipment that is long-lasting in the harsh environments, and reduced
maintenance costs, downtime, and interruptions to supply.
When direct consultation is not possible, other means of assessment can ensure programme
relevance. In addition to direct consultation with beneficiaries, UNICEF put a system of
assessments and Knowledge, Attitudes, and Practices (KAP) surveys in place. These studies
kept UNICEF staff and sector participants informed of changes in the camps and other
locations and assisted the programme in staying relevant and responding quickly to
eventualities.
Based on the documentation that was shared with the evaluation team as well as its
interviews with UNICEF staff, partners, and other stakeholder, the programme appears to
lack analytical rigor for determining the programme’s objectives, indicators or resource
allocation, which has led it to overcommit to initiatives for which it did not have resources or
capacity, primarily in host communities. The WASH programme should implement a
monitoring and evaluation system, and comply with the strategic programme development
protocols and reviews that are detailed in the Jordan Country Programme 2013-2017.
6
The WASH Programme and Context The object of this evaluation is UNICEF’s response to the Water, Sanitation and Hygiene
(WASH) needs in Jordan as a result of the Syrian refugee crisis (July 2012 to July 2017).6
The response took the form of a programme of activities that began as an effort to provide
WASH facilities and services to refugees entering Jordan as a result of the Syrian civil war.
At its peak, the programme served refugees in six camps and settlements, host communities
and informal tented settlements (ITS) across Jordan, and national government ministries. The
programme’s main activities included:
• Provision of water, sanitation, and hygiene facilities and services in the camps and
settlements the programme serves.
• Promulgation of hygiene and water conservation messages to refugees and host
community members.
• Provision of WASH infrastructure support in host communities to benefit host
community members, refugees, and inhabitants of ITSs.
• Promotion of Wash in School (WIS) improvement, including infrastructure support, a
nationwide assessment, message promulgation, and capacity building for government
ministries to better support WIS.
• Delivery of technical support to the Ministry of Water and Irrigation (MoWI) to
develop strategies and accompanying documents, and leading sector planning.
• Coordination of the WASH sector in camps and at the national level.
Description of the WASH programme
Programme History UNICEF Jordan launched its WASH programme in Jordan in response to the influx of Syrian
refugees during the first half of 20127. Prior to the programme’s launch in July 2012,
UNICEF provided WASH services at Jordanian transit centers as an activity under
UNICEF’s health programme8. The centers, which acted as Syrian refugees’ entry point into
Jordan, predated the refugee camps UNICEF served as part of the WASH programme. Syrian
refuges at transit centers that could prove that they had a sponsor in a Jordanian host
community stayed at a transit center for one week, and then were permitted into Jordan.
UNICEF’s WASH activities at transit centers included providing toilets, water tanks, water
trucking to fill the tanks, and sanitation services to these centers9.
As the number of refugees at transit centers increased, the Government of Jordan, the United
Nations High Commissioner for Refugees (UNHCR), and the Jordanian Hashemite Charity
6 Evaluation Terms of Reference, page 2.
7 UNICEF did not participate in the WaSH sector before the refugee crisis (Source: Evaluation Terms of
Reference page 4.)
8 Evaluation Terms of Reference, page 5
9 Source: Interview with UNICEF Staff- 26 July 2018
7
Organization (JHCO) obtained a tract of land owned by Jordanian’s armed forces to design
and build the Za’atari refugee camp (Ledwith, 2014).
UNHCR and its partner agencies cooperated to determine how to delegate the work for
managing the crisis and the camp10. In the delegation of responsibilities, UNHCR asked
UNICEF Jordan to take responsibility for leading the WASH sector. UNICEF was quick to
rally expertise and funding for WASH programme management11.
UNICEF’s primary role as WASH sector lead in July 2012, the start date of the scope of this
evaluation, was to coordinate WASH activities for the Za’atari refugee camp as well as the
Cyber City and King Abdullah Park camps, which by January 2013 housed approximately
1,500 people (UNICEF Jordan, 2013). Coordinating WASH during this phase was an
expensive endeavor, further complicated by the Government of Jordan’s moratorium on
building permanent structures in Za’atari. The response in this phase was expensive because
UNICEF had to rely on contractors to provide water and mobile sanitation facilities quickly
during the fast-moving crisis. UNICEF’s WASH team had five days to prepare for the
opening of Za’atari camp12. The WASH team’s first objectives were to provide contingency
supplies, supply water, acquire mobile toilets, and engage a contractor to provide desludging
services.13
The WASH team also sought to engage partner NGOs to undertake WASH infrastructure
construction, operations, and hygiene promotion in the camp. In Jordan, two organizations
were available and had the necessary resources on hand to serve as partner organizations.
UNICEF signed an agreement with one of the partners. However, that partner backed out of
the agreement two days before the camp opened, forcing UNICEF to request that the
remaining partner, the German Federal Agency for Technical Relief (THW) mobilize
quickly. THW mobilized in 24 hours and was ready to serve the camp when it opened14.
By September of 2012, the pace of new arrivals slowed from 100 per day to 50 per day15.
UNICEF advocated with the Water Authority of Jordan to transition to more efficient means
of serving the refugee population living in Za’atari. This advocacy resulted in UNICEF
building WASH blocks to serve refugees living in Za’atari. WASH block construction started
in the part of the camp where the first arrivals lived and moved to the newer areas. UNICEF
also transitioned sanitary facilities away from mobile toilets to sealed pits and steel tanks
made for sewage collection, commissioned Mercy Corps to drill and install two boreholes in
the camp, and partnered with Japan Emergency NGO (JEN) and Relief International (RI) to
provide hygiene promotion. In late 2012, UNICEF began serving host communities, starting
with water system construction in Ramtha and Mafraq (UNICEF Jordan, 2012).
10 The evaluation team did not have access to programmatic documentation, such as proposals from UNICEF or
agreements with donor agencies, that would have described why UNICEF was awarded responsibility for
WaSH activities, or what UNICEF proposed. The presented information comes from evaluation interviews.
11 Source: UNICEF Staff Interview – 26 July 2018
12 Ibid.
13 Ibid
14 ibid
15 ibid
8
As it became apparent at the end of 2012 and beginning of 2013 that the refugee crisis would
not soon subside, UNICEF Jordan developed a results framework that included output level
WASH results for its 2013 and 2014 commitments. These included ensuring that relevant
core commitments on WASH were met for Syrian refugees and affected Jordanians in all
humanitarian settings (Output 1.3)16. The core commitments provided for:
• Establishing a working group to coordinate the WASH sector,
• Providing safe access to water for drinking and domestic use,
• Improving sanitation facilities,
• Providing hygiene items for those that needed them,
• Promoting hygiene messages,
• Providing for WASH services in schools, and,
• Working with the MoWI and sector actors to address the impact of the influx of
refugees on the sector’s longer-term chronic needs.
UNICEF’s targeted WASH results changed between 2014 and 2015 as the programme
aligned its objectives with government policies and sought to run its WASH operations more
efficiently. In Za’atari UNICEF found that camp inhabitants preferred private WASH
facilities to community WASH blocks and converted WASH infrastructure to meet that
demand, resulting in the phasing out of 400 WASH blocks and working with the community
to set standards for private facilities17. In 2014, a new camp, Azraq, was completed and
UNICEF led its WASH activities as well. By 2016, UNICEF’s targeted results included
working with the national government to finalize and disseminate the new policies, conduct
research on water conservation, create standards for WASH in schools, promote water
conservation and hygiene among the most vulnerable populations in host communities,
camps, ITSs, and to provide for the WASH needs of vulnerable women and children through
contracted tankering and also water networks in the camps and host communities.
WASH programme Purpose, Components, and Management The WASH programme’s purpose evolved as the programme’s context changed from
emergency response to longer-term operations. From 2012 to early 2014, the programme’s
purpose was to provide for the immediate WASH needs of refugees arriving from Syria, and
to respond to contingencies, such as the tensions that developed in host communities, as they
arose18. In 2014 UNICEF began the process of planning for more efficient, longer-term
operations in refugee camps. An example of an initiative developed through this process is
the construction of a water network in Za’atari camp, which reduced the cost of water
provision, improve the sustainability of the system, and make access to water more
equitable19. In 2014 UNICEF also contributed to the Jordan Response Plan (JRP) 2015,
which sought to “bridge the divide between resilience and humanitarian systems,” (Ministry
of Planning and International Cooperation, 2014). UNICEF’s WASH programme coordinated
with the JRP by formally planning for host community interventions, leading on the refugee
response and supporting the government on the resilience component. The UNICEF WASH
programme also aligned itself with the GoJ National Water Strategy 2016 - 2025, assisting
16 UNICEF’s WaSH results framework is included as Annex H.
17 Source: UNICEF Staff Interview – 26 July 2018
18 Source: UNICEF Staff Interview – 6 August 2018
19 Additional examples are discussed in the Efficiency section of the Findings chapter of this report.
9
the government to meet its objectives regarding efficiency and the National WASH in
Schools Assessment (Ministry of Water and Irrigation, 2016).
By 2017, the WASH programme divided its activities into four activity components:
1. Provision of equitable WASH services and the dissemination of WASH messages in
the four camps (Za’atari, Azraq, King Abdullah Park and Cyber City) and two
settlements in Rukban and Hadalat, to the most vulnerable people, namely children
and women.
2. Provision of gender equitable support to the most vulnerable people in host
communities (in all 12 Governorates) namely to children and women and including
ITSs.
3. Support to the WASH sector through technical support to the Ministry of Water and
Irrigation.
4. Coordination of the sector at camps and national level.
As July 2017, the end date for the scope of this evaluation, UNICEF continued to implement
the WASH programme along the four components described above.
The WASH programme’s budget was approximately $355 million USD from 2013 to 2017. It
averaged approximately $71 million USD per year.
WASH Programme Logic Model and Results Framework The WASH programme has operated without an explicit strategy document or logic model to
guide its implementation. The evaluation team also found no evidence of a programme
document that would have detailed donor expectations. However, each phase of the
programme demonstrated an implicit logic model that has evolved20.
In the first phase of the programme, July through December 2012, the evaluation team found
no evidence of a planned structure to the programme, such as a results framework or
workplan21. The first results framework set objectives, outcomes, and output indicators for
the years 2013 and 2014. Outcome level indicators in this framework were not entirely
focused on WASH goals. The WASH programme still used a general UNICEF outcome
framework that included indicators such as “existence of a comprehensive national
monitoring system for school readiness to inform policy decisions on child disparities,” and
“existence of a child and neonatal information system in support of child and maternal death
audits.” The 2013/2014 output indicators seek to measure results from general activities and
do not disaggregate achievement by targeted population. For example, they measure the
cumulative “number of people provided with safe access to sufficient water for drinking and
domestic use.” An output indicator from the 2013/2014 framework states “Relevant Core
Commitments on WASH are met in all humanitarian settings.”
In 2015, the WASH programme set the objective level indicators that it used at least through
December 2017. In this framework, outcome indicators disaggregated by target population
are specific to the WASH programme, such as “Proportion of population at camps and host
communities including schools using improved drinking water as per Jordan standards.”
Output targets and indicators are also adapted to the WASH sector. Examples include,
“children, parents and community members are aware of and practice hygiene and water
conservation,” and, “Number refugees provided with safe access to sufficient water for
drinking and domestic use through water trucking.” While those indicators are not
20 The WaSH programme’s results frameworks from years 2012 to 2017 are attached as Annex H.
21 After the conclusion of the evaluation, UNICEF provided a workplan for the 2016/2017 period.
10
disaggregated by target population, the baseline data collected in support of them was, as was
the end of year reporting that the WASH programme completed.
In 2016, the results framework kept the same outcome level indicators, and evolved to
provide an indicator for each population that the programme targeted. Examples include
“Number of people in camps supplied with water (tankering or network),” and, “Number of
people with access to improved WASH facilities in institutions.”
By the most recent results framework that falls within the scope of this evaluation, which
covers the period from 2015 to 2017, an implicit logic model had evolved. The WASH
programme’s documentation implies a logic model that seeks to provide an effective response
to the refugee crisis while protecting Jordan’s peace, prosperity, and resources. The
programme has sought to achieve that objective by attempting to either implement or
coordinate WASH activities in four areas:
• Building capacity at the national level to equitably provide WASH services,
• Promoting hygiene and water conservation practices in camps, host communities,
schools, and ITSs,
• Providing WASH services directly for vulnerable people, and,
• Building the capacity of institutions, utilities, and other organizations to provide
equitable access to water and sanitation services.
The WASH programme states its objective, its highest-level target, as follows22: “Quality
WASH facilities are sustained and utilised, and hygienic behaviours practiced, by the most
vulnerable while ensuring the protection of the environment.”
The programme measures the achievement of this objective through four outcome indicators.
• Indicator 1: Existence of strategic planning capacity in MoWI and sector and other
relevant institutions in Emergency preparedness and response.
• Indicator 2: Proportion of population at camps and host communities including
schools using improved drinking water as per Jordan standards.
• Indicator 3: Proportion of population at camps and host communities including
schools using improved sanitation facilities.
• Indicator 4: Proportion of population practicing hygiene promotion at targeted camps,
host communities and schools.
Each outcome indicator’s achievement is in turn defined by output-level indicators. The
implied logic model is illustrated in figure 1.
22 Results frameworks from 2013 – 2017 are included as Annex H
11
Figure 1 WASH Programme Implied Logic Model
:
12
The programme’s outcome statement and output statements, as articulated in the above logic
model, come from UNICEF Jordan’s 2016 annual report (UNICEF Jordan, 2016)23. The
activities reflect the reported activities and accomplishments of the WASH programme. The
results framework reflects a broad measurement of achievement under three of the four
programme components; briefly, 1) support to the camps and settlements,24 2) support to host
communities and ITSs, and 3) support to the WASH sector through technical support to the
MoWI. The results framework does not include a direct measurement of the fourth
component - Coordination of the humanitarian WASH sector at a camp and national level.
However, the degree of the programme’s achievement under the coordination component is
closely related to the degree of achievement under the other three components.
Programme Context The WASH programme faced difficult social, political, and institutional challenges from its
inception in 2012.
Managing an influx of refugees from a relatively water-sufficient area
There is a large disparity in access to water resources among countries in the Middle East.
Regionally, the average per capita total renewable water resources is 645 m3, however, the
median country, Palestine, has 179.3 m3 for each of its residents25. Jordan and Syria tend
toward opposite ends of the water access spectrum. Jordan has access to 123.4 m3 per capita,
ranking the country below the median and well below the average for the region. Syria has
greater than average access to renewable water resources, amounting to 908 m3 per capita26.
The difference in access to resources sets the scene for the challenges that UNICEF would
face in supplying a population used to ample water resources who had relocated to a region
with far less water (UNICEF, 2012). Jordan’s largest refugee camp is the Za’atari camp
located in Mafraq governorate. In 2013, 90 percent of the population of the Za’atari refugee
originated in Daraa Syria. Daraa is a well-irrigated agricultural region, sourcing water from
groundwater (wells), or the ample (but steadily declining) lake Muzayrib (Leestma, 2017),
(Salman & Mualla, 2008) People from Daraa fled the Syrian civil war and began relocating
to Za’atari camp in the Mafraq governorate in 2012. UNICEF’s challenging task was to meet
the daily water needs of the refugee population, which fell short of the quantity of water
refugees perceived that they needed, and balancing that against local Jordanian’s perspective
that their scarce resources were being extracted and given away (REACH, 2014).
Za’atari is located in the Mafraq governorate and has grown to be the 4th largest urban
settlement in Jordan. In Mafraq, like much of Jordan, nearly all households are covered by a
piped water network. However, water is only supplied one or two days a week into household
storage containers. While the frequency of delivery is common for urban areas in the region,
water is delivered less frequently to Mafraq than Za’atari. When that water is consumed,
residents must purchase water privately. Jordanian’s living in Mafraq had stretched declining
resources before the new influx of refugees increased water demand by 21 percent
23 The annual report can be found at the link below. Note that while the report lists 5 output statements, the
activities under output 1 and output 4 are nearly identical, and the output statements are quite similar:
https://www.unicef.org/about/annualreport/files/Jordan_2016_COAR.pdf
24 Settlements such as Rukban are not explicitly mentioned in the results framework.
25 FAO Aquastat: http://www.fao.org/nr/water/aquastat/data/query/results.html
26 Ibid.
13
(International Monetary Fund, 2017). By mid 2013, tensions over those resources would lead
to local animosity and protest (Wildman, 2013).
Pre-existing political attitudes toward refugees Within two years of Jordan’s independence in 1946, the Kingdom had to dedicate resources
to host the approximately 100,000 refugees fleeing the 1948 war that established Israel as a
nation. In 1967, war broke out again and 140,000 additional people fled into Jordan (Bocco,
2010). Over the subsequent decades, further inflows of refugees included Iraqi refugees in the
1970s, 80s, and 90s; 300,000 Jordanian expatriates returning unexpectedly in the 90s, and
with the second war in Iraq in the early 2000’s approximately 750,000 returnees and refugees
(Al Wazani, 2014).
As a result of previous experiences hosting refugees, the GoJ has made decisions that
influence the current refugee crisis and particularly the WASH programme. These decisions
are discussed throughout this document. Examples of decisions that influenced the program
were the GoJ’s initial decision to not allow the construction of permanent structures in
Za’atari refugee camp, and the GoJ’s steadfast decision that they will not take ownership over
the refugee camps, including the water network and supply of WASH materials and services.
Legal and Institutional Framework of the WASH sector in Jordan One benefit of having low water usage per capita is that Jordan has achieved nearly universal
coverage in water services (International Monetary Fund, 2017). Achieving strategic results
such as universal coverage is a result of the water sector’s institutional and policy framework.
The central institution responsible for the water management in Jordan is the Ministry of
Water and Irrigation (MoWI). Under the MoWI, there are 15 units that manage
administrative and technical functions. Included among these units are two agencies that deal
with water issues and policies in Jordan and report directly to the Minister; The Water
Authority of Jordan (WAJ) and the Jordan Valley Authority (JVA). The WAJ manages
municipal water supply and wastewater services. It also manages operations, maintenance,
planning and construction. Municipal water utilities fall under WAJ and are responsible for
managing subscription issues and their own facilities. The JVA is responsible for the “social
and economic development of the Jordan River Valley” including the development and
protection of water resources (Kis, 2016).
Several other institutions are important to Jordan’s water management. There are three water
companies, The Yarmouk Water Company, The Aqaba Water Company, and the Miyahuna
Water Company. Other ministries that play a role in water management are the Ministry of
Agriculture and the Ministry of the Environment.
The Ministry of Health regulates drinking water quality in Jordan. It monitors sources for
water quality and inspects potential sources of pollution. The Ministry of Health issues
permits for all produced or imported potable water.
Economic Tension In 2012, as Syrian refugees were entering into Jordan and the newly constructed Za’atari
camp. initially, patient Jordanians watched as free water and other resources, carried by a
large number of trucks flowed into the camp (UNICEF, 2012).
Meanwhile, Jordan was confronting an economic crisis. In March of that year, the
International Monetary Fund (IMF) determined that the country’s debt levels were
dangerously high given its economic turndown as a result of the 2008 global financial crisis
(International Monetary Fund, 2017). The IMF recommended a series of measures meant to
14
stabilize the economy, including cutting subsidies for fuel.27 By November, protests erupted
in Amman and Irbid.28 The image of Syrians receiving daily water deliveries extracted from
depleted Jordanian resources while poor Jordanians were made even poorer exacerbated
tensions in camps and in communities (Luck, 2013).
Funding Environment In 2018, international donors have only provided approximately 43 percent (Financial
Tracking Service, 2018) of what is required to respond to the regional Syria Refugee Crisis,
and about 34 percent of the cost of providing a WASH sector response (Financial Tracking
Service, 2018). In 2017 donors funded 52 percent of the requirements of the regional
response plan for the Syrian Crisis (ReliefWeb, 2017). The international community’s
difficulty in coming up with sufficient funds to sustain an effective response to the crisis has
created a heavy burden for UNICEF’s WASH programme in Jordan. The lack of funding has
particularly reduced UNICEF’s ability to serve host communities and ITSs (UNICEF Jordan,
2013)29. Figure 2 below shows the amount UNICEF budgeted for interventions across
programs for the Jordan Response Plan against the amount that was raised and spent by
year30.
Figure 2 Funds budgeted vs mobilised by year (USD in millions)31
27 https://www.imf.org/external/pubs/ft/scr/2012/cr12119.pdf 28 https://uk.reuters.com/article/uk-jordan-protests-idUKBRE8AD00F20121114 29 Lack of funding was also the reason given for missing host community targets even in 2016, when UNICEF
Jordan exceeded its fund raising targets. 30 UNICEF Jordan notes that the table includes rolled over funding and earmarked amounts totaling $30 million
from German donors for Za’atari. 31 The 2015 Annual report included funds mobilized through November. The chart adds an additional 1/12 to
funds mobilized to estimate funds utilized in December.
0
50
100
150
200
250
300
2013 2014 2015 2016 2017
Budget Mobilised
15
Description of the Evaluation
Purpose and Objective The purpose of this evaluation was to assess the degree to which UNICEF’s WASH
programme had achieved its intended results from its beginning in July 2012 through July
2017. UNICEF intends to use the lessons that the evaluation process generates to inform
future WASH programmes that occur in a similar context.
The Evaluation’s objective was to independently assess the degree to which the WASH
programme’s design and implementation were relevant, effective, efficient, and will lead to
some definition of programme sustainability. The evaluation also sought to discover if the
programme had covered all relevant populations, especially the most vulnerable, and the
degree to which the programme coordinated well with other similar initiatives and
government strategy. UNICEF excluded the impact criterion from the evaluation’s scope,
“due to the relatively short time period since the start of the interventions.32”
The evaluation questions are included in the findings section of this report, and in Annex A:
Evaluation Terms of Reference.
Scope This evaluation covers UNICEF’s WASH programme from July 2012 to July 2017. This
evaluation assessed WASH programme activities, including water provision, sanitation
facilities and services, hygiene promotion, solid waste management, desludging, WASH in
Schools initiatives, WASH sector coordination, and support to the Government of Jordan
through the Ministry of Water and Irrigation (MoWI). Geographically, it covers
programming in four camps (Za’atari, Azraq, Cyber City, and King Abdullah Park (KAP),
host communities, informal tented settlements, and Rukban and Hadalat, the settlements on
the Syrian/Jordanian border. It also covers refugees and Jordanians living in host
communities with high numbers of refugees.
Evaluation Methodology ISG conducted a summative evaluation of UNICEF’s WASH programme. The team used a
mixed method approach that included the review of UNICEF’s qualitative and quantitative
data, and the collection of qualitative data through interviews and Focus Group Discussions
(FGD). In addition, the team reviewed WASH programme initiatives through on-site
assessments. The Evaluation Matrix, included as Annex G, provides the evaluation questions,
detailed answers to those questions, the key judgement criteria that was used in constructing
the evaluation team’s response to evaluation questions. It also includes the sources of
information that provided evidence for the response to each question. The matrix is
organized according to the evaluation questions in the evaluation’s ToR and by evaluation
factors, namely; relevance, effectiveness, efficiency, sustainability, coverage, and
coordination.
Desk Review The desk review established the programme’s implied logic model and delineated its goals
and objectives. UNICEF provided the evaluation team with an initial set of documents at the
start of the evaluation. The evaluation team reviewed these documents to understand how the
WASH programme evolved as the refugee crisis developed, estimate the WASH
32 Evaluation Terms of Reference – page 4.
16
programme’s achievements related to goals and objectives, understand how programme
management determined strategies, set targets, designed work, and the degree to which this
planning led to successful outcomes. For the inception phase of the evaluation, the desk
review was also used to create interview and focus group guides. The evaluation team also
noted where information was incomplete or missing and provided requests to UNICEF for
additional documentation to fill in information gaps over the course of the evaluation.
Document collection and review occurred throughout all phases of the evaluation. The list of
documents that the evaluation team reviewed is included in Annex D, with additional
documents listed in the Works Cited section.
Quantitative Analysis As described in the evaluation’s ToR, quantitative analysis relied on a review of the
programme’s documents and other secondary sources. The review identified the
programme’s goals, objectives, and output targets for each year of implementation and
sought to understand the degree to which targets were met. The indicator framework that
informed this analysis is included as Annex H to this report33.
The evaluation team also used expenditure reports to attempt to understand the programme’s
financial management and generate analysis for the questions related to efficiency in the
evaluation’s TOR. UNICEF’s financial information provided to the evaluation team was
incomplete and not well documented. UNICEF does not track expenditure by year,
programme component, or result. It also does not track indirect costs associated with the
programme’s implementation. These limitations prevented a detailed analysis of efficiency.
The evaluation team estimated costs from an incomplete list of the WASH programme’s
expenditures and estimates included in partner organization’s budgets. Documents that
included quantitative data are also included in Annex D.
Primary Research: Qualitative Data Collection Methods and
Sampling Qualitative data collection was carried out through interviews with stakeholders,
beneficiaries, programme partners, UNICEF staff, donors, government officials, and other
key informants. The evaluation team also conducted FGDs among programme beneficiaries.
The goal of interviews and FGDs was to capture information representing the range of
experiences among each targeted population group, with a focus on the most vulnerable.
Thus, for interviews, the evaluation team utilized a purposive sampling strategy in camps,
host communities and ITSs. The number of interviews for each targeted population sought to
fully answer each research question to the point that each source of information is saturated.
The number of interviews location and type is included in Annex B34. Interview and focus-
group guides are included in Annex E.
Coverage in Interviews and Focus Groups The evaluation team sought to ensure that the views of all different groups covered by the
WASH programme were represented through interviews and FGDs. Interviews and focus
groups included women and men, people living near resources and more distant, people new
33 Annex H details information extracted from UNICEF’s annual workplan and End of Year Reports.
34 The number of interviews assumes that the team is able to access each camp and host community and that
appropriate respondents are available and willing to participate in interviews.
17
to their areas of inhabitation and people that have lived in the area for a long time. However,
the numbers of people interviewed does not represent all groups equally across topic areas.
Rapid Assessments and User Interviews The WASH programme carried out rehabilitation and new construction of WASH
infrastructure, and organized WASH services to improve water access and sanitation. The
evaluation team used a rapid assessment methodology to evaluate specific projects intended
to improve water and sanitation. Infrastructure evaluations included facility inspections and
interviews with key informants about the infrastructure, including how it was and is being
utilized. Through this approach the evaluation team developed a clear picture of the
construction quality and operational effectiveness of the infrastructure.
The team evaluated UNICEF interventions in two parts. The first part was the observation
component. In this component, the evaluator inspected the intervention and determined where
critical control points or bottlenecks are apparent, while at the same time observing how
people interact with the intervention. The second component comprised qualitative data
collection. During this component the evaluator interviewed people who had varying levels
of access to the intervention. The evaluator sought out people that should have had access to
the intervention but don’t to understand why access is limited.
WASH Messaging and Practices The evaluation assessed the WASH programme’s messaging and behaviour change activities
according to the evaluation criteria. The evaluation explored the degree to which messages
were absorbed and converted into new practices by reviewing the data provided for desk
review and analysing the information gathered in stakeholder interviews.
Partner Interviews The evaluation team interviewed UNICEF partner organizations that were involved in the
construction or rehabilitation of WASH facilities, the provision of WASH Services, and
promotion of hygiene initiatives. The purpose of these interviews was to understand partner
perspectives on the degree to which interventions met evaluation criteria and to understand
ways in which management of the programme excelled and would benefit from
improvement. A list of partners that the team interviewed is included in Annex B.
Duty Bearers and Staff The evaluation team coordinated with UNICEF Jordan to interview appropriate duty bearers
in government ministries, utilities companies, municipalities, and communities. These
interviews included, the MoWI, the Yarmouk Water Company, and the Water Authority of
Jordan. These duty bearers have assisted UNICEF in program delivery and/or benefited from
UNICEF’s capacity building and sector coordination activities. The evaluation team also
interviewed UNICEF administrative, programmatic, and senior staff.
Stakeholder Interviews Where applicable, the evaluation team interviewed stakeholders that had an overview of the
WASH programme’s implementation in their community. These stakeholders include WASH
committee members, camp or host community duty bearers, and other stakeholders that
UNICEF staff thought would offer insight into the programme.
Analysis Method Durng the desk review and after field visits, the ISG Team coded qualitative data to ensure
that themes matched evaluation criteria and that issues of gender, vulnerability, and
protection were highlighted. Coding the data facilitated identifying clear themes across
primary and secondary research.
18
Much of UNICEF’s quantitative data is maintained in unsearchable .pdf documents. To
extract as much of the data as possible, the evaluation team used optical character recognition
software to recover as much quantitative information as possible. This data was particularly
useful in answering questions related to efficiency and effectiveness.
Limitations to the Evaluation There are several factors that may limit the evaluation’s analysis. These are listed below
• Quality of data provided by UNICEF – The evaluation team’s ability to verify
indicator targets and select appropriate respondents for qualitative data collection
depends on the quality and organization of the WASH programme’s data. Populations
that have participated in the programme, but for which no records were kept, or
records are incomplete, may have received insufficient treatment by the evaluation.
These populations include host communities benefiting from infrastructure
improvements and residents of camps that have closed. Records for the early years of
the programme may have been incomplete and/or poorly organized.
• Financial and Procurement Data Provided by UNICEF – Efficiency and value for
money calculations depend on the reliability of the financial and procurement data
that UNICEF provides. UNICEF does not track financial information by result,
activity, programme component, or year. UNICEF also does not track indirect costs
allocable to a specific programme. This evaluation provides an analysis of the data
available to the evaluation team, but cannot audit its accuracy or reliability.
Evaluation findings regarding efficiency and value for money are estimated.
• Sample Selection – The qualitative data collection employed by this evaluation relied
on purposeful sampling rather than utilizing random sampling. While the qualitative
data collection creates a narrative around the evaluation’s criteria, it also suffer from
particular biases, including UNICEF’s ability to provide information about
appropriate project sites, time restrictions in data collection, and access to small
numbers of beneficiaries. Additionally, the evaluation was conducted during summer
months, when most schools were closed, which limited the evaluation team’s
collection of data related to Wash in Schools.
• Availability of Respondents – Many of the WASH programme’s activities were
carried out four or more years ago. Many interventions may have been carried out
without the knowledge of direct beneficiaries, especially those that involved general
infrastructure or were implemented through partners or contractors. The lack
availability of respondents limited the number of interviews and focus groups the
evaluation team carried out.
• Reliance UNICEF for guidance – The evaluation team relied on UNICEF staff for
access to stakeholders and guidance for examples of project successes and failures.
Even with the best of intentions this type of reliance may create biases in the final
report, which the evaluation team will identify and describe.
Ethical Principles in Conducting the Evaluation ISG’s work is based on four core principles:
1. Participatory evaluation design and implementation: To the extent possible, all
people who have a stake in the outcome of a study or evaluation must have a chance
to identify risks in conducting the study and opportunities to suggest ways to reduce
those risks. Following this principal, we solicit comments and input from our clients
on inception and design documents, as well as data collection and research tools.
When clients have created terms of reference, we review the terms for input from
stakeholders and ask for a representative stakeholder review when possible.
19
2. Respect the rights, privacy and dignity of evaluation stakeholders: ISG aims to
minimize risk in evaluation management and outcomes. The primary aim of our work
is to benefit the people who are most affected by its outcomes. We put the safety,
dignity, and privacy of those that participate in our projects above the rewards that we
hope to achieve for ourselves or our firm. As such, we ensure that survey, interview,
and focus group participants are fully informed of the nature and purpose of the
research that we are conducting, obtain their consent before asking any questions or
engaging them in any other research, and allow them the opportunity to deny or
remove consent at any point in the process. We do not use names or identifying
information in reports, except in specific circumstances and then only if the
participant is fully informed and in agreement. We minimize risk to participants,
including carefully designing questions that may recreate traumatic or harmful
feelings. Finally, ISG believes that participants in our work have the right to benefit
from it. We work with our clients to produce multiple versions of documents and
materials to facilitate the distribution of results.
3. Informed and reasonable judgements: The work that ISG conducts often influences
the distribution of resources and activities in vulnerable communities. We consult
with our clients to ensure that conclusions are drawn from rigorously vetted evidence,
and that following actions are based in reliable findings. ISG’s evaluators detail the
strengths and weaknesses of our methodology and the limitations of the study given
available resources and contextual barriers.
4. Secure Data Storage Protocol: ISG utilizes secure data protocols to ensure that
respondents’ information is not used in any way beyond that which they have
provided permission. At the end of each research study, evaluation, or project that
ISG concludes, project team members submit hard copies of records, including but
not limited to research notes, photographs, or portable recordings to ISG’s designated
project manager. The project manager ensures that notes, recordings, and all other
records are moved onto ISG’s secure servers and deleted off of laptops and other
portable storage devices. ISG stores our clients’ data, as well as data collected through
primary research on secure servers through which only the project team has access. If
requested, ISG destroys all data related to a project, including paper records, or
records kept through all other means. If not request is made, ISG’s director of
programs oversees the maintenance of data on the secure server in perpetuity, or for
the time period each client or agreement mandates.
Findings This section details the evaluation team’s findings. The section is generally organized by
evaluation criteria, question, and then geographic intervention target. The sections discuss
findings by programme year, where possible. A summary of the judgement criteria,
indicators, sources of documentary information, and data collection tools used to derive each
finding is included in Annex G: The Evaluation Matrix.
Relevance Evaluation questions 1 through 3 related to the programme’s relevance.
Evaluation Question 1 • To what extent were affected people consulted on their preferences with regards to
water and sanitation service options and level of services?
• Were interventions appropriate in terms of meeting their basic needs?
20
• Was there a feedback loop and monitoring system in place for reflecting the learning
and evidence to improve programming, especially for the humanitarian response?
The emergency response period (2012 – 2013)
UNICEF and its partners were assigned responsibility for providing WASH services in the
newly opened Za’atari refugee camp in July 2012. Construction of the camp began on July
20th and the camp was opened on July 30th (Wilkes, 2012). During those ten days, the WASH
programme had to create and implement a response plan for the 10,000 refugees that would
shortly arrive, and the approximately 100,0000 more that would come over the following two
years.
UNICEF and partner organization staff that the evaluation team interviewed recalled the
urgency of operations in that time period. One staff member recalled that once he was
notified, “we had five days to respond. All we had was some contingency supplies and orders
from Copenhagen.” A UNICEF partner team member explained. “At the beginning, it was an
emergency response. We were not worrying about vulnerable populations. I don’t think there
was a lot of consultation at the time35.”
UNICEF’s determined how to best provide for refugees’ ‘WASH needs through talks with
partners such as WAJ, Mercy Corps, ACTED, Oxfam, and Relief International, and based on
the assessments and information that they’d gathered. The tactics that they employed took
into consideration the Convention of the Rights of the Child (CRC), UNICEF’s Core
Commitments to Children in Emergencies, and the Convention on the Elimination of All
Forms of Discrimination against Women (CEDAW). Through these means, UNICEF created
estimates of the best way to provide WASH services for the newly arriving refugees.
UNICEF also consulted humanitarianresponse.info for guidance on issues such as
accountability and feedback mechanisms (WaSH Sector Working Group, 2012).
UNICEF utilized institutional resources in this phase. A regional WASH Specialist was
mobilised and a Regional WASH Advisor from South Asia was sent to Amman to coordinate
the response. Also, staff reported that experts from other offices assisted the effort as did
UNICEF headquarters. The evaluation team did not find documentary evidence of support
during this phase, and it is unclear which internal resources UNICEF consulted for guidance
on WASH response in an emergency situation. For example, UNICEF may have consulted
the Global WASH Cluster, an organization that UNICEF leads, published the “WASH
Cluster Coordination Handbook” in 2009. The handbook includes specific guidance and tools
for managing WASH in emergency situations, including a schedule of activity and
methodology for conducting assessments.
UNICEF’s first concern was providing a steady flow of water and adequate sanitation
facilities. Once minimum water and sanitation standards were met, UNICEF began to focus
on providing facilities for women and people with disabilities. A UNICEF staff member who
worked in Za’atari recalled that “At the beginning, water supply was the main focus. There
was no electricity, no toilet access for night, especially for women. There were no proper
toilets for people with disabilities at the beginning. Some NGOs tried to have a chair and a
small pit for people with disabilities36.”
35 World Vision Staff Interview – 30 July 2018
36 UNICEF Staff interview – 26 July 2018
21
To determine the needs of vulnerable people, including people with disabilities, UNICEF
relied on the advice of other organizations operating in Jordan and Sphere standards. For
example, in November of 2012, Handicap International provided the WASH sector with a
document titled. “Accessible WASH Facilities to promote inclusion of persons with
disabilities, injuries, and other vulnerabilities; Za’atari Refugee Camp.” The document details
standards for communication, entrances, fixtures, toilets, and showers. Facilities that met
these criteria were eventually installed, although it is unclear from the documentation when.
Other documents from the 2012-2013 period illustrate the criteria the WASH programme
used to determine service options and levels. The Jordan RRP5 Update in August 2013 notes
that WASH services were to be designed and delivered in camps with “households living in
temporary settlements, female headed households and the needs of children, disabled, and
elderly (The United Nations, 2013).” The next priority listed is “host communities with the
largest concentration of refugees." The degree to which these priorities were addressed in a
timely fashion is unclear from the documentation. However, the referenced documentation
from 2013, the earliest to which the evaluation team had access, shows that providing water
supply and basic sanitation was the first priority, followed by ensuring that it covered
vulnerable populations.
In the emergency response phase of the programme, hygiene was poor and waterborne
diseases were common. In 2013 World Vision, a WASH programme partner, estimated that
there were 110 cases of waterborne disease for every 10,000 people; a very high rate (World
Vision Programme Cooperation Agreement 19-13 ). UNICEF and its partners would address
this issue in the next phase of the programme.
Achievements and Results (2013 – 2015)
In the emergency response phase, the WASH programme had some remarkable
achievements. Principal among them, the programme supplied water and sanitation for all
inhabitants of the camp including those that arrived newly each day. The 2015 KAP survey
for Za’atari revealed UNICEF’s responsiveness to refugees needs regarding sanitation.
During the planning phase for the waste water network, 94 percent of residents were aware of
the planned network and 96 percent of those felt it would improve life in the camp (Za'atari
Hygiene Promotion Working Grouip, 2016). A clear indication that the WASH programme
was working in concert with Za’atari’s residents. These achievements are discussed in detail
in the effectiveness section of this report.
However, the unavoidable process of quick decision making and implementation that
occurred in 2012/2013 had some consequences. Locating Za’atari over one of Jordan’s major
aquifers led to expensive measures to protect wastewater leakage. The materials of the initial
septic tanks were not adequate to protect the groundwater, and so they had to be replaced.
The WASH blocks in Za’atari were difficult to maintain because they were prone to
vandalism and theft, and unpopular with the camp residents. As a consequence, camp
residents improvised private sanitation facilities, leading to pools of greywater throughout the
camp37. Working with local partners, UNICEF developed a plan to drain the greywater and
provide a sufficient drainage solution, including desludging on a daily basis. Eventually,
37 WaSH Sector Coordination; Minutes of weekly Za’atari camp WaSH sector coordination group. Sunday 24
November 2013
22
UNICEF followed the lead of camp residents and installed household sanitation facilities for
vulnerable households that did not already have facilities, leading to another round of
removing and installing environmentally effective septic tanks. The WASH blocks would be
decommissioned. Though recycling happened where possible, it was an expensive process38.
Another consequence of the need for quick decision making was that, in the initial emergency
response phase, the camp’s effect on Jordanians living near the camp and in other host
communities with high concentrations of refugees was not taken into consideration
adequately. As an official at the MoWI said, “The more high-quality services you provide
inside the camp, the more people outside the camp will care.” As described in the context
section of this report, Mafraq, where Za’atari camp is located is a poor area and per capita
receives less water than the refugees were accustomed to in their home of origin. In addition,
Mafraq residents pay for inconsistently available water, while refugees inside Za’atari
received regular shipments of free water. In October 2012, tensions between refugees and
Jordanians living in Mafraq were erupting (Seely, 2012). This led to engagement in host
community projects, discussed in the next section.
Stability and Efficiency (2014 – 2017)
In April 2014, the Azraq refugee camp was opened. The WASH programme used a number
of lessons learned from Za’atari in designing WASH services for Azraq. For example, water
supply and sanitation facilities were less centralized and located closer to smaller clusters of
households, household clusters were composed of people that came from similar areas of
Syria, cleaning was organized by cluster with no incentives for cleanliness provided,
community mobilization and hygiene promotion was initiated as soon as refugees moved into
the camp, refugees were more involved in the day to day operation of WASH facilities, and
WASH committees were set up earlier than in Za’atari camp (World Vision Programme
Cooperation Agreement 19-13 ).
However, the public toilets and showers in Azraq remained unpopular as in Za’atari. Focus
group discussions that the evaluation team held in Azraq indicated that, as in Za’atari camp,
women resist using public facilities due to lack of privacy.39 However, In 2017, a KAP study
conducted by Action Against Hunger indicated that 100 percent of survey respondents said
that they used WASH blocks in Azraq and felt safe (ACF, 2017).
In 2014, UNICEF began seeking other means to understand needs and preferences of
refugees living inside and outside of the refugee camps, and of Jordanians living in
communities that were hosting large numbers of refugees. In January 2014, a WASH Sector
Gender Analysis was published, to guide future programme implementation. Broad
assessments were also carried out to understand targeted populations’ needs and preferences,
with a focus on the most vulnerable populations. The first such assessment, undertaken in
August 2014, was a multi-sector assessment by REACH, an international initiative to develop
information tools. This assessment covered 125 ITSs and included a household survey as
well as semi-structured ad hoc interviews and focus group discussions. The assessment was
used to demonstrate how the most vulnerable in the ITSs could be engaged in defining their
38 UNICEF Staff Interview – 26 July 2018
39 Focus Group Discussion number 1– Azraq
23
needs (REACH, 2014). In addition, UNICEF conducted a number of Comprehensive Child
Focused Assessments in Azraq and Zaatari camps in June 2015. The process was repeated in
Azraq in February 2017. These assessments were also carried out in partnership with
REACH. During this evaluation’s focus groups and interviews, residents of Za’atari and
Azraq indicated that they are asked for their suggestions and opinions during awareness
raising sessions. Residents in Azraq said that at the beginning of their residency, they
participated in the process of improving the camp40.
After the emergency response phase, UNICEF worked with its partners to adapt WASH
facilities and services to the needs and behavior of camp inhabitants. These needs and
behaviors were identified through observation and through feedback and requests from
community members. For example, in Za’atari UNICEF observed residents creating private
facilities inside their homes rather than using the public
WASH blocks. Private bathrooms were provided in some
homes, while in may people obtained equipment and
installed their own bathrooms. In response, UNICEF
adapted its intervention to provide communal tanks for
households. Each tank served up to five households. This
adjustment better suited the customs and privacy needs
of the refugees. It also responded to the realization that
the camp’s residents would live in Za’atari for an
undetermined period rather than the initially planned for
six-month period41. In Azraq, UNICEF increased the
number of water tap stands and reduced the distance
from housing units. As of July 2017, water was
distributed from multiple points within the camp (tap
stands) in two shifts, from 7am to 11 am and again 2pm to 6 pm. It was then carried to
homes via containers and then stored and used over 24 hours or more. Usually these tap
stands were provided with 4 faucets, where each faucet served 5 to 6 neighborhoods. Each
neighborhood has 12 households with an average 5 individuals. Participants in this
evaluation’s focus groups stated that the increase in number water tap stands was an
important improvement. They said that when the number of tap stations was limited, access
to water was difficult and water was scarce. The process of getting water was not well
controlled as people had to arrange themselves in queues, which made the waiting time
longer and resulted in conflicts. These issues have been reduced since the number of taps has
increased. Interview respondents said that walking home with the heavy weight was
burdensome, though not as much as previously42.
Focus groups in Azraq also indicated lack of satisfaction with sanitation facilities. The
number of toilets is limited and insufficient43. They expressed their concern regarding the
40 Azraq FDGs 1 - 4
41 Source: Interviews and on-site assessments in Za’atari camp.
42 Source: Interviews and on-site assessments in Azraq camp.
43 UNICEF notes that there is 1 male toilet and shower and 1 female toilet and shower per 12 households. The
facilities are on opposite ends of a 30-meter plot.
Figure 3 Public Latrine at Azraq camp
24
ratio of users to toilets, uncleanliness of toilets, and cultural taboos regarding women using
public showers. They also explained that the public showers are not used by women or girls.
UNICEF staff explained that in Za’atari, most households installed their own toilets. There
are fewer building and financial resources in Azraq, so households are less able to construct
their own toilets44.
Along with adopting facilities to the requirements of camp inhabitants, the WASH
programme focused on efficiency, such as reducing the cost of water provision through the
development of boreholes, pumping stations, pipelines, and a wastewater network. The
programme also sought to control the supply and standards of services, and to depend less on
contractors incase of an increase in price, a reduction of quality, or a strike.
Settlements
In Rukban and Hadalat, as in the camps, UNICEF served a population with dire needs. Time
pressure and the desperation of the Settlements’ inhabitants made consulting them about their
preferences impractical. Hygiene services for the people in the settlements were not managed
as the population rose rapidly. In January 2016 the population reached 17,000 in the
settlements, with 1,400 in Hadalat and 15,600 in Rukban. By March 2017, the population had
grown to over 77,500 in habitants (Jordan and the Berm Rukban and Hadalat 2017-2018,
2017). The WASH Programme also assisted with transitioning 28,000 people from the
settlements to Azraq camp. The urgency of the situation is detailed in UNICEF’s six-month
emergency plan for these settlements. The emergency plan notes that there were a
“disproportionate number of pregnant women” among the residents of the settlements, that
the risk of disease because of unsanitary conditions was growing, that the difficult winter
weather posed risks for children, and that there
were ongoing protection concerns for
vulnerable populations. Between January 2015
June 2016, the WASH programme
installed mobile sanitation trailers and
distributed general hygiene kits , jerrycans,
and buckets. UNICEF also provided training of
health staff and community workers. By
December of 2016, The WASH programme
was operating more stable and cost-effective
solutions to provide water and sanitation.
The evaluation team’s assessment reviewed
Rukban’s borehole, pumping stations, and
treatment system. The relevance of the water
system was demonstrated by its strong design,
needed in harsh and resource scarce areas such
as Rukban. The water system has capacity for multiple contingencies including leaks, power
outages, or effluent quality issues (figure 4). There is 800 cubic meters of storage at the
borehole site with tanks with the ability to operate independently in case of leakage. The
system appears easy to operate as well. Gama Engineering and contracting Corporation is the
44 UNICEF’s explanation was provided as a comment to the first draft of this report.
Figure 4 The stand-by power generation increases the ability
of the water supply and distribution system to function
25
operator of the system. They maintain staff on-site 24 – 7. Staff performs daily O&M
including: i) pumping from the borehole to the tanks, ii) checking the chemical
concentrations every 2-3 hours, iii) routine cleaning of the sand filter daily, and iv) repair and
replacement of parts as needed45.
UNICEF provided water and sanitation services to respond to the settlements’ exigencies.
They moved quickly and effectively, with clear goals, such as ensuring that each person had
at least 15 liters of water per day. The WASH programme also recognized that its
intervention would have an impact on the larger Ruwaished host community, so they
undertook to rehabilitate sanitation facilities there. UNICEF also implemented projects in a
hospital, schools, and vulnerable households chosen through a vulnerability assessment.
The WASH programme has maintained weekly updates since the initiation of work. The
updates monitor operations and needs. The programme’s monitoring includes regular surveys
and assessments. These tools ensure that WASH programme activities remain relevant and
effective.
Host Communities and ITSs
It is difficult to judge the degree to which the WASH programme is relevant to host
communities and ITS needs. The WASH programme’s purpose was to respond to “Water,
Sanitation, and Hygiene needs in Jordan as a result of the Syrian refugee crisis.” Eighty
percent of the WASH programme’s funds and resources have gone into the management and
construction of the refugee camps, which host 20 percent of the refugees in Jordan, while 80
percent of Syrian refugees living in Jordan are in host communities46. Many other
international organizations and agencies provide services in these communities. In the
emergency response phase of the programme, the WASH programme focused on the camps
because the programme was the only source of water sanitation for that population, whereas
refugees in host communities and ITSs presumably were availing themselves of other
sources.
UNICEF’s improvement of the water and wastewater infrastructure in host communities is
significantly important. However, the degree to which it addresses the needs generated by the
refugee crisis and the needs of the most vulnerable is uncertain. UNICEF’s work in ITSs is
guided by vulnerability maps which indicate areas of greatest need and targets vulnerable
households. UNICEF also focused on areas where government has indicated a specific need.
Even with UNICEF’s improvements, some communities continue to only get water every 7
to 9 days and just for few hours, as was the case prior to UNICEF’s involvement47. Also,
Non-Revenue Water (NRW) in many communities is higher than 50%. More relevant and
strategic initiatives may have involved promoting wide scale NRW reduction, rainwater
water harvesting, water conservation, and groundwater recharge programs as real strategic
interventions that would yield long standing results.
45 Onsite assessment and interview with Gama Engineering
46 UNICEF notes that donors earmarked much of the funds for specific infrastructure interventions.
47 Host community interviews in Irbid
26
UNICEF reports that they have served 114 ITSs, but the total number of ITSs is unknown,
and the level of that service is unclear.
Monitoring System for Learning
The WASH programme has had a monitoring system set up in the camps since the beginning
of the program. A contractor initially managed the monitoring system.. However, the
contractor’s performance was found to be unsatisfactory, so the programme decided to
directly manage the system. As a demonstration of the improvement in services, logged
points of contact increased from 7 per week at the beginning of the program to 20 per week
shortly after UNICEF took over management. At its peak, the programme managed 60 points
of contact per day48.
The evaluation team could not find evidence that the WASH programme is structured as a
learning organization. It does not appear to have taken guidance from UNICEF’s experience
in WASH programming at the outset of the programme. While it has certainly made
improvements to its activities in terms of efficiency and responsiveness to people’s needs, it
did not build monitoring networks or feedback loops that fed into a system that would allow
for learning or informing strategic management. The evaluation team found no documentary
evidence that UNICEF’s WASH programme adhered to its Monitoring, Evaluation and
Programme Management policy or annual reviews, a mid-term review in 2015, and specific
programme performance assessments (UNICEF Jordan, 2012). It also did not use its
experience or other information to create programme strategies. A 2016 report, “UNICEF
WASH Actions in Humanitarian Situations: Synthesis of Evaluations 2010 – 2016,” The
report found that there is a “need for a more consistent process of learning, disseminating,
and applying knowledge and good practices generated through experience (Hūls, 2017).”
This evaluation finds that this conclusion also applies to the UNICEF Jordan WASH
programme. This theme is further developed in the efficiency, effectiveness, and
sustainability sections of this report.
Evaluation Question 2 • To what extent did the interventions consider the needs and the priorities of the MWI
and were they coherent with global references and the regional and national response
to the Syria Crisis?
UNICEF’s work in WASH is in line with the priorities of the National Water Strategy, for
which the key reference document is the National Water Strategy 2016 – 2025. UNICEF is
focused on the sustainable operation of boreholes to supply water and create resilience.
UNICEF also recognizes the government’s concern that Jordan’s northern governorates have
not benefited from the government’s efforts to increase water supply (Ministry of Water and
Irrigation, 2016). In response, UNICEF has focused much of its host community
interventions on improving WASH in the northern governorates. Those efforts are based
UNICEF’s vulnerability mapping results.
48 Interview with UNOPS Staff – 1 August 2018
27
Both the MoWI and WAJ reported in interviews that the GoJ-UNICEF partnership is
successful. The WAJ praised UNICEF’s success at sector coordination despite budget cuts49.
The MoWI appreciated UNICEF’s role in coordinating the sector50.
Interviews at MoWI and WAJ demonstrate the Government of Jordan’s struggles with its
strategy and UNICEF’s response. Senior staff at the MoWI praised UNICEF’s
responsiveness inside the camps but sees that project’s host communities are more dependent
on available funding51. MoWI said that when the WASH programme first took on host
community projects, there was a miscommunication regarding the meaning of the protocol
UNICEF established with the MoWI. The protocol included a mandate and budget for
UNICEF to improve infrastructure in governorates. The ministry didn’t realize that the
budget was prospective and moved those items off of its budget for the year. When
fundraising fell short, those governorates were underserved52.
An interview with senior staff at MoWI indicated two concerns. The GoJ has made it clear
that the government will not take over responsibility for WASH services in the camps in any
form. The MoWI has prevented studies of potential government involvement. The MoWI
wonders what UNICEF’s plan is for sustainability or handover in the camps. Secondly, the
MoWI wondered about UNICEF’s medium to long-term role in the sector and was desirous
of closer collaboration if UNICEF were seeking to look at medium to long term
investments53.
Evaluation Question 3 • How responsive was the UNICEF WASH programme over time to changes in the
external environment?
UNICEF has responded to changes in the external environment, while maintaining standards
in its programs.
In the initial phase of the programme, the GoJ and INGOs in Jordan prepared for a temporary
camp. This preparation included procuring tents, portable sanitary facilities, and trucked in
water supplies. While these interventions were effective, their expense made them
unsustainable. When it became apparent that the need to house refugees was more than
temporary, UNICEF devised solutions that required investment, but would be cheaper and
more sustainable to operate in the medium to long-term. These interventions included WASH
Blocks, borehole construction, water network construction and operations, establishment of a
treatment plant, and established hygiene campaigns. UNICEF also made these adjustments in
the settlements, establishing water infrastructure that was both resilient enough to withstand
difficult conditions and use, but also reduced operating funding needed for maintenance54.
49 WAJ Senior Staff Interview – 30 July 2018
50 MoWI Senior Staff Interview – 30 July 2018
51 UNICEF notes that in 2018, MWI asked UNICEF inflate budgetary figures so that they include all host
community projects.
52 MoWI Senior Staff Interview – 30 July 2018.
53 Ibid.
54 UNICEF Staff Interviews – 10 October 2017
28
As senior UNICEF staff noted, “Transitioning from humanitarian response to systems and
infrastructure…is incredibly challenging, particularly in environments like the berm
(Rukban).55”
One type of transition in which the WASH programme struggled was in implementing host
community projects. As an interview respondent at the MoWI suggested, UNICEF did not
originally plan on implementing host community projects. UNICEF incorporated host
community projects into the programme when faced with external pressure from
municipalities where refugees were putting stress on local water resources56. The WASH
programme boldly took up the challenge and set high goals for itself, such as increasing the
percentage of the population in Jordan that has access to adequate sanitation facilities (2014),
or increasing the number of inhabitants with safe water and storage facilities by 1.4 million
(2015)57. While UNICEF missed these targets in each year, it continued to set high targets in
this component. As mentioned in the Funding Environment section of this report, UNICEF
blamed shortfalls in fundraising for missing host community targets in each year of the
programme. However, the evaluation team did not find evidence that the WASH programme
considered its funding outlook when planning these targets, or, given the regular shortfall in
funding, how it might better spend these funds. We discuss this topic further in the
Effectiveness and Efficiency sections.
In this evaluation’s focus groups, camp residents expressed what they felt were the most
important improvements that had taken place in WASH services, and which issues had not
yet been addressed. In Za’atari, focus group participants said that the installation of private
toilets was the most important improvement. They said that public toilets had cleanliness and
security issues, and the ad hoc pit latrines that many households dug next to their housing
units were unsanitary and dangerous. Private toilets were an enormous improvement. As far
as issues that have gone unaddressed in Za’atari, participants mentioned that they have
observed water supply and sewage disposal pipes overlapping in the same trench, and they
fear sanitation threats. They also mentioned households receiving the same quantity of water
regardless of the number of household members as an issue that requires a solution58.
In Azraq, participants said that the increase in tap stands was an important improvement.
Participants in Azraq had a number of complaints, mainly focused on lack of private
facilities, high chlorine content in water59, wasted water through misuse of some residents,
and lack of hygiene products such as detergent and soap60.
In King Abdullah Park, focus group respondents felt that improvements were particularly
made in the hygiene of the latrines and solid waste collection. They credited prompt and
frequent maintenance61.
In the current phase, the WASH programme is moving to address longer-term, nationwide
issues of sustainability. These include, climate change, depleting water resources, and
assisting communities that are becoming water scarce62.
55 UNICEF Senior Staff Interview – 6 August 2018
56 MoWI Senior Staff Interview – 30 July 2018 57 See results framework, Annex H. 58 Source: Focus Group Discussions – Za’atari Camp 59 UNICEF Jordan reports that chlorine content is within Jordanian standards. 60 Source: Focus Group Discussions – Azraq Camp 61 Source: Focus Group Discussions, KAP Camp 62 Source: Interviews with UNICEF senior staff – 6 August 2018
29
Effectiveness
Evaluation Question 4 • To what extent did UNICEF’s response achieve its intended outcomes?
• To what extent did UNICEF achieve equity results, especially for children and
women specific interventions?
The WASH programme’s achievements were tracked against different outcome statements,
output targets, and indicators in three phases; 2013/2014, 2015, and 2016/201763.
In 2012, when the programme launched, it operated without a results framework64. The
programme’s main objective was to respond to refugee needs in a quickly changing, resource
constrained context.65 In 2013 the programme drafted indicators and partially reported
achievement compared to them. Details were provided for of IR 1.3 indicators, but not IR
1.6. indicators66.
The full results framework for each year is included as Annex H. The next section provides
the programme’s objective and/or intermediate results and output statements under each
programme phase67. Following that, the report provides an overall comment on the
evaluations measure of effectiveness, and then the report presents evaluation’s findings
against indicators under each of the programme’s four components for each year.
Objective and Output Statements
2012
UNICEF and the WASH programme were operating as an emergency response programme
in 2012. In the second half of 2012, the WASH programme’s main focus was serving
refugees in the newly opened Za’atari refugee camp and at transit centers. Later in the year,
UNICEF began work in host communities, KAP and Cyber City.
The evaluation team used available documentation to assess achievements under each
component. Much of the information in this section came from Results Assessment Module
(RAM) reports, which UNICEF uses to capture achievement against indicators. The WASH
programme did not provide a workplan, grant agreement from donors, or RAM report from
2012. The programme’s documentation of its activities and achievements begins on October
4,, 2012 with the publication of UNICEF’s weekly “Syrian Refugees Response Overview
(UNICEF Jordan, 2012) .” Below are the objective and output statements for 2013 – 2017:
63 See results framework, Annex H
64 The evaluation team found no evidence of a results framework from this year.
65 Source: UNICEF, World Vision, and Mercy Corps staff interviews
66 IR 1.3 was Relevant Core Commitments on WASH are met in all humanitarian settings. IR 1.6. was Support
the Ministry of Water and Irrigation and Sector partners to address the impact of refugee influx as well as
chronic sector needs
67 UNICEF’s indicator frameworks used different language to describe result levels in 2013 and 2014.
30
2013/2014
Objective
National Institutions provide improved health and neonatal care
services and quality early childhood care with a focus on
disadvantaged groups.
Intermediate Result
1.3
Relevant core commitments on WASH are met in all
humanitarian settings.
Intermediate Result
1.6
Support the Ministry of Water and Irrigation and Sector partners
to address the impact of refugee influx as well as chronic sector
needs.
2015
Objective
Quality WASH facilities are sustained and utilised, and hygienic
behaviours practiced, by the most vulnerable while ensuring the
protection of the environment.
Output 4.1 Government and WASH sector partners plan and optimally utilize
WASH resources and services for equitable access.
Output 4.2 Children, parents and community members are aware of and
practice hygiene and water conservation.
Output 4.3 Urgent needs of vulnerable people especially children are met
through quality and lifesaving WASH services.
2016/201768
Objective
Quality WASH facilities are sustained and utilised, and hygienic
behaviours practiced, by the most vulnerable while ensuring the
protection of the environment.
Output 4.2 Institutions, utilities and organisations provide equitable access to
sustainable water and sanitation services for the most vulnerable.
Output 4.3 The most vulnerable women and children are aware of and
practice key hygiene behaviours and practice water conservation.
Output 4.4
The urgent WASH needs of the vulnerable women and children
are met through safe, predictable and sustainable water and
sanitation services and facilities.
68 The evaluation team did not have access to the 2017 RAM report. We assume targets were the same for 2016
and 2017. Results are based on UNICEF’s public reports and any internal records that the WaSH Programme
provided to the evaluators.
31
Overall comment on effectiveness
This evaluation’s ToR asks that the evaluation team judge the WASH programme’s
effectiveness, in part, by the degree to which the programme achieved its intended
outcomes69. This question is difficult to answer. First, in the emergency response phase of the
programme the intended outcome was singular, provide sufficient WASH services to the
refugees who so direly needed them70. In subsequent phases, the evaluation team found no
evidence that the WASH programme conducted strategy development or work planning
processes, which would have produced specific, measurable targets that performance could
be judged against.
The WASH programme has filed annual RAM reports since 2013. These reports include
UNICEF’s estimate of its achievements against a set of outcomes. However, the intended
outcomes are often vague in that they do not describe a means of addressing specific
population’s need that has been identified as strategically important. An example of an
intended outcome as described by RAM for 2017 is:
Outcome 4 Sustainable and equitable WASH system and key behaviours practiced.
Output 4.2 Institutions, utilities and organizations provide equitable access to
sustainable water and sanitation services for the most vulnerable.
Indicator 2 3.2 million people with access to improved municipal water services.
While those targets are certainly worthwhile, they don’t indicate the specific behaviours,
institutions, organizations, and people that the programme has identified as important to
responsiveness to the Syrian refugee crisis. Also unclear is whether the 3.2 million people is
the annual target or the cumulative programme target. These objectives do not meet the
requirements for specificity, measurability, relevance, or time-boundedness usually required
in results frameworks71.
Secondly, the WASH programme maintains limited monitoring data. Through UNOPS, the
programme monitors activity in the camps to ensure coverage and equity. The programme
collects reports from its partner organizations that include accomplishments against specific
targets defined in work plans, and budgets disaggregated by activity and result. The WASH
programme also collects water monitoring reports, and regularly completes its own reports
that include an accounting of the programme’s outputs and outcomes. However, UNICEF
does not aggregate the information in a format that would allow for its analysis. Partner
reported data is maintained in a format such as Microsoft Word, or an unsearchable .pdf
document, making it impractical for the evaluation team to extract and use the data as a
source of support for the claims made in the RAM and other reports.
69 Evaluation Terms of Reference, page 6.
70 Source: Multiple interviews with UNICEF and partner staff.
71 See, for example, The WaSH Cluster Coordination Handbook, pg 258; or UNHCR’s Practical Guide to the
Systematic Use of Standards and Indicators, pg. 57.
32
Lastly, the WASH programme’s donors do not require that UNICEF report results or
financial information disaggregated by year, achievement, activity, or other programme
category72. The lack of requirement on the part of donors provides a disincentive for UNICEF
to implement a standard financial reporting or monitoring and evaluation system.
72 Source: UNICEF Senior Staff and PMU Interviews
33
Component 1: Provision of equitable WASH services and dissemination of WASH messages in the four camps
Achievement of Intended Outcomes Overall Findings73:
Under this component, the WASH programme achieved high levels of effectiveness for the
targeted population across all years.
Details
2012 Component 1 results:
• Provision of between 1 and 1.3 million liters of
water per day to Za’atari through water
trucking.
• The completion of two boreholes to provide a
lower cost and more secure source of water for
Za’atari camp.
• The completion of 122 WASH Blocks
completed with construction of another 160 in
progress. The WASH blocks provided 662
latrines and 546 showers. The WASH units are
equitably provided for women and men, and
facilities are included for people with
disabilities.
• 151 mobile latrines and 68 mobile showers installed.
• 9,931 hygiene promotion sessions held by UNICEF’s partner organization, ACTED.
• Employment of 262 beneficiaries in a cash-for-work program in Za’atari, 38 in KAP,
and 20 in Cyber City.
• Removal of between 400 and 650 m3 per day of solid waste from Za’atari.
• Partners ACTED and THW monitor water quality arriving in Za’atari through
residual chlorine tests.
• Removal of 300 to 500 m3 per day of waste water from Za’atari.
• Maintenance of water and sanitation facilities at transit centers.
2013
In 2013 the WASH programme successfully achieved its targets under this component. 100
percent of refugees in Za’atari camp provided with drinking water. (IR1.3, Indicator 2)
• 100 percent of refugees in Za’atari provided with soap and benefited from hygiene
promotion activities. (IR 1.3, indicator 3 and Indicator 5)
• 100 percent of refugees in Za’atari have access to and use secure sanitation facilities.
• 2,340 toilets made available in Za’atari. (IR 1.3, indicator 4)
• 2,000 toilets and showers made available to serve the refugees in Azraq camp. (IR
1.3, indicator 4)
• 2,379 septic tanks installed in Azraq, or one tank for every 12 residents. (IR 1.3,
indicator 4)
73 Results in this section come from UNICEF SitRep reports, Annual Reports, and RAM Reports
Figure 5: Za'atari Camp 2012 (Source:Times of
Israel, 29 August 2012)
34
In addition to achieving its targets, the WASH programme contributed to the design and
construction of Jordan’s second largest refugee camp, Azraq. The programme incorporated
lessons learned from Za’atari and made improvements to its activities and infrastructure in
the camps that increased overall effectiveness. UNICEF made these strides under difficult
circumstances. In 2013, Jordan experienced extreme winter weather conditions, including its
heaviest rainfall in 20 years. The WASH programme struggled with wastewater removal
services provided by a contractor, and drilled a new borehole for Azraq that failed to meet
national water standards. Meanwhile, the inflow of refugees nearly doubled over the previous
year.
2013 Component 1 results not measured in UNICEF’s indicator framework include:
• Winterization of 72 WASH blocks completed by the end of March.
• Za’atari services increased to providing a daily average of 3,322,000 litres of water,
or up to 25 liters per inhabitant.
• Removing 1,250 m3 of waste water and 1,400 m3 of solid waste by the end of the
March.
• Water storage facilities/tanks are built in Za’atari, which are operational at the end of
May.
2014
In 2014 the WASH programme reported that it successfully achieved most of its targets
under this component74. It met its targets to provide water and sanitation services to residents
of the refugee camps. The WASH programme failed to achieve its targets under the
distribution of hygiene items, providing items to 107,000 of the 525,000 targeted
beneficiaries75. It is unclear what proportion of hygiene items were distributed to people in
camps versus other locations. In 2014, Azraq camp opened, a waste water treatment plant
was constructed, and new hore holes at Za’atari and Azraq were drilled.
Code Indicator Target76 Result
Output
1.3, Ind
2
Number of people provided with
safe access to sufficient water for
drinking and domestic use
(cumulative beneficiaries
reached)
100 percent
of refugees
in camps
Target met including 98,000
refugees in four camps provided
with at least 35 liters per person
per day.
Output
1.3, Ind
3
Number of persons benefiting
from improved sanitation
(cumulative beneficiaries
reached)
100 percent
of refugees
in camps
Target met including 98,000
refugees in four camps
benefiting from desludging of
communal, household and
institutional latrines
74 2013 & 2014 RAM Report
75 UNICEF reports that one reason for underdistributing items was that the market in Za’atari had become more
efficient at meeting demand than distribution activities.
76 The target figure represents the target for the population specified under each component when possible. For
example, some indicators report provision of a service for all refugees, whereas the component describes
provision to those in camps, host communities, or elsewhere. In each component the target indicator is
reported for the beneficiary group described under that component.
35
Output
1.3, Ind
4
Number of people provided with
hygiene items
100 percent
of refugees
in camps
Target met including UNICEF
reports 107,000 of the 525,000
targeted beneficiaries were
reached. It is unclear what
portion were in camps and which
were in other locations.
Output
1.3, Ind
5
Number of people involved in
hygiene promotion activities
100 percent
of refugees
in camps
Target met including 98,000
refugees in four camps reached
through direct messaging.
Output
1.3, Ind
8
Number of people provided with
access to solid waste
management (cumulative
beneficiaries reached)
100 percent
of refugees
in camps
Target met including 98,000
refugees in four camps benefited
from of 800 m3/ day on average
of solid waste removal
2015
In 2015, the WASH programme reported that it met its targets for water provision and
sanitation in the camps. A key achievement in 2015 was a general improvement in health in
Za’atari camp. In both the 2013 and 2014 KAP surveys in Za’atari, 58 percent rated their
health as good or excellent. The percentage increased to 69 percent in 2014 and 81 percent in
2015 (Za'atari Hygiene Promotion Working Grouip, 2015).
While the WASH programme reported 100 percent water coverage in Zaatari, the 2015 KAP
survey showed that only 61 percent of residents accessed water from the public tanks.
Support for the piped water network was high with 63 percent rating this project as “very
positive”.
The 2016 Azraq KAP Survey reported that 95 percent of people felt the water supply was
adequate or very adequate77. Access was good with 46 percent collecting water within 15
minutes, well within the 30-minute Sphere standard (The Sphere Project, 2011). Ninetly-six
percent accessed water for free (from public tapstands and not private vendors). Ninety
percent were happy with the water quality (ACTED, World Vision, ACT, RI, and Save the
Children, 2016).
The third indicator under this component, “proportion of population practicing hygiene
promotion (sic)”, went unmeasured by UNICEF in 2015. The programme reported that
monitors had observed increasing rates of key hygiene practices, such as handwashing with
soap. UNICEF reported that 100 percent of refugees in camps were provided with safe solid
waste management. In 2015, the quality of that management may have not been sufficient. In
2014 88 percent said their area was generally clean but this dropped to 71 percent in 2015.
No reason was given for the drop.
Accomplishments in 2015 include Za’atari’s waste water treatment plant becoming
operational, work on Za’atari’s water network starting, and Azraq’s borehole becoming
operational. Comprehensive child focused assessments were carried out in Azraq and
Za’atari, and KAP surveys were conducted. UNICEF also managed to provide services
77 The 2016 Azraq Kap survey was completed in January, representing 2015 results.
36
during a five-day truck driver strike in Za’atari and implemented activities that prevented any
further major strikes.
Note that for the output indicators in the table below, UNICEF estimated that 140,000
refugees would reside in the camps. However, the population never grew that large, so the
number of refugees reached represents a higher proportion that represented using the 140,000
figure.
Code Indicator Target Result
Outcome
4, Ind 2
Proportion of population at camps
using improved drinking water as
per Jordan standards.
100% of
refugees in
camps
100% of refugees in camps
were provided with safe water
(with the tankered quantities in
excess of the minimum standard
of 35l/p/d)78.
Outcome
4, Ind 3
Proportion of population at camps
using improved sanitation
facilities.
100% of
refugees in
camps
100% of refugees in camps used
improved sanitation facilities in
either communal WASH blocks
or their own constructed
facilities in their shelters.
Outcome
4, Ind 4
Proportion of population
practicing hygiene promotion at
camps.
100% of
refugees in
camps
No result reported for camps
Output
4.2, Ind
1
Number refugee in camps through
Hygiene messages.
140,000 100,000 reached,
Output
4.3, Ind
1
Number refugees provided with
safe access to sufficient water for
drinking and domestic use through
water trucking.
140,000 122,055 provided with 35 litres
per day
Output
4.3, Ind
2
Number of refugees provided
with safe waste water disposal
through dislodging of septic tanks.
140,000 122,055 provided with
wastewater disposal
Output
4.3, Ind
3
Number of refugees provided
with safe waste water disposal
through provision of Waste Water
Network in Zaatari Camp.
140,000 No result reported. The
wastewater network was not yet
complete.
Output
4.3, Ind
4
Number of refugees provided
with safe solid waste management
facilities.
140,000 122,055 provided with solid
waste management.
78 Statement appears exactly as written in UNICEF’s 2015 RAM report
37
Output
4.3, Ind
5
Number of refugees provided
with safe access to sanitation
facilities by operating and
maintaining the existing WASH
Blocks through WASH
Committees and/or community
involvement
140,000 122,055 provided with WASH
blocks in four camps.
Output
4.3, Ind
6
Number of refugee Boys and Girls
in camps provided with WASH in
schools
45,000 50,691 benefited from WASH
in schools.
2016
In 2016, the WASH programme reported that it met or exceeded its targets under component
1.
Definitive measurements were not available for Outcome 4, Indicator 4 related to hygiene
practices. However, a KAP study conducted in Za’atari and the output figures for hygiene
activities in settlements imply that the target was met (UNICEF Jordan, 2016).
Other WASH programme achievements in 2016 include managing the large population
increase in the Rukban and Hadalat settlements, managing the transfer of refugees to Azraq,
and continuing management of Azraq and Za’atari water networks.
Code Indicator Target Result
Outcome
4, Ind 2
Proportion of population at camps
using improved drinking water as
per Jordan standards.
100% of
refugees in
camps and
settlements
100% of refugees in camps
were provided with safe water
(with the tankered quantities in
excess of the minimum standard
of 35l/p/d).
In the camps and at the border,
244,052 people were provided
with water over the course of
2016
Outcome
4, Ind 3
Proportion of population at camps
using improved sanitation
facilities
100% of
refugees in
camps
100% of refugees in camps and
settlements used improved
sanitation facilities in either
communal WASH blocks or
their own constructed facilities
in their shelters.
38
Outcome
4, Ind 4
Proportion of population
practicing hygiene promotion at
targeted camps, host communities
and schools
100% of
refugees in
camps
Za’atari 2015 KAP indicated
between 80-96% of females
with childcare responsibility
washed their child’s hands at
key times and 92% said that
they did so using soap and
water.
Hygiene products and services
delivered to the settlements.
2017
In 2017, achievements included high levels of satisfaction in Azraq. In the 2017 Azraq KAP
survey, 68 percent of the participants replied the water was very adequate compared to 44
percent in the 2016 survey. In 2016, water storage was a concern, but residents felt it had
been resolved. 88 percent of respondents in the 2017 survey said they need less than 15 mins
to collect water compared to 46 percent in 2016. Satisfaction with water quality dropped in
2016 to 75 from 90 percent in 2016. The reported reason was poor taste, smell, and colour
(UNICEF, ACF, 2017).
In 2017, the programme completed the waste water system at Rukban. The programme also
ceased tankering, at Azraq realizing cost savings.
Code Indicator Target Result
Outcome
4, Ind 2
Proportion of population at camps
using improved drinking water as
per Jordan standards.
100% of
refugees in
camps and
settlements
100% of refugees in camps
were provided with safe water
(with the tankered quantities in
excess of the minimum standard
of 35l/p/d).
In the camps and at the border,
19.7 litres per person per day
were provided to refugees in
settlements.
Outcome
4, Ind 3
Proportion of population at camps
using improved sanitation
facilities
100% of
refugees in
camps
100% of refugees in camps and
settlements used improved
sanitation facilities in either
communal WASH blocks or
their own constructed facilities
in their shelters.
Outcome
4, Ind 4
Proportion of population
practicing hygiene promotion at
targeted camps, host communities
and schools
100% of
refugees in
camps
Za’atari 2015 KAP indicated
between 80-96% of females
with childcare responsibility
washed their child’s hands at
key times and 92% said that
39
they did so using soap and
water.
Hygiene products and services
delivered to the settlements.
Component 2: Provision of support in the host communities including WASH in schools
Overall Findings: As detailed with citations below, the WASH programme consistently
struggled to achieve targets under this component. Two issues created difficulties: funding
and lack of strategic planning
As described in the Funding Context section of this report, UNICEF Jordan struggled with
funding issues. Maintaining services in the camps necessarily received first priority for funds
at the expense of achieving targets in host communities.
That said, after several years of missing targets, programme management documentation
indicates a lack of adjustment to strategy to make the programme more effective or realistic79.
For example, from 2013 to 2017 the programme included an indicator intended to set goals
for provision of WASH services to students80. The target was missed each year of the
programme by a wide margin (table 1). Other indicators following this example include those
regarding hygiene promotion, are illustrated below.
Table 1 Students benefiting from improved WASH facilities in the learning environment, Target vs
Achievement
Year Target Achieved Percent Achieved
2013 230,633 72,985 31.65%
2014 188,000 100,000 53.19%
2015 200,000 91,757 45.88%
2016 60,000 37,693 62.82%
2017 114,000 7,980 7.00%
Additionally, the indicators aren’t sufficiently specific under this component, so it’s difficult
to know what the programme is trying to achieve. Using the above example again, in 2016,
the indicator was stated as “Number of people with access to improved WASH facilities in
institutions,” and the year’s achievement was stated as “37,693 students benefited from
provision of support to 45 schools.” It is unclear if the initial intention was that “people” in
institutions should mean students, or if the achievement was retroactively fitted to the
indicator. UNICEF’s Outcome 4, Indicator 2 target is that 1.5 million host community
79 UNICEF reports that the outcome level was fixed in 2014 for the period of 2015 – 2017. The target was
increased in 2016/2017 because of an optimistic fundraising outlook in 2015, which did not materialize.
80 The indicator was stated differently over the course of the programme: 2013/2014 – IR 1.3, Ind 6: “Number
of children (girls and boys) provided with access to safe water, sanitation and hygiene facilities in their
learning environment and in child friendly spaces.”, 2015 - Output 4.1, Ind 3, “Number of students provided
by (sic) WASH services.”, 2016/2017 (Target and baseline were general, but achievement reported in terms
of students) – “Number of people with access to improved WASH facilities in institutions.”
40
residents use improved drinking water as per Jordanian standards. In its 2015 RAM report,
the WASH programme stated that “In Host Communities, 41% (634,858) of the target
population of 1,550,500 benefitted from improved access to water facilities and systems as a
result of rehabilitation of water infrastructure, as well as support to vulnerable households.”
In its 2016 RAM report, for the same indicator the WASH programme reported, “In Host
Communities in four governorates (Mafraq, Madaba, Balqa and Irbid), 237,360 additional
people were reached in 2016. Since 2013, an estimated 1,482,402 people have improved
access to water services …which equates to 96 percent of the Country programme target.” It
is unclear if the target is that UNICEF serve 1.5 million per year under this outcome as the
2015 report implies, or 1.5 million cumulative people as the 2016 report implies. It is also
important to consider the improvement. For example, in Irbid UNICEF made improvements
to the water distribution system. This should not imply that everyone served by the system
received better water services as a result of the intervention. Water services are still very
limited, and there are still serious problems with the systems distribution network, water
safety, and availability.
Achievement of Intended outcomes
2012
2012 results:
• The first work on host community water systems, in Ramtha and Mafraq, was
completed.
• Relief International hygiene promoters complete house-to-house visits in March with
hand-washing messaging and soap distribution to 2,000 households in Mafraq and
Ramtha.
Other evidence of component effectiveness in 2012:
Za’atari opened in Mafraq governorate in July, 2012. Tensions began to rise both within the
camp and between the camp and communities outside of the camp. In response, the UN
identified serving vulnerable Jordanian’s as an important component of the Syrian crisis
response (Seely, 2012). The WASH programme expanded its scope to serve community
needs as well as camp needs.
2013
In 2013, the WASH programme met its commitments to refugees, but underachieved in terms
of the targets it set for serving Jordanians, as the results against targets below demonstrate:
41
Code Indicator Target Result
IR 1.3,
Ind 2
Number of emergency-affected
population (male and female)
provided with access to drinking
and domestic water
Approximately
215,00081 non-
camp refugees
and 180,000
Jordanians
150,000 refugees in HC’s
benefit from rehabilitation of
existing boreholes82.
Projects in host communities
benefit 400,000 people (UNICEF
Jordan, 2013)
IR 1.3,
Ind 3
Indicator 3: Number of
Emergency affected population
(male and female) provided with
access to soap and other hygiene
items
Approximately
215,000 non-
camp refugees
and 180,000
Jordanians
201,920 refugees provided with
soap and hygiene kits83.
IR 1.3,
Ind 4
Number of Emergency affected
population (male and female)
with access to appropriately
designed toilets and sanitation
services
180,000
Jordanians
No results outside of camps
reported achieved.
IR 1.3,
Ind 5
Number of population (male and
female) covered through face to
face (including HH and Group
sessions) hygiene promotion
messages
Approximately
215,000 non-
camp refugees
and 180,000
Jordanians
201,920 women, men, boys and
girls have been participating in
face to face hygiene promotion
activities including focus group
sessions and house to house
visits
IR 1.3,
Ind 6
Number of children (girls and
boys) provided with access to
safe water, sanitation and hygiene
facilities in their learning
environment and in child friendly
spaces
230,633 72,985 benefit from 91
rehabilitated WASH facilities
in schools.
WASH facilities were
rehabilitated in over 100
Jordanian schools benefiting
23,000 Jordanians and 8,000
Syrian children.84
81 Target is estimated. UNICEF described its target as “345,000 refugees (in camps and off camp) and 180,000
affected Jordanians.” The evaluation assumes the most conservative estimate of refugees in camps and
subtracts that from the total number.
82 2013 RAM Report
83 2013 RAM report.
84 Ibid.
42
IR 1.3,
Ind 7
Number of children (girls and
boys) not practicing open
defecation in host communities.
100% No measurement reported. The
programme reported, “host
communities (have) improved
including hygiene awareness
messages on risks of open
defecation.
2014
The WASH programme struggled to meet its targets under this component in 2014, as the
table below demonstrates. The exception was in the area of WASH in Schools (indicators 6
and 7). The programme significantly exceeded its target promoting hygiene messages to
students.
Code Indicator Target Result
Output
1.3,
Ind 2
Number of people provided with
safe access to sufficient water for
drinking and domestic use
(cumulative beneficiaries reached)
Approximately
255,000 non-
camp refugees
and 180,000
Jordanians
approximately 7,000 people in
vulnerable households benefited
from household water
improvements.
an estimated 450,000 people
(refugees and host community)
benefitted from increased
access to water as a result of
rehabilitation of existing water
systems.
Output
1.3,
Ind 3
Number of persons benefiting
from improved sanitation
(cumulative beneficiaries reached)
Approximately
255,000 non-
camp refugees
and 180,000
Jordanians
10,000 people benefited from
removal of sewage network
blockage.
7,000 people in vulnerable
households benefited from new
or rehabilitated facilities
Output
1.3,
Ind 4
Number of people provided with
hygiene items
Approximately
255,000 non-
camp refugees
and 180,000
Jordanians
No results reported for Non-
camp refugees or Jordanians.
Output
1.3,
Ind 5
Number of people involved in
hygiene promotion activities
Approximately
255,000 non-
camp refugees
and 180,000
Jordanians
10,000 people were reached
through mobilisation activities
on key hygiene messages
43
Output
1.3,
Ind 6
Number of children benefitting
from improved water, sanitation
facilities in their learning
environment (cumulative
beneficiaries reached)
100,000 188,000 WASH facilities in 164
schools85
Output
1.3,
Ind 7
Number of children benefitting
from improved water, sanitation
facilities in their learning
environment (cumulative
beneficiaries reached)
100,000 116,788 students reached in
host communities
Output
1.6,
Ind 3.
Percent increase of targeted host
population accessing sufficient
using appropriate sanitation
facilities
5% increased
specifically in
the Northern
Governorate
No result reported86
2015
The WASH programme underperformed against most of its targets under this component in
2015. The closest target the programme came to meeting was Output 4.1, Indicator 4,
Number of population at host communities benefiting from sewage systems improvement,
where the programme improved sewerage for about 83 percent of its target population.
Note that in the table below, student targets were set before UNICEF conducted the
Nationwide WASH in schools assessment. UNICEF reports that it anticipated a higher need
for interventions before the assessment.
Code Indicator Target Result
Outcome
4, Ind 2
Proportion of population in host
communities using improved
drinking water as per Jordan
standards.
1,550,000
host
community,
200,000
students
634,858 HC residents benefited
from improved water
infrastructure.
91,757 students benefited from
school WASH facility
rehabilitation and construction
Outcome
4, Ind 3
Proportion of population at host
communities and schools using
improved sanitation facilities
545,000 host
community,
200,000
students;
390,065 benefited from
improved access to sanitation
systems and facilities.
85 This figure, reported by UNICEF, implies 1,150 students per school, which is possible considering that large
classes sizes and double shifts common at many Jordanian schools (Middle East and North Africa Out-Of-
School Children Initiative, 2014))
86 In the 2014 RAM report, the WaSH Programme reported that 10,000 people benefit from removed blockages
in the sewage network. However, this result does not apply to this indicator, and was already reported as the
result of Output 1.3, Indicator 3 in that year.
44
91,757 students benefited from
school WASH sanitation
facility rehabilitation and
construction
Outcome
4, Ind 4
Proportion of population
practicing hygiene promotion at
targeted host communities and
schools
150,000 host
community,
200,000
students
100,000 people reached through
mobilization sessions.
73,268 students received
hygiene kids.
Output
4.1, Ind
2
Number of population benefiting
from safe water and storage
facilities improvement.
150,000
refugees and
1,400,000
residents in
HCs
619,450 people benefitted from
improvements to the municipal
water network
10,021 people benefitted from
provision of key infrastructure
items
5,387 people in Vulnerable
Households were supported
with improvements to
household wash facilities
Output
4.1, Ind
3
Number of students provided by
(sic) WASH services
200,000
students in
HC
91,757 students served by
WASH facilities
Output
4.1, Ind
4
Number of population at host
communities benefiting from
sewage systems improvement
150,000
Syrians in
HC, 395,000
in HC
331,872 people benefitted from
support to improvements to
sewerage systems.
8,255 people benefited from
mobile latrines.
52,806 people benefitted from
provision of key infrastructure
items.
Output
4.2, Ind
1
Number refugee in host
communities and students reached
through hygiene messages
150,000 at
host com
and 200,000
students
100,000 people reached through
mobilization sessions.
73,268 students received
hygiene kids.
2016
In 2016 the programme did not meet targets under this component, falling short more
drastically than in previous years. The evaluation team believes that at this point it becomes
apparent that the issue was more of a lack of strategic design process than underperformance
given programme resources and objectives.
45
Code Indicator Target Result
Outcome
4, Ind 2
Proportion of population in host
communities and schools using
improved drinking water as per
Jordan standards.
1,550,000
host
community,
200,000
students
237,360 reached
37,593 students benefitted from
improved WASH facilities
Outcome
4, Ind 3
Proportion of population in host
communities and schools using
improved sanitation facilities
545,000 host
community,
200,000
students;
35,000 people in Irbid have
improved sanitation access as a
result of the rehabilitation of the
sanitation system.
37,593 students
Outcome
4, Ind 4
Proportion of population
practicing hygiene promotion at
targeted camps, host communities
and schools
150,000 host
community,
200,000
students
No result reported
Output
4.2, Ind
1
Number of people living in
vulnerable Households with
access to improved household
WASH facilities and services
18,000 No direct result reported
Assessment carried out.
Output
4.2, Ind
2
Number of people with access to
improved municipal water
services
600,000 237,260 benefited from
improvements to water
infrastructure.
Output
4.2, Ind
3
Number of people with access to
improved municipal sewerage
services
400,000 35,000 people gained access to
improved municipal sewerage
services.
Output
4.2, Ind
4
Number of people with access to
improved WASH facilities in
institutions
60,000 37,593 students benefitted from
the provision of support to 45
schools.
Output
4.3, Ind
3
Number of people in the host
community reached through
social/community mobilisation on
key WASH messages and water
conservation.
18,000 No direct result reported.
Assessment carried out in
preparation.
Output
4.3, Ind
4
Number of people in institutions
in the host community reached
through social/community
mobilisation on key WASH
messages and water conservation
60,000 37,593 students
Output
4.3, Ind
5
Number of people in informal
settlements reached through
social/community mobilisation on
key WASH messages and water
conservation
15,000 7,298 vulnerable people in ITSs
received key WASH messages.
46
2017
It is difficult to tell how UNICEF performed against WASH targets from January – July of
2017. The difficulty arises because the WASH programme does not maintain a monitoring
database that tracks progress against indicators on a regular basis. Below we use the 2017
Annual Report, where possible, to show the WASH programme achievement over the course
of the year. Though much information is missing, UNICEF reports that the WASH
programme underperformed against targets in this component. One important
accomplishment in 2017 was that the national WASH in Schools Standards were released.
Code Indicator Target Result
Outcome
4, Ind 2
Proportion of population in host
communities and schools using
improved drinking water as per
Jordan standards.
1,550,000
host
community,
200,000
students
258,000 people in host
communities (UNICEF Jordan,
2017)
14,000 students
Outcome
4, Ind 3
Proportion of population in host
communities and schools using
improved sanitation facilities
545,000 host
community,
200,000
students;
258,000 people in host
communities (UNICEF Jordan,
2017)
14,000 students
Outcome
4, Ind 4
Proportion of population
practicing hygiene promotion at
targeted camps, host communities
and schools
150,000 host
community,
200,000
students
No measurement provided
Output
4.2, Ind
1
Number of people living in
vulnerable Households with access
to improved household WASH
facilities and services
44,000 No measurement provided87
Output
4.2, Ind
2
Number of people with access to
improved municipal water services
3.2 million 258,000
Output
4.2, Ind
3
Number of people with access to
improved municipal sewerage
services
2.1 million No measurement provided.
87 In the 2017 Annual Report, UNICEF states, “Little progress was made toward the target for sanitation. This
was due to sectoral lack of funds, as well as prioritization of water projects for the Government, due to the
enormous pressure to deliver water. UNICEF continued to work with the Government to advocate for
appropriate allocation of resources to meet sanitation needs. “
47
Output
4.2, Ind
4
Number of people with access to
improved WASH facilities in
institutions
114,000 7,98088
Output
4.3, Ind
3
Number of people in the host
community reached through
social/community mobilisation on
key WASH messages and water
conservation.
44,000 No measurement provided.
Output
4.3, Ind
4
Number of people in institutions
in the host community reached
through social/community
mobilisation on key WASH
messages and water conservation
114,000 No measurement provided.
Output
4.3, Ind
5
Number of people in informal
settlements reached through
social/community mobilisation on
key WASH messages and water
conservation
15,000 No measurement provided.
Component 3: Support to the WASH sector through technical support to the MoWI, as
well as National WASH in School Standards
Overall Findings: Government officials at the MoWI and WAJ reported satisfaction with the
WASH programme contribution to planning documents, strategies, and policies89. The MoWI
said that, as head of the WASH Taskforce, UNICEF always has a good plan, and staff that
serve together on technical committees have good partnerships with UNICEF. The Ministry
also praised the work done by the consultant that UNICEF seconded to Ministry, and wished
he could have stayed longer.
Achievement of Intended outcomes
2013
In 2013 the WASH programme had one target under this component; IR 1.6, Indicator 5:
“Increased institutional capacity in the MWI for undertaking planning and coordination
leadership role in various Sector Coordination platforms (WG/TF). Development of Sector
M&E sytem/MIS. Evidence based Advocacy to increase funding flow to the sector. The
target was the development and effective dissemination of hygiene messages. “ It is unclear
whether this target was achieved in 2013. However, the WASH programme did arrange for a
seconded staff member to work with the ministry in the following year.
88 In the 2017 Annual Report, UNICEF states, “The achievements for WaSH in Schools were also significantly
lower than planned, at 7 per cent of the target. This was primarily due to reduced funding and a focus on
smaller scale and higher impact interventions through the connection of schools to the municipal water
network, with vulnerable households connected en route.
89 Source: Interviews at MoWI and WAJ
48
2014
In 2014, WASH programme support together government included:
• Conducting a water vulnerability assessment, including data on per capita water usage
and refugee distribution.
• Ensured that the sector prioritized areas of greatest need for investment.
• Secondment of a senior advisor to lead in the development of the National Water
Strategy.
2015
The WASH programme main target under this component in 2015 was to “build strategic
planning capacity in MoWI and sector and other relevant institutions in Emergency
preparedness and response (Outcome 4, indicator 1). In 2015, the WASH programme
achieved this target by supporting the development of several strategies, regulations, and
policies. They were:
• The Jordan Response Plan (2016 – 2018)
• The National Water Strategy (2016 to 2025)
The WASH programme also developed assessments and tools to assist the government,
including:
• Water and Sanitation Vulnerability Maps, which were upgraded using internal WASH
capacity and more complex criteria.
• An assessment of the Geographic Information System (GIS) structure and capacity of
the government.
• A project monitoring database.
• A Nationwide WASH in schools assessment.
2016
In 2016, the WASH programme worked with the government to achieve the following
results:
• The National Water Strategy (2016 - 2025) was finalized, with direct support from
UNICEF, and released in April.
• The Water and Sanitation Vulnerability maps were updated in September using data
generated from the 2015 census.
• The Jordan Response Plan was updated to cover the period 2017 to 2019.
• A number of result-based tools were developed which have facilitated the generation
of systematic data, including the Comprehensive Vulnerability Assessment.
• The Water Authority of Jordan coordinates amongst the different water companies to
ensure they have sufficiently planned for emergency responses and has Long Term
Arrangements with borehole owners and suppliers which can be activated, as needed.
2017
UNICEF supported the MoWI to review and finalize the baseline values for access to safely
managed water and sanitation services under the SDGs. This included UNICEF working with
the Ministry to strengthen the decision-making framework that ensures accurate baseline
measurements. The transition from MDGs to SDGs for water entails a small reduction of 5.3
per cent. The change for sanitation is significantly higher, at 24.6 per cent. The dramatic
reduction in Jordan’s sanitation rating is expected to redress the imbalance of the sector on
49
water, and to scale-up sanitation interventions, particularly for the most vulnerable. To
increase the percentage of the population using safely managed sanitation services, key
activities were included in UNICEF’s new country programme, and will be progressively
scaled up over the programme period. In 2017, UNICEF also led the publication of National
WASH in Schools Standards.
Component 4: Coordination of the sector at camps and national level
UNICEF began leading as sector coordinator in 2014. Coordination happens at three levels;
national level, camp level, and working group level.
Achievement of Intended outcomes
Overall Findings: Stakeholders and UNICEF partners felt that UNICEF was effective as
WASH sector lead. One contractor that serves Rukban said “UNICEF staff are super
cooperative and available anytime of the day or night”.90
The evaluation team also spoke with two of the WASH programme’s donors, The Bureau of
Population, Refugees, and Migration (BPRM) at the United States State Department, and the
German owned development bank Kreditanstalt für Wiederaufbau (KfW). BPRM
commented that UNICEF was effective in leading the sector, and excellent to work with from
a donor perspective. BPRM felt that they had benefited from UNICEF’s guidance in setting
their own strategy for addressing the crisis. KfW appreciated the guidance UNICEF could
provide in constructing infrastructure in the camps and assistance in host communities.
As sector lead, UNICEF has led WASH activity coordination in the Camps, including
coordinating responses to the major storms and flooding that occur in the camps, particularly
Za’atari each year. UNICEF leads contingency planning for these storms and the emergency
reporting structure that allows for tracking storm impact. UNICEF also manages other
emergencies such as the outbreak of Hepatitis A in Azraq that occurred in 2015 and 2016.
As part of camp coordination, UNICEF leads a bi-weekly Za’atari coordination meeting that
has been effective in setting standards and avoiding duplication of effort. UNICEF also
improved the MoWI’s use of data to create water management strategies, and led the sector’s
WASH in schools initiatives.
One area in which UNICEF could improve its coordination is in strategic planning and
communication with partner organizations. Several partner organizations, particularly
INGOs, mentioned that they were surprised by UNICEF’s decisions in the camps, often
requiring that they make major adjustments on short notice, or conduct work for which they
hadn’t planned or budgeted91. Examples included sudden decisions to hand operations over to
other organizations or contractors with short notice, without guidance on the handover
process. Other partners mentioned contracting processes that took a long time only to have a
requirement that initiatives start days after a contract is signed. Decisions such as these, that
do not appear as part of a strategically designed plan, may force risk and costs on to partner
organizations. It may also create risks for UNICEF that its monitoring and accounting
systems cannot bear. These risks could also be mitigated by an annual strategic planning
90 Source: Interview with Gama Engineering
91 Interviews with Mercy Corps, World Vision, ACF, and Oxfam Staff
50
process that would set expectations and guidelines based on clear analysis and reduce last
minute decision making92.
2014
• Met output 1.3, indicator 1 target: established a WASH working group that guided the
sector response.
• Met output 1.6, indicator 1 target: Created a comprehensive WASH master plan
aligned to the National Plan (2022).
• Led the coordination of WASH activities with meetings conducted on a monthly basis
at a national level, in addition to weekly meetings at camp levels. Key achievements
of the coordination are summarised as:
- Incorporation of key WASH issues in the Regional Refugee and Resilience
Plan (3RP) and the Jordan Response Plan (JRP).
- standardisation of hygiene promotion messages and materials.
• Five Task Forces were established;
- WASH in Schools;
- Hygiene Promotion;
- Water distribution network in Zaatari;
- Wastewater in Zaatari; and,
- Development of Minimum Standards.
• Provided support to the development of the Strategic Framework to implement the
National Resilience Plan (2014 to 2016).
• Led partners in an exercise to jointly identify the WASH sector’s needs.
• Facilitated sector coordination that enabled partners to jointly identify sectoral needs.
• Provided technical assistance for a water vulnerability assessment, which identified
areas in need of support to improve water access. The assessment criteria were agreed
to with the Government, which ensured the use of the vulnerability map by the
Government to plan and prioritise areas of intervention.
2015 - 2017
During 2015 - 2017, UNICEF continued its coordination of the sector in the camps, including
the development of contingency plans for a number of scenarios and emergency possibilities.
UNICEF also coordinated the response to the Hepatitis A outbreak in Azraq and undertook
hygiene campaigns on important topics, such as effective handwashing. UNICEF also
coordinated interagency response at the Berm, coordinated task forces on Solid Waste and
Community mobilization, oversaw the transition of 28,000 refugees to Azraq camp, oversaw
the response to the May 2015 contractors strike, and coordinated the Azraq water system
redesign.
Extent that UNICEF Achieved Equity Results
The UNICEF WASH programme operates in a rapidly changing dynamic environment,
particularly in its first two years. Achieving a permanent state of equitable delivery of
services is a very difficult task. The UNICEF WASH programme continuously strove to
92 The UNICEF Jordan Country Programme Document 2013-2017 mentions a two year rolling workplan which
is measured through annual component reviews and a mid-term review in 2015. The evaluation team did not
see evidence that the WaSH Programme had complied with this mandate.
51
provide equitable results. In response to the quickly changing environment, the programme
established comprehensive feedback systems to rapidly respond to problems and complaints
as quickly as possible. Infrastructure was rapidly assessed, and when deemed appropriate,
implemented to get as near equity as possible.
Providing WASH services equitably is an enormous challenge and a moving target. Initially,
all people entering the camps were by definition the most vulnerable and UNICEF’s
challenge was to provide as many as possible a minimum acceptable level of water per day
and sanitary facilities93. As the situation stabilized, UNICEF has more resources to focus on
equitable access and focusing on women and children. In WASH sector coordination
meetings from 2013, the issues of gender, children, and equity are not discussed. The first
documentation of a focus on gender is a brief WASH Sector Gender analysis dated January
2014. The document focuses on safety for women and girls, as well as providing for their
hygiene and sanitation needs.
By March, 2014 the WASH working group began to focus on the safety of women and girls
in relation to WASH services.94 That meeting also reported on an assessment done of access
for people with disabilities.
In 2014 and 2015, UNICEF and its partners made sure equitable results were factored into
the design of its activities in the camps. ACTED recruited female Hygiene Promoters and
Social Mobilisers and Cash For Work personnel to promote equitable allocation of the
resources, incorporate women’s feedback. ACTED also tailored its hygiene promotion
activities in camps. It developed approaches that targeted young children and mothers in
Za’atari and Azraq (ACTED Programme Cooperative Agreement 23-13). UNICEF supported
vulnerable households with WASH facilities and ensured they met standards. A household
sanitation assessment was used to guide the household level improvements.
Action Contre la Faim (ACF) implemented a women-centered model for community
mobilization in Azraq called the Lead Mother Model. The model involves training a lead
mother, who is responsible for disseminating messages on hygiene, sanitation, and water for
12 households. The lead mother works with the other 12 mothers in her area to promote
hygiene, sanitation, and water conservation in their area. As of this evaluation, the initiative
had worked with 732 mothers95.
In 2016, Oxfam paid particular attention to ensuring WASH services for women, teenage
girls, and boys and girls under the age of 12. They noted that large percentages of these
groups felt threatened when seeking WASH services in the camps in previous years. They
indicated that the water and wastewater network, when completed, could solve this problem.
In the meantime, they recruited female staff and held weekly gender segregated feedback
sessions with the community. They also employed a dedicated gender advisor and worked
with UNICEF to provide protection staff.
Improvements in camp infrastructure improved results for women and children. Completion
of the water network in Azraq included a reduction in the distance from households to tap
stands, which made access less burdensome and safer. Access to private sanitary in facilities
93 Source: Interview with UNICEF Staff – 6 August 2018
94 WaSH Working Group Meeting # 3, Minutes of Meeting: March 2014
95 Source: Interview with ACF - 1 August 2018
52
has improved safety in Za’atari, though women in Azraq still struggle with using public
showers due to cultural norms around privacy and immodesty.
The WASH programme has struggled with equity in water distribution throughout the
programme and has made a continuous effort to improve equitable access. ACTED was
responsible for water delivery in Za’atari, and developed a computer based application to
monitor distribution in the camp. In 2015, ACTED highlighted that the network of tanks
around Za’atari provided an opportunity for corruption of the water system. Drivers of water
trucks were pressured to deliver water first to some tanks, leading to other tanks not getting
replenished as often as required96. Oxfam noted that water drivers incompletely filled public
water tanks, diverting water to wealthier households’ private tanks for pay. Oxfam also noted
in 2016 that wastewater drivers illegally charged households to desludge pits97. Hotlines were
established and there were follow-ups to complaints.
Both organizations hoped that the construction of the water network would solve this
problem. However, the water network also struggles with equitable delivery. In interviews
and focus groups conducted for this evaluation, residents of Za’atari noted that each housing
unit is delivered the same amount of water, regardless of the number of people that live in the
unit. Household size in Za’atari ranges from one member to more than eight members.98
Thus, smaller households receive more water per person than larger households. Also, some
residents game the system by removing floats that trigger a shut off mechanism from their
tanks or using suction to increase the flow of water to their units. Both mechanisms result in a
shortage of waters to others, further along the network. One focus group participant in
Za’atari stated that he needs to call the hot line daily and complain of water shortage so that
he can get enough water for his 30+ family members.
It is difficult to point to ways that UNICEF has promoted equitable results in host
communities. While its host community projects are important, and help municipalities with
severe water and sanitation issues, the projects tend to cover a wide area and don’t
particularly target women and children99. Improvements to household facilities did focus on
vulnerable households, but they were relatively small in number. Regarding hygiene
promotion, in the Jordan Response Plan for the Syria Crisis 2017 – 2019, the evaluation team
notes that the overall budget for Result 6.1, to Improve Hygiene Practices in Zaatari, Azraq
and KAP camps, is nearly the same as the budget for Result 3, improving hygiene in host
communities, despite 80 percent of refugees living in the host communities.100
96 Source: Interview with ACTED staff
97 Source: Interview with Oxfam staff
98 WaSH Infrastructure and Services Assessment in Za’atari; March 2017
99 Source: Site inspections of host community projects in Irbid.
100 Jordan Response Plan pg 124.
53
Evaluation Question 5 • What were the strengths and weaknesses of the interventions in achieving results in
the camps/settlements, ITSs and host communities on the following:
o Water (Litre/day, supply, quality, consistency, reliability, accessibility, and
equity)?
o Sanitation (adequacy, appropriateness, accessibility)?
o Hygiene promotion (appropriateness of the messages, timeliness, behaviour
change)?
o WASH in schools?
o National level support?
Camps
Water
In 2012, such as 67 percent of KAP survey respondents in Za’atari saying that they did not
have enough water (Za'atari Hygiene Promotion Working Groups). However, this high
percentage is likely the result of refugees moving from a relatively high-water access area in
Syria to a resource constrained environment. The shortage may also have been due to
restrictions on water delivery put in place by the Government of Jordan. By the 2013
iteration of the KAP survey, 80 percent said that they had enough water, a clear improvement
in perception as well as reality. Service levels were strained when the population numbers
rose to a peak in 2013/2014 and have since improved as population numbers declined and
steadied. An evaluation focus group participant in Za’atari commented that “before there
were complaints about the quantity of water supply. Up to 30 people would share water from
one tank101. There are fewer complaints now except from those not connected yet to the
network or those who suffer from low pressure.”
For both Za’atari and Azraq, the interventions were effective. Residents of both camps
received at least 35 liters of water per day, well in excess of the 15 liters per day that the
Sphere handbook defines as sufficient (The Sphere Project, 2011). The evaluation team
inspected water tanks outside a shelter in Za’atari and found the tanks to be. easily accessible,
directly outside the home, and water was present in the tank. Azraq’s two boreholes (2 and 3)
combined produce 120 cubic meters per hour. The water is treated, disinfected and sent to
the network. The treated water is pumped to storage/pumping structures and then to the
network tap stands. The evaluation team believes that this structure represents an appropriate
level of service given available funds. The system is a lower cost option for delivering safe
water as close to households as possible. UNICEF reports that some households use hoses
from the tap stands directly to their residence.
In Za’atari, the evaluation team’s opinion is that the installation of the water network is both
needed and appropriate. Za’atari is the size of a city and could exist for some time to come.
Water improvements such as network water connections to dwellings reduce NRW costs and
improve customer satisfaction and willingness to pay for the service. It makes water and
sanitation services more accessible and equitable to all of the camp residents.
101 One tank holds 1,000 litres.
54
Sanitation
While camp sanitation systems may not be ideal in
terms of cost efficiency, given political realities and
physical limitations, evaluation team site inspections
revealed that they are appropriate and adequate. The
exception is the wastewater management in Azraq
camp, where the treatment plant is non-operational
and septage is still being hauled vast distances ant a
huge expense.
The system in Za’atari, described below, is an
innovative approach that works well and is cheaper to
install than a traditional sewer system. The boreholes
and water systems at both Zaatari and Azraq produced high quality, and are well designed
systems are being run by high quality, experienced Jordanian contractors with the assistance
of beneficiaries working on a cash-for-work basis. These arrangements contribute to
sustainability.
In Za’atari, UNICEF initially constructed WASH blocks that were primarily used in the early
days upon refugee’s arrival to the camp. Za’atari’s residents were uncomfortable with public
facilities, which were necessary to accommodate the large numbers of arrivals. As the camp
evolved, residents progressively took responsibility for constructing their own WASH
facilities inside their shelters. Leading to over thousands of open pits throughout the camp.
Residents that the evaluation team interviewed said that, before connection to the sewerage
network they had access to septic tanks which they could have emptied upon request102.
Although, residents reported that sometimes it was difficult to get a response103.
UNICEF addressed the shortcoming in infrastructure by constructing communal tanks and
connected them, which doubled the volume of wastewater collected in the camp. The
evaluation team observed a household toilet, which was a pour flush latrine within a
household compound. The toilet was fully functioning and connected to the sewage network.
Household members stated that the toilet had functioned this way for 6 months.
Currently, UNICEF is connecting the communal household tanks to a constructed treatment
plant via the wastewater network.
An issue that came up frequently in focus groups and household interviews was the quantity
of dust that is produced by water system construction. Interview respondents and focus group
participants stated that the dust results in health issues such as triggered allergies and
respiratory infections.
For Za’atari, the evaluation team believes that the common septic tank and small diameter
sewer is an appropriate wastewater collection model. It has enabled the camp to reduce its
septage trucking from a fleet of 39 trucks to 5 trucks, which will no longer be needed once
102 Source: Interviews with ACTED staff
103 Household interview in Za’atari.
Figure 6 Very clean household latrine observed in
Za'atari camp.
55
the network is completed. The gravity flow from common septic tanks to the sewer trunk
line, and then to the lift stations where it is pumped to the treatment plant is a major
infrastructure improvement. These are both appropriate and necessary for the number of
people served. The treatment system composed of both membrane bio reactors and trickling
filters is effective. The system functions well and provides the level of treatment required by
national law, which is a key requirement. Obtaining labor for cash from camp residents also
helps reduce costs and add value for money104. It also contributes to sustainability as workers
have a sense of pride and ownership.
The biogas plant, a project implemented by the Food and Agriculture Organization, is also an
appropriate technology that should yield positive benefits to the energy balance. Mixing food
waste with septage biosolids is a recognized method of maximizing the volume of biogas
produced from anaerobic digestion.
In Azraq, the WASH programme has implemented gradual improvements of basic service. In
2015 shelters in Azraq were not connected to a greywater network, meaning greywater and
mud tends to accumulate in the ditches surrounding the shelters. UNICEF with partners
constructed a plot-level grey water network, and provided income generating opportunities
within the camp contributing to camp cleaning and sustainable waste management practices.
Communal toilets remain unpopular in Azraq. Women that the evaluation team interviewed
in Azraq reported that they did not like having to fully dress and cover their hair every time
they go to a toilet; children get dirty on their way to and from bathrooms, and pit toilets have
holes that are too big for young children. As a solution, refugees often construct bathroom
inside their kitchen space, but this leads to problems related to odors and hygiene. Also, there
is no space for drying laundry, and it is culturally inappropriate to dry underwear in a place
where it can be seen by others. Respondents also commented that the latrines for women and
men, though separate, are very close to each other and perceived as not safe105. That said, the
2017 Azraq KAP survey indicated that 100 percent or respondents said that they used the
WASH blocks and that they felt safe, compared to 96 percent in 2016.
For Azraq, the installation of the Moving Bed Biofilm Reactor (MBBR) system, which
provided by a donor turned out to be a poor choice, (although the
rationale for selecting it was at the time thought to be valid) and did
not meet the needs of the people served or the program in general.
Wastewater strength was higher than anticipated and after lots of
money spent, the system was eventually abandoned. Wastewater is
now extracted from tanks and transported also at great expense.
Until there is some centralized wastewater treatment option,
septage will require hauling some 90 KM away106.
The evaluation team believes it is surprising that the Azraq camp
would have more difficulty in establishing a wastewater system as
104 UNICEF reports that it has supported skills training initiatives in Za’atari to upgrade from low skilled to
higher skilled labor.
105 Comments from interviews and focus groups in Azraq
106 Source: Azraq site assessment
Figure 7 Rubbish Bin in Azraq
56
it is neatly laid out, compared to Za’atari, which has developed an elegant and lower cost
solution and has evolved into something that could become sustainable over the long term as
economic development in the camp increases.
Solid waste collection in the camps is adequate. The evaluation team observed rubbish bins,
which appeared substantial in terms of their volume and appropriateness. Large bins on
wheels that help minimize vectors such as flies and rats. Camp residents reported that rubbish
is collected once per day. Focus groups that the evaluation team held in Za’atari reported that
solid waste collection had improved over time.107
Hygiene Promotion
The absence of assessment data with respect to the early stages of the response made it
difficult to determine what the key public health issues were in relation to hygiene, and the
behaviours that needed to be targeted, at least in the programme’s initial stages. The first
Azraq camp KAP was conducted 21 months after the camp opened, making it difficult to
establish an initial baseline. This contrasts to the initial Za’atari KAP survey conducted only
4 months after opening. The Za’atari baseline showed considerably lower perceived health
status levels.
A general observation by the evaluation team was the lack of a consistent connection to the
health status of the population and the infectious disease risks prevailing at various times of
the year. It is acknowledged that the health sector would house such data also but the
connection to health status and infectious disease risk was not always present in the KAP
surveys. For example in the 2013 KAP survey for Za’atari, reference is made to frequency of
respiratory problems, and a suggestion is made to conduct messaging in 2014 (at least 18
months after the camp opened) on methods of preventing respiratory infections. Despite this
suggestion, the only disease related questions in the 2014 KAP survey relate to diarrhoea. A
notable exception was the Hepatitis A outbreak that afflicted Azraq.
A notable feature for Za’atari and Azraq was the strong commitment to repeating KAP
surveys in both camps on an annual basis enabling one to measure change over time.
The evaluation team did not have access to hygiene data with respect to Host Communities,
those in the ITSs, or the settlements (Hadalat and Rukban). However, the evaluation team
understands that given the circumstances it was not feasible to gather hygiene data in those
areas.
In Za’atari, the baseline KAP survey (November 2012) indicated that hygiene knowledge was
low. Only 50 percent named handwashing before eating as a diarrhoea prevention
mechanism, and 36 percent named handwashing after using the latrines. This result was
explained by UNICEF staff, who indicated that the residents of Za’atari at this time were
coming from rural areas of Syria where levels of education were low. The percentage of
people measured through observed practice though was high (an unusual result) with 55
percent of adults observed handwashing. As a point of comparison, an estimated global
average for the number of people who wash their hands with soap after going to the toilet is
just 19 percent (Freeman & al, 2014).
Over time the KAP survey results for Za’atari showed an improvement. In terms of
knowledge for example the baseline indicated that only 50% knew that handwashing before
eating helped to prevent diarrhoea which rose to 76% in the 2015 survey. In 2013 for
107 FGD 1, 5 August 2018.
57
example the KAP survey reported an improvement in handwashing, with 98 percent claiming
to wash hands with soap and water - up from 68 percent in 2012. This high level was
supported/triangulated by observed data where 95 percent of assessors reported seeing
evidence of soap in the house. This particularly high reported result should be viewed with
caution and suggests potential bias in the results, as the results are exceptionally good.
Nevertheless they do show an upward trend. (Za'atari Hygiene Promotion Working Groups).
Some direct observations by the evaluation team also demonstrated high hygiene standards in
areas such as water storage, cleanliness of toilets and general cleanliness of the household.
The exception to continuous improvement was in 2015 when there was a drop in General
Cleanliness from the 2014 results from 88 percent reporting their area was generally clean to
71 percent reporting in 2015. The KAP survey reports don’t provide much in the way of
analysis to explain results but in 2015 this might have been explained by the increase in the
camp’s population which was at its height in 2015 at a reported 120,000 (Za'atari Hygiene
Promotion Working Grouip, 2015).
In the area of hygiene supplies, interview respondents in Za’atari indicated that they did
receive hygiene items, but not enough of certain items. For example in the 2015 KAP survey
88 percent reported receiving soap in the distribution but 88 percent also reported it was not
enough. Other items commonly felt lacking in 2015 were laundry detergent and shampoo in
addition to soap. From a gender perspective only 21 percent of respondents reported
receiving sanitary pads in the 2015 survey, an indication that perhaps the contents of hygiene
kits were not gender proofed. Again the results of the KAP survey do not offer a rationale
behind such a low result.
Note that of the 88 percent who answered that soap received was not enough, 86 percent still
managed to access extra soap in the market. UNICEF indicated that the people of Za’atari
had more means to access hygiene items in the market and therefore did not need “free”
distributions of such items.
This result was supported by the evaluation team’s interviews and focus groups. During the
evaluation, respondents provided feedback on hygiene related items that indicated that
people did receive the items they needed but sometimes not enough of certain items like soap
powder and sometimes at irregular intervals. Hygiene was communicated in a number of
different ways including through community level hygiene promoters, children’s hygiene
clubs, campaign days linked to key global days such as global handwashing day and use of
the mass media. A disappointing aspect for the evaluation team was the paucity of
Information, Education, and Communication (IEC) materials provided for review108.
In Zaatari, one of the useful indicators to highlight effectiveness of the WASH programme
overall was reported general health. In the 2013 survey 58 percent answered their health was
good or excellent which rose to 81 percent by 2015.
Azraq opened in April 2014 quite some time after Zaatari. The first survey reviewed was
from January 2016, 21 months after the camp opened which is a significant time lag before
undertaking a KAP survey . The fact that the survey was undertaken in winter indicates some
results, such as satisfaction with the quantity of water supplies could be very different if
undertaken in the summer months when the demand for water is much higher. The baseline
108 The UNICEF WASH Programme provided materials used for hygiene promotion after the evaluation period.
These included flyers, posters, and other materials promoting practices including dental hygiene, water
conservation, sunstroke prevention, maintaining good health in winter, and other key practices.
58
survey in 2016 does indicate a high level of satisfaction with the water supply, with 95
percent expressing the supply was adequate or very adequate. Access was good, with 46
percent collecting water within 15 minutes. Quality also scored highly with 90 percent
responding it “satisfies” them. Results in 2017 compare favourably, with a higher percentage,
up from 44 percent in 2016 to 68 percent in 2017 satisfied. Satisfaction with quality dropped
from a high of 90 percent in 2016 to 75 percent in 2017 with the issue of bad taste and/or bad
smell cited as the reason. No explanation is given for this result.
The Border Settlements (Rukban and Hadalat)
Water
The intended results at the settlements were to
provide water to the Rukban and Hadalat
Refugee Camps as effectively as possible while
achieving value for money. The evaluation team
inspected the equipment and operations of the
water system at Rukban. The borehole and
related equipment are of high quality, the design
is appropriate, the construction is of high quality,
and the operations and maintenance activities are
performed properly and on schedule. Staff
working for the contractor that, UNICEF hired to
supply and operate the system at Rukban
reported in an interview that UNICEF always
purchases high quality equipment.
The borehole in Rukban was drilled in December
2016 with an estimated yield of 80 m3/hour. The system contains: i) the production well
(borehole), ii) chlorination, iii) sand filtration, iii) de chlorination, iv) reverse osmosis water
filtration, and v) post processing chlorination. At the time of the inspection, chlorine residual
was being maintained at between .6 and .8 mg/l, which meets or exceeds international
standards. The throughput of the system is estimated at 56 m3/hour post-treatment. This
water treatment plant has a capacity of 800 cubic meters of post-treatment water storage. In
addition to the borehole at Rukban, UNICEF has constructed two pumping stations; the old
pumping station (with two sets of storage tanks at the old station) and the new pumping
station (with a storage capacity of 760 m3). “The old pumping station is located
approximately 3km west of the Rukban borehole/reverse osmosis plant and the new pumping
station is located 10km further west of the old pumping station and is approximately 13km
west of the Rukban borehole/reverse osmosis plant. The two pumping stations pump across
No Man’s Land to water distribution systems along the Syrian border and into the camp
(Terms of Reference for Service Contracting - Rukban).” The new borehole came on line in
September 2017. It has a capacity of 85 cubic meters per hour, which is greater than the
expected 70 cubic meters per hour, and is 572 meters deep. Today the new borehole and
treatment unit are connected with the old infrastructure by pipeline as the best way to secure
the most water possible for the refugees.
This treatment system is sophisticated and costly, but warranted and necessary due to the
concentration of certain constituents of the raw water, and the fact that the system has to
work properly and consistently with no down time. Water quality of the groundwater is poor
with TDS at 3,230 us/cm and high levels of fluoride and iron. In consideration of the very
high cost of the trucking operation, full cost recovery will be achieved from the savings alone
within a matter of months. So far, the investment has saved almost 1 year of trucking, which
cost 18.5 JOD per cubic meter. At 600 cubic meters per day, that is a cost of 4 million JOD
Figure 8 Power generators and fuel storage at Rukban
water system. Extra power generation and fuel storage
capacity minimizes the potential for water system down
time.
59
per year that is not being spent on trucking. The UNICEF team reports that the system
implementation is 85% less expensive than trucking operation and delivers 30% more water.
Other important reasons for installing the water system include:
• increased volume of water resulting from no longer needing to comply with Army
restrictions;
• eased management resulting from reduced tension with the host community in
Ruwayshed;
• improved water quality and more consistent chlorination; and,
• improved consistency of water delivery resulting from removal of truck and trucker
management.
Therefore, the value for money from this investment is high.
The design of the system is strong with a capacity for multiple contingencies including leaks,
power outages, or effluent quality issues (figure 2). There is 800 cubic meters of storage at
the borehole site with tanks with the ability to operate independently in case of leakage. The
system appears easy to operate as well. The UNICEF hired contract that manages the system
maintains staff on-site 24 – 7. Staff performs daily O&M including: i) pumping from the
borehole to the tanks, ii) checking the chemical concentrations every 2-3 hours, iii) routine
cleaning of the sand filter weekly, and iv) repair and replacement of parts as needed.
The evaluation team held one focus group at Rukban. Participants claimed that water has
high levels of chlorine that is solely detected by smelling the water itself. One participant
elaborated that she believed the chlorine caused intestinal inflammation and diarrhea in two
of her family members. This could refer to enteritis (an intestinal inflammation that is usually
accompanied by diarrhea, or perhaps other medical cases that are related to gastrointestinal
diseases. All participants agreed that they were not willing to wait for the chlorine to fully
evaporate before using their water.
The team observed a chlorine test of the water coming out of the tank that goes through the
pipe that is directly connected to the one of the tap stands. The indicator showed chlorine
levels are at 0.5-0.7 mg/l.
Participants in the FGDs indicated that they also have experience multiple water cutoffs and
sometimes water was supplied 2 hours a day, sometimes more, sometimes less. They believe
that cut-offs take place when there is a high demand on the tap stands. While this may have
been the case during the earliest days of the water operations, it does not appear to be the case
now, or at least since April of 2016 when UNICEF staff created a log book for operational
issues, which indicates very minimal disruptions.
UNICEF invested close to $1 million for the water system at Hadalat but it was only
operational for 6 months. At the time when the investments were made, Hadalat was served
by trucking operations that originated at Ruwayshed. The trucking was very expensive and
unreliable with truck drivers going on strike demanding more money. The decision to
improve the borehole at Hadalat and develop the water treatment system was sound, and
actually saved 6 months of trucking expenses. It is also noted that water from Hadalat was
used to supplement the water in Rukban to avoid increasing the load on Ruwayshed.
It was impossible to tell at the time that the camp would be abandoned, so it is difficult to
criticize the decision to invest in new infrastructure. UNICEF reports that the cost for the RO
unit and the new pumps/gensets were recovered by the operation of the system, as well as
supplementing water to Rukban. The system remains operational should a viable water
supply be required for renewed trucking operations or should the camp become populated
60
again. UNICEF also reports that the system was handed over to WAJ in November 2018. The
system remains operational should a viable water supply be required for renewed trucking
operations or should the camp become populated again.
Sanitation
The evaluation team could not enter the camp to visibly inspect. However, FGDs on these
points offered the following:
• Numbers of toilets, privacy, adequacy and functionality is currently acceptable.
• Vandalism of public and communal toilets early on was an issue that helped drive the
movement towards individual toilets, one per family which is mostly the current case
• Solid waste management is adequate
• Desludging is being conducted on a regular basis. Some odors are observed but
overflows are rare.
While the FGDs were interesting, it should be noted that they were conducted under the
observation of government minders, which may have influenced the comments of the
participants. For example, while the FGD participants say they are receiving desludging
services which are just fine, UNICEF notes that in actuality, no formal desludging has been
accomplished since June of 2016.
Hygiene Promotion
Hygiene promotion in the settlements seems to be limited to the distribution of hygiene kits
on a periodic basis. Initially, the plan was to distribute once a month but focus group
discussions revealed that gaps in distributions may have lasted as long as four months. The
dynamic situation and insecure environment limits, and often prevents, UNICEF’s ability to
access the area, which explains gaps in distribution. Contents of the kits included collapsible
jerrycans, detergent, sanitary pads, solar lights, soap, toothpaste, shampoo, and diapers. In
addition, UNICEF have delivered training to 20 – 25 Community Health Volunteers for
delivery of face to face hygiene promotion in Rukban. There appeared to be a lack of
Information, Education and Communication(IEC) materials provided to the settlement
population. IEC materials could have been included in the hygiene kits distributed
periodically.
UNICEF reports that hygiene promotion efforts were carried out through ACF until the June
2016 attack, when activities were suspended. Following the attack, UNICEF distributed
flyers and soap through its clinic, as well as carrying out training to health workers. Activities
were limited and difficult to arrange due to extreme security measures imposed by the
military.
ITSs and Host Communities
It is difficult to gauge the WASH programme’s success in ITSs and host communities. The
evaluation team had no access to numbers and locations of refugees living in host
communities and the percentage of those served by the programme. It is also difficult to
know if conducting projects in host communities serves UNICEF’s WASH mission in
Jordan. Large donors like USAID and GIZ work with the MoWI to do large infrastructure
projects. UNICEF’s projects are small and general.
The projects that the WASH programme implements in host communities and ITS are
effective. For example, in Irbid, where the evaluation team observed WASH programme
initiatives, the needs of the host community are extreme regarding water and a key focus of
UNICEFs efforts. While just a fraction of the amount spent on camps is going to the host
community, interventions that UNICEF does implement are well planned and executed. In
61
Irbid, the interventions were sewer line improvement projects in city center, the rehabilitation
of two boreholes, and a pumping station improvement project. The interventions were
appropriate given the level of funding. Most of Irbid receives water every 5 to 8 days, so it is
extremely scarce. However, the water utility has a NRW of almost 50 percent. In addition,
more boreholes appear to be needed. It is assumed that the aquifer is well mapped in Irbid,
however, operators report that static water levels are down almost 24 meters in one year.
Operators don’t know if this is aquifer wide or a localized phenomenon because proper
monitoring is not conducted. Modeling the aquifer through drawdown testing would be
extremely useful. It would indicate if drilling a new borehole and connecting it to the system
would be useful.
The evaluation team visited two ITSs; camp 702 and camp 709. Life in camp 709 is harsh but
basic services of shelter, toilets, water and sanitation appear to be met. The people that live
there are extremely vulnerable and UNICEF aid is reaching them. One-hundred and twenty
people live in the camp and work for low wages in agricultural endeavors. They have one
latrine for every 20 people. There is a school with a UNICEF trained teacher on site. People
pay 3 JOD for potable water per cubic meter from a vendor, or .9 JOD per cubic meter for
piped water meant for agricultural use.
Respondents to evaluation interviews
reported that they find the water storage
volume and number of toilets lacking.
They would prefer to have one of each per
family instead of having to share with
other families. It was reported that there is
a fear that toilets and water tanks will be
stolen if too many are present. However,
the family head said he would take
personal responsibility for these if
UNICEF would just provide more. There
is no indication of waterborne disease, and
the health of the children seems good
according to the residents. When asked if UNICEF related services have been improving over
the years, the residents said yes, very much. UNICEF reports that it would like to do more in
ITSs, and is encouraged by the benefits of innovations such as the modular collapsible toilet,
which UNICEF developed. However, lack of donor interest, and earmarking of funding for
WASH in ITSs, limits their ability to make progress in the area.
Camp 702 is a cluster of 12 houses with 72 people. It is a nomadic agricultural camp similar
to camp 709. Residents move seasonally between this site and a similar site in the Jordan
Valley. Residents have refugee status, but it is still considered illegal to live in the ITS.
People fear they will sent be to the formal camps, even though they have employment and
livelihood opportunities here.
Sanitation
In the ITSs, UNICEF considered the need for on-site wastewater systems and determined that
pour flush toilets and offset pit latrines would be most effective in terms of cost and function.
This is the sanitation scheme in camps 702 and 709, where such latrines are shared by 3
families. For the most part, the pits were in poor shape, not having fly-proof or safe covers.
Venting was also absent, so while the need for sanitation was considered, at least in this
regard the implementation was poor. Upgrading latrines from single pit systems to proper
Ventilated Improved Pit (VIP) systems would be a good next step.
Figure 9 Unimproved pit latrine cover at an ITS near
Amman
62
WASH In Schools
Between December 2014 and April 2015, the WASH programme assessed 3,681 schools,
identifying needs and prioritizing schools for receiving WASH services. Of this number 400
have been reached. There were significant gaps to correct for all schools to reach the National
WIS standards, which were only finalized and released in December 2017. For example, 35
percent of schools (1,297) failed to meet the standard for student to latrine seat ratio.
It is important to note that the Ministry of Education requires all school interventions,
including new classrooms and WASH facilities, to be delivered as a package. This adds to the
challenge of delivering WASH in schools as it must be combined with non-WASH
components.
The evaluation team observed the WIS project implemented by WASH programme partner,
the Norweigan Refugee Council (NRC). The quality of the project was clear from NRC’s
report dated November 2017. The programme had a clear plan to both address infrastructure
deficiencies (WASH blocks built in 3 of the 5 schools NRC targeted), and also build capacity
(169 teachers trained on classroom activities on cleaning and hygiene. NRC produced a
Manual with guidance for schools on how to carry out the “soft” part of the WASH in
Schools programme). The evaluation team noted that NRC’s KAP survey was very high
quality and its report was objective and informative.
For WASH in schools, there is a complex regulatory environment when it comes to
connecting new WASH infrastructure to existing sewer systems, as was found out in Irbid,
where UNICEF invested in WASH blocks, only to have them still not connected to a sewer
after more than a year. The lesson learned is to secure the permits prior to beginning
construction.
Coverage with respect to WIS in schools would appear to be low at roughly 10 percent of the
total assessed (3,681). This finding should be put into context for a number of reasons.
Firstly, UNICEF are not the only agency working on WIS. Secondly, not all schools were
below standard and therefore did not require targeting. As mentioned above 35 percent did
not meet the standard for toilet seat to student ratio. Lastly, in the schools targeted water
supply was extended into the community adding to the per student cost and therefore limiting
the coverage possible with limited funding.
UNICEF did oversee several successful WIS pilots, such as the NRC project mentioned
above. However, the successful results often didn’t receive follow-up for a variety of reasons,
including the lack of suitable partners, lack of capacity within UNICEF to conduct
community mobilization109, and lack of donor support. Partners also reported that the
coordination of WASH in schools was a sub-group under the WASH Sector and only “ran
sporadically.110” NRC also said that coordination engaged WASH in Schools Implementing
Partners but not the Ministry of Education itself, who it is reported were invited but did not
attend. NRC records indicate that UNICEF did not promote or try to expand results, possibly
indicating a lack of focus on the WIS portion of the WASH programme.
UNICEF decided to carry out the WASH in Schools KAP survey in December 2016 and
repeated again in May 2017. This was too short a time frame to adequately assess changes in
knowledge and behavior.
109 Statement made in UNICEF staff interviews with those that managed WIS efforts.
110 Source: Interview with NRC Staff – 29 July 2018
63
National Level Support
UNICEF’s approach to national level support is solid at a basic level. Despite the highly
politicized nature of the WASH sector in Jordan, UNICEF has provided assistance and
guidance that the government appreciates. Examples include assisting with the development
of the National Water Strategy and then disseminating it to promote its implementation.
UNICEF also commissioned and developed the National WASH in Schools Standards in
cooperation with the government and facilitated the distribution of the standards. UNICEF
also conducted national vulnerability mapping, developed project databases, and developed
GIS hubs that will serve the Government of Jordan for many years. UNICEF’s challenge will
come in future phases as it works toward sustainable solutions for the camps that the GoJ
might resist.
Efficiency
Evaluation Questions 6 and 7 • To what extent did the programme use available resources in an economic manner to
provide WASH services and facilities in camps/settlements and host communities, to
ensure child rights, equity and gender equality?
• To what extent were the costs to deliver water and wastewater services, as well as
solid waste management, rationalized and optimized in the camps and settlements to
ensure Value for Money? How do these compare to similar situations at the local or
regional level? Could more cost effective operations/interventions have been
undertaken at an earlier stage?
The evaluation ToR asks that the evaluation team assess the WASH programme’s economic
use of resources and the extent to which costs to deliver water and services optimized.
Assessing the programme’s use of resources is difficult for several reasons. First, UNICEF
Jordan’s financial systems are organized to manage the organization’s risk, manage
compliance issues, ensure operational solvency, and protect the organization against fraud111.
Neither UNICEF Jordan nor that WASH programme track expenditures for management
purposes112. The evaluation team could not obtain documentation that demonstrated
expenditure by year, activity, programme component, or beneficiary group. Also, the
programme does not track its indirect costs or general and administrative expense rates
related to the programme, making it difficult to know the resources required to manage and
execute the programme or to compare that to other similar programmes. UNICEF Jordan
reports that UNICEF deducts 8 percent of all raised funds for HQ support and limits
international partners to 7 percent allocation to partners headquarter costs. Additionally, a
maximum of 7 percent is used to pay for Salaries of WASH staff and 5 percent is taken for
cross sectoral support costs. The evaluation team did not see year by year information on
these costs however, which would have indicated whether efficiencies were realized as the
programme progressed from year to year and as new innovations were implemented.
111 UNICEF Finance staff interviews.
112 After the evaluation period, the WASH Programme provided estimates of its efficiency gains and cost
savings. Those estimates are attached to the document as Annex I.
64
The WASH programme was able to provide some documentation regarding contract
expenditure and Programme Cooperative Agreements (PCA)113. The evaluation team was
provided information that accounted for $268,614,178114 of the approximately $355 million
that programme managed between 2012 and 2017. However, the evaluation team notes that
some significant agreements were missing from the documentation, such as PCAs with
Mercy Corps, which financed Za’atari’s boreholes among other activites, and the PCA with
the Norwegian Refugee Council, which implemented WASH in Schools projects on behalf of
UNICEF115.
Extent to which the programme used available resources in an economic manner to provide WASH services and facilities in camps/settlements and host communities, to ensure child rights, equity and gender equality
The evaluation team understands this question to ask for evidence of the extent to which the
programme used its resources in an economic manner, and the extent to which child rights,
equity, and gender equity were part of the resource use equation.
Broadly, the WASH programme’s resource allocation has five drivers in order of
importance116:
1. WASH service and facility coverage for people living in the camps and settlements.
2. Improvements that increase efficiency in coverage in camps and settlements, as well
as continuously seeking to ensure equity of service coverage and quality of services.
3. Improving water systems in host communities and ITSs.
4. Wash in Schools
5. Sector support for the MoWI.
This decision-making hierarchy developed as a consequence of the programme’s evolution,
the earmarked nature of its funding, allocations of areas of responsibility in response to its
programme management structure. Camps and settlements were allocated resources first
because, in the early days of the crisis, they were seen as having had no other options for
WASH services other than UNICEF, as opposed to refugees who could prove sponsorship
and had government or private options in host communities. After UNICEF was certain basic
needs in camps and settlements were covered, resources were invested in efficiency
improvements in the camps and settlements. Budget that was available after the effort in the
camps and settlements was allocated to host communities, ITSs and Schools117.
Expenditure and Efficiency - Camps
In July 2012, Syrian refugees had two legal options. Refugees that could provide proof of
sponsorship could live in a host community. Refugees that did not have sponsorship accepted
a tent in Za’atari. At this stage, the international community mobilized and assigned degrees
of urgency to the populations that needed assistance. The population that was under the direst
circumstances was clearly the population moving into the Za’atari camp. Believing that this
113 PCA’s are the form of agreement UNICEF uses with non-profit partner organizations.
114 This figure includes the amount that UNICEF reported on each contract, and the value of the PCAs’ budgets
that the evaluation team were provided. The evaluation team could not verify the degree to which partner
organizations had spent or exceeded the budgeted amount included in each PCA.
115 UNICEF Jordan shared these documents after the evaluation period (29 November 2018)
116 This hierarchy is evident in UNICEF Jordan Annual Reports and it its allocation of resources.
117 Source: UNICEF Staff Interview – 6 August 2018
65
population had the most urgent needs, and that the crisis would not last long, UNICEF and
other international actors in Jordan quickly mobilized to provide WASH services and
facilities however they could be obtained.
There is no indication that the cost of providing water services in the early stages of the
intervention was rationalized. UNICEF was operating in an emergency situation and cost
was secondary to providing lifesaving water, which involved expensive trucking operations,
and renting mobile sanitation facilities until more permanent arrangements could be made.
Design mistakes were made that would later prove costly, such as the WASH Blocks
constructed in Za’atari that would eventually have to be dismantled because they didn’t meet
Syrian cultural standards for cleanliness or privacy, and the location of Za’atari over one of
Jordan’s largest aquifers, which provides 37 percent of Jordan’s water.118
UNICEF took steps to bring down the unit cost of providing water to refugees in the camps.
As figure 10 below shows, the evaluation team estimates that between 2013 and 2016, the
cost of delivering water to refugees in the camps dropped from 3.34 Jod/m3 to 0.24 Jod/m3.119
Figure 11 presents UNICEF’s estimate of cost savings realized in Za’atari. The drastic drop
between 2014 and 2015 was a result of UNICEF reducing Za’atari’s dependence upon
external water trucking by over 50 percent through new storage facilities and the
operationalization of a third borehole.120 Another contributing factor was the operalization of
a borehole at Azraq camp which led to external tankering. The drop in cost between 2015 and
2016 represents phase 1 of the water network coming into operation.121
The expense and inefficiency of the
emergency phase of the response was a
natural result of the urgency, limited
capacity, limited options, and limited
resources that were available. As soon
as the camps were stabilized, the
WASH programme began taking steps
to reduce costs and improve delivery.
It also mitigated risks by reducing
vulnerability and inconsistency in
water access caused by trucker strikes.
In Zaatari and Azraq, the investments
in borehole construction and
upgrading, water treatment and
pumping equipment, and eventually
network development, were all made
with the specific goals of improving
quantity, quality, access and the
efficiency of the entire water service delivery operation, while reducing the high costs of
118 Source: UNICEF Jordan WaSH Specialist interview
119 Financial data estimated from UNICEF partner budgets included in PCAs, RAM reports, Annual Reports,
and other sources of demographic data. Costs of delivering water come from partner organization budgets
and may not accurately reflect actual expenses.
120 https://www.unicef.org/appeals/files/UNICEF_Syria_Regional_Crisis_SitRep_Sept_2015.pdf
121 UNICEF Annual Report 2016 - Jordan
Figure 10 Cost of providing a cubic meter of water by year
based on partner budgets
66
trucking. To a great extent, UNICEF accomplished this goal through these investments. The
investments were optimized by: i) utilizing high quality equipment and components, ii)
training local operators, and iii) utilizing local labor for network construction.
Figure 11 Water provision cost savings in Za'atari (Source: UNICEF WASH Programme)
There is some indication that trucking water into Za’atari camp went on longer than it should
have, draining the programme’s resources. Construction of the network began in 2014 and
was not completed until 2017. The delay was related to securing funding for the pipeline, and
a long process to approve the design of the network. However, in Azraq, UNICEF
transitioned there in there in May/June 2017 and it appeared to be very smooth with no
subsequent issues or need to supplement with external water. There, the system is run by
gravity, while simultaneously reducing the walking distance for the camp residents. UNICEF
didn’t have the funds for these improvements, but instead advocated across the sector
successfully for others to mobilize, which they did. This is an important indicator of success
of UNICEF's leadership role, not only in addressing the immediate crisis, but the broad array
of stakeholders as well.122
Sanitation (WASH facilities, solid waste management and desludging)
Initially wastewater management was inefficient. Issues such as contractors claiming to
remove up to 40 percent more sludge than they actually removed burdened the programme.
Drivers claimed trucks were capable of holding more volume than they actually did and
sometimes carrying waste in to the camp from outside, which was subsequently charged to
the UNICEF operation123. Also, on some occasions, it was learned that drivers claimed for
the official licensed volume of the trucks, but they had been modified to carry less. This
inefficiency was fixed by UNICEF introducing a comprehensive 3rd party monitoring system
whereby trucks were physically inspected entering and leaving the camp, and the dispatch
point of the trucks was controlled through the voucher system.
UNICEF developed the Waxi app to ensure that wastewater was collected on a priority basis
and to reduce the risk of full tanks (people put blankets and other materials into the network
and tanks) – this allowed the efficient and optimal collection of wastewater across the camp
and ensured a near continuous flow of wastewater into the tanks at the Za’atari Wastewater
Treatment Plant, which ultimately kept the drivers happy.
122 Source: Interviews with UNICEF senior staff.
123 Source: Interview with UNOPS
67
The wastewater system in Za’atari
evolved from unimproved pit latrines to a
system of shared septic tanks, a
wastewater collection network, and
eventually a wastewater treatment and
reuse system. The program realized
value for money by: i) utilizing a lower
cost and innovative wastewater
collection scheme, ii) utilizing cash for
work in network construction, and iii)
utilizing appropriate wastewater
treatment and reuse technologies (MBBR
and Trickling filter) which appear to be
currently working reasonably well. The
network and the internal system allowed
UNICEF to have leverage with the
desludging contractor. UNICEF also rebid services and infrastructure regularly to reduce the
ultimate cost.
In Azraq, the wastewater treatment plant is not functioning and has been abandoned. The
anecdotal evidence from stakeholder interviews indicates that this was a MBBR system
(moving bed biological reactor) that was a donation and sized based on wastewater
characteristics from Zatari, and before Azraq was even inhabited. The wastewater strength
turned out to be very strong, and also toxic, which killed the media regularly. UNICEF was
assured that the treatment facility would be appropriate and it was a donation and so they did
not pay for it (they paid for the installation and configuration) and it was a better alternative
than expensive long distance tankering. UNICEF is currently looking at local alternatives
In the area of Solid Waste Management, there were initially 180 waste pickers employed in
Za’atari. This was rationalized down to 40, who only picked waste in neighborhoods once per
week in an attempt to improve attitudes and practices with regard to waste among residents.
The reduction is staff improved efficiency124. Unicef was the driver of this, but experienced
resistance from the other NGOs, something that ultimately reduced the efficiency and
effectiveness of the activity.
Another innovation that UNICEF developed was the “Waste Taxi”, which served as an on-
demand removal system. It was a predictive model designed to address with the various
different volumes of wastewater tanks that were introduced when people began constructing
their own private washing and toilet facilities in Za’atari. This system predicted when tanks
were full or close to full and ensured tanks were emptied at just the right time thereby saving
money125.
Rukban and/or Hadalat
Water
UNICEF was paying for the water tankering, and it was causing tension with the host
community. In Rukban, and to a lesser extent in Hadalat, the issue was not just the cost, but
the consistency of the service, strikes, and water quality. Providing water to Rukban
124 Ibid.
125 Source: Intervuew with UNOPS Staff Member – 24 August 2018
Figure 12 Membrane bioreactor wastewater treatment system
functioning properly at Za'atari Camp.
68
required: i) initial emergency water hauling, ii) improvements to the water sources at
Ruyashed and Hadalat, iii) conducting the hydrogeologic study, installing the new borehole,
treatment and pumping equipment, plus a 13km pipeline and two parallel water distribution
systems. Delivering water by truck across the desert 90 KM one way was very expensive but
necessary for the function of the camp during the initial stages of the intervention.
Investments to improve the water source at Ruyashed were necessary to ensure water quality
met the applicable standards and was a necessary expenditure. The hydrogeologic study and
subsequent new source development represented value for money as it defrayed the cost of
trucking. UNICEF estimates that the solution was 85 percent less expensive than trucking.
The rationalization was that these investments directly reduced the cost of the trucking, and
were therefore appropriate, which in retrospect was sound.
Water trucking to Rukban (from Ruwaished) used to cost 380,000 JOD per month. It cost
18.5JOD as it included 15.5 JOD plus 3JOD/m3 as a tariff for the Govt, for both) per cubic
metre) and went on for 15 months. After an investment of 500,000 JOD (for borehole)
UNICEF paid for the RO / sand filtration unit, and generators, fuel and O&M costs at
Rukban, resulting in the supply cost being reduced to 100,000 JOD per month – a reduction
of 74 percent of the original cost for 30 percent additional water.126.
Costs associated with the Hadalat water system, were rationalized at the time. However, the
sudden evacuation of the camp significantly reduced the return on investment. The system
still has value as a viable but unused water source in an arid region that is short of water. The
situation at Hadalat, as well as the timing for making the new source investments at Rukban,
illustrate the difficulty of making decisions on infrastructure expenditures in a climate of
unknowns. The evaluation team believes that that UNICEF invested appropriately for water
infrastructure, was timely in their investments, and received value for money related to the
new source in Rukban and Hadalat.
Host Communities and ITSs
In the beginning of the emergency response phase of the programme, Jordanians were
supportive of efforts to make arrangements for Syrian Refugees. Tensions quickly developed
however between Jordanians that lived close to Za’atari in Mafraq governorate, and the
refugees that lived in Za’atari. These tensions developed largely over the demands that
refugees put on local resources (REACH, 2014)
The WASH programme has allocated funds for host community projects based on
availability and procured contractors or INGOs to carry out the projects through UNICEF’s
procurement system and in consultation with the government.
An example is HC projects carried out in Irbid. There was little funding available for water
improvements in the host city of Irbid. Consultations with the water utility, city government,
and ministry were conducted as well as an evaluation of the water network. Settling on
investments to improve the existing boreholes and pump station significantly improved the
efficiency of the water source network. The project represents great value for money as these
were sorely needed interventions that in the end did not cost that much compared to the
added volume and efficiency of the water system that resulted.
126 Source: Interview with UNICEF WaSH Officer
69
Evaluation Question 8 • To what extent did UNICEF try to minimize duplication/gaps in WASH interventions
in camps? And how early did UNICEF move to more cost-efficient interventions?
In this section we describe how UNICEF minimized duplication in camps. Please see the
evaluation team’s answer to Evaluation Question 7 regarding UNICEF’s evolution to cost
efficient interventions.
UNICEF minimized duplication or service gaps in camps by assigning different partner
organizations responsibility for WASH services in each sector127. UNICEF’s role as sector
lead allowed it to unify service standards among the different partners.
From July through December
of 2012, UNICEF worked
with Mercy Corps128 to
establish and manage WASH
services in Za’atari, and also
partnered with ACTED, who
conducted hygiene
promotion. In 2012, UNICEF
also began working with
Oxfam followed by ACTED
to provide WASH services
Cyber City, KAP, and “the
stadium,” three transit centers
as they were known at that
time129.
Between January and June of
2013, the refugee population in Za’atari grew from 32,000 to over 116,000 (OXFAM PCA
32-13 Executive Summary). To address the task of managing WASH services for the huge
population growth and to avoid duplication of services in the camp, UNICEF worked with
UNHCR’s division of Za’atari into twelve districts and three sectors. UNICEF partner
organizations each managed one sector and coordinated activities at weekly Za’atari
Coordination meetings. ACTED managed districts 1,2,9,10,11, and 12; JEN managed 3,4 and
5; and Oxfam managed 6, 7, and 8.
In Azraq, a similar coordination structure was followed as Za’atari. ACTED, World Vision,
and THW provided WASH services with Relief International providing hygiene promotion
and WASH in schools. However, coordination among the three entities appears to have been
difficult. There is far less documentation on coordinated efforts in Azraq. UNICEF reports
that there were weekly coordination meetings in Azraq. UNICEF also comments that partner
organizations often brought in activities funded from other donors, which added to the
challenge of coordination in Azraq.
127 Some UNICEF partners were assigned responsibility for camp-wide services, such as solid waste collection.
128 WaSH Sector Coordination meeting minutes indicate that Mercy Corps worked with UNICEF through 2016.
The evaluation team had no documentation of Mercy Corps’ involvement in the WaSH Programme.
However, the team was able to interview two Mercy Corps staff who participated in WaSH activities.
129 The source of this information is the PCA’s signed with each organization.
Figure 13 Division of labor among UNICEF Partner organizations. (Source:
WaSH in Za'atari Snapshop, Jan-Feb 2014)
70
In June 2015, UNICEF decided to simplify operations in Azraq by partnering with one
organization to handle WASH services. It asked partners to submit proposals and awarded
WASH operations to ACF. With the assistance of World Vision, ACF took over WASH
operations in October 2015. While these transitions were handled abruptly, causing some
discomfort for UNICEF’s partner organization, ACF reports that they reduced operations and
management costs bringing efficiency to the camp. Recently, UNICEF has transitioned
programme management in camps again from ACF to a private contractor. Moving from an
International Non-Governmental Organization (INGO) to a local business is a new direction
for the programme and carries some benefits, such as reduced indirect costs. The new
approach also carries some new risks, as a private contractor may not have the same
incentives to manage depreciation of equipment and infrastructure or project cost overruns as
an INGO, and UNICEF does not currently have systems capable of monitoring and
accounting for those items.
Likewise, in Zaatari, the WASH programme has reduced costs and realized efficiency by
reducing the numbers of partners that they are managing. In Za’atari, UNICEF has reduced
the number of partners to ACTED and Oxfam.
Evaluation Question 9 • To what extent were resources (financial and human resources) allocated for the range
of intervention types, with particular attention paid to the respective allocations for
humanitarian response and resilience, to achieve the intended results?
The WASH programme has four components, briefly; WASH in the camps, WASH in host
communities and schools, support to the national government, and coordination of the
humanitarian sector. During the period covered by this evaluation, the great majority of the
WASH programme’s resources went to humanitarian response as represented by WASH
management in the refugee camps, which was their direct responsibility. UNICEF’s
responsibility for financing the WASH services in the camps constrains its ability to address
more strategically important issues, such as the approximately 80 percent of refugees that live
in outside of the camps, or the high levels of NRW in Jordan.
UNICEF Staff estimated that approximately 5 percent of UNICEF’s budget goes to host
community interventions. The evaluation team’s estimate, which as mentioned is based on
incomplete information130, is that from January 2013 - -July 2017, about 63 percent of project
funds went to camp operations, about 15 percent is spent on host communities, and another
15 percent is spent in settlements, Rukban and Hadalat (figure 13).
130 The financial numbers used in this report were derived from UNICEF’s estimates of expenditure on contracts
and PCA agreements provided by UNICEF, which were often in a format that was difficult to read. PCA
numbers were budgeted amounts, not actual expenditure. These figures also do not include any indirect costs
that may not have been allocated proportionally by beneficiary group.
71
Figure 14 Percent of UNICEF Total Expenditure by Beneficiary Type (August 2012 – July 2017)
Figure 14 represents 97 percent of expenditure over the life of the programme, according the
data the evaluation team was able to access. In terms of types of intervention, and water
infrastructure construction and sanitation infrastructure construction are the largest
expenditure category (WatCon and WatProv), together describing about 47 percent of the
programme’s resources since its launch. Next is water provision (WatProv), which accounts
for another 20 percent. The remainder is Sanitation Mobilization (SanMo), Desludging, Solid
Waste Collection (SWCol), distributed kits, and funds that went to activities to support
MoWI.
Figure 15 WASH Programme expenditure by activity category (August 2012 – July 2017)
Note that following the evaluation period, the WASH Programme did provide some
additional financial information regarding the breakdown of expenditure by activity
categories. That information is summarized in Annex J.
72
As regards human resources, the evaluation team could not find documentation of how
human resources were utilized, such as organization charts or job descriptions.
Evaluation Question 10 • To what extent did UNICEF advocate with the Ministries to mobilise their available
resources to address the identified needs? And how efficient was UNICEF’s support
to ministries (i.e. MWI and MOE) to reaching most vulnerable people? Did the
WASH programme maximise the potential collaboration with other UNICEF
programme sections and partners?
Over the time frame that this evaluation covers, UNICEF worked with the MWI to develop
and update vulnerability maps, design the Jordan Response Plan, and provide financial
support to GoJ priorities. UNICEF’s engagement with the WI included a consultant UNICEF
hired to assist the government in developing strategies and response plans. UNICEF also
interacted with the government to support host communities.
The evaluation team saw no documentation of WASH programme advocacy with the MoE
during the period described by the scope of the evaluation. UNICEF reports that it proposed
the WASH in Schools Standards to the MoE and led their development, as well as
coordinating the review. UNICEF Jordan has education programmes that may have had
primary engagement with that government ministry. Still, a UNICEF partner reported that
there is more demand for education services than the MoE can handle. UNICEF could work
more to support the capacity of the MoE and should consider building those activities into its
future strategy in this area.
Sustainability
Evaluation Question 11 • Was the infrastructure in the camps constructed in such a manner to ensure long term
functionality, and will continue to service the most vulnerable people even when
UNICEF is no longer directly supporting? This should be investigated in terms of
Water and Wastewater infrastructure.
Following the emergency response phase of the programme, the infrastructure in the camps
was constructed to ensure long-term, sustained functionality. The GoJ prohibited the
construction of permanent structures at the beginning of the crisis, forcing UNICEF to make
use of mobile sanitation and trucked in water. When it became apparent that the refugee crisis
would not abate quickly, the government recognized the need for longer-term solutions and
UNICEF provided lasting, sustainable solutions.
UNICEF was provided with highly mechanized treatment systems for both Za’atari (trickling
filter and MBR) and Azraq (the MBBR system, which eventually failed). These systems are
expensive, but the decision to use these systems was sound and valid. The national laws of
Jordan specify stringent effluent quality from municipal wastewater sources. Even though
camps are surrounded by vast unused parcels of government land, wastewater systems must
comply with environmental standards. Also, the 4-month rainy season that makes it more
difficult for passive wastewater treatment systems to produce quality effluent year-round.
As written in the Efficiency section of this report, UNICEF has worked to maximize
sustainability in Za’atari with innovative wastewater collection models, elimination of
trucking, and the use of local labor. UNICEF has also installed dataloggers in boreholes to
measure water levels and monitor water level changes. UNICEF also points to the location of
73
Za’atari and that the GoJ’s involvement in Za’atari’s WASH service design will facilitate
turning the system over to the government if/when the refugees leave the camp.
For Azraq, though the MBBR system failed, the rationale for selecting it was valid.
UNICEF’s plan for sustainability in Azraq is difficult to discern. As UNICEF reports, “It is
very unsustainable to build a massive wastewater network across (an area) the size of Azraq,
in the middle of the desert, with no alternative use for it, and that the Syrian situation
suggests that peope in Az would not stay.”
For Azraq, the installation of the MBBR system, translocated from Afghanistan and provided
by, turned out to be a poor choice, although the rationale for selecting it was valid.
Unfortunately, the system did not meet the needs of Azraq’s inhabitants or the program in
general. Wastewater strength was higher than anticipated. Following an expensive attempt to
make it work, the system was eventually abandoned. Wastewater is now pumped and
transported at great expense. There is a greywater management scheme that has yet to come
on line fully, and until there is some centralized wastewater treatment option, septage will
require hauling some 90 KM away. It is surprising that the Azraq camp would have more
difficulty in establishing a wastewater system as it is neatly laid out, compared to Za’atari,
which has developed an elegant and lower cost solution, which has evolved into something
that could become sustainable over the long term as economic development in the camp
increases.
The water systems in the camps are designed, installed, operated and maintained to provide
long term functionality, even if UNICEF were to leave, which is unlikely. The quality of the
equipment is high, which adds to the longevity of the systems. The design provides
maximum operational flexibility, with multiple redundancies. The system is relatively easy
to operate, which also adds to the long-term sustainability. In Azraq, local camp residents are
performing the operations tasks with oversight provided by a local company, which is a
strong indication of sustainability. However, actual operational costs are high. The high costs
are largely due to the energy it takes to run the systems, but also the cost of consumables,
spare parts, water fee payments to the WAJ and generator costs in Azraq. UNICEF also
reports that it prefers paying more for high quality contractors, as lower capacity contractors
become much more expensive if system disruptions require a return to tankering. If money
is not available for these, the systems will not achieve long term sustainability.
The wastewater system at Za’atari is functioning well and should continue to do so given the
required operation and maintenance. Wastewater systems have design lives of 20 years or
more. This system is very robust and should provide long term and compliant service, again,
given proper O&M. The system itself, MBR and Trickling filter technology is relatively easy
to operate, which adds to its long-term sustainability. The collection system utilizing
common septic tanks for between 5 and 8 families requires a camp-wide organized
desludging program. This program is an integral component of the collection system that
will require long term funding. The use of cash for labor on the installation of the sewer
network built a sense of ownership of the infrastructure for some camp residents and also
adds to long term sustainability.
The water system at Rukban is constructed of high quality components that are well designed
and relatively easy to operate. A local engineering firm has been contracted to operate the
system and they are doing so effectively. As long as funding is available for O&M, the
system should function sustainably for many years. During KIIs of the water operators, it
was learned that UNICEF tends to purchase high quality equipment and replacement parts,
which minimizes down time and should add to long term functionality.
74
The upgrades to the boreholes and pump station in the host community of Irbid are well
designed and constructed and should continue to operate give the required O&M for many
years. While the UNICEF interventions are sustainable, the overall water situation in Irbid
however, is not. The extreme water scarcity, depleting aquifer, population growth, and high
NRW rates are all causes for concern. They suggest areas for next steps should donor
funding be available.
Evaluation Question 12 • To what extent did UNICEF take sustainability into account at the early stage for
water, wastewater and solid waste management services (e.g. transition to local
government institutions and local capacity)?
The evaluation team has no evidence that UNICEF has a transition plan to government or any
other institution in place. The GoJ at all levels has made it clear that they will not take
ownership or spend public funds on the crisis. The government understands that it is
providing an international public service in using internal resources such as water to sustain
the refugees and expects the international community to provide for the rest of their needs131.
UNICEF’s approach to sustainability has three aspects132:
• Build high quality infrastructure: Building systems and infrastructure that are
consistent with national standards and have long term potential to serve Jordanians
and the national government.
• Leverage other, non-governmental resources to contribute to ongoing operations
of the camps: This approach includes building the capacity of refugees to manage
their own WASH issues and partnering with national entities to engage in
management.
• Become as efficient as possible: UNICEF continues to find ways to bring down costs
so that resources are used as efficiently as possible, and available as long as refugees
need WASH services.
A question remains of whether running a municipal water utility, which is essentially
UNICEF’s role in the camps, is an appropriate one for the organization. Strategically,
UNICEF will have to decide whether it wishes to orient itself toward permanently providing
WASH services in the camps or should develop a strategy to hand it over to an organization
that may be better suited for those types of long-term administrative activities. A lack of
alternative organizations and guaranteeing payment for a replacement organization are
complicating factors. UNICEF reports that they investigated options for transferring solid
waste and desludging management to municipalities but lacked the resources and interest on
the part of municipalities required to make the transition.
Evaluation Question 13 • To what extent did UNICEF’s interventions increase the resilience of the Government
and respective target populations? To what extent will UNICEF-supported projects
(schools, ITSs, water and wastewater infrastructure, mobilisation) continue to be
operational after the end of UNICEF’s financial support
131 Source: Interviews at the MoWI, WAJ, and UNICEF.
132 Source: Interviews with UNICEF executive staff.
75
This evaluation question is difficult to answer because it appears that target populations will
require UNICEF’s financial support well into the future. UNICEF increased the resilience of
supported populations and the GoJ in the following ways.
• Designed projects in accordance with the JRP, linking refugee assistance in Za’atari
and Azraq with resilience-oriented projects that rehabilitate or extend systems in
urban areas with large refugee populations.
• Highlighting need through vulnerability maps and mobilizing for funds for resilience
projects described in the JRP to assist the GoJ.
• Instituted consistent standards and policies around water access in camps to promote
equitable access and reduce conflict.
• Worked with the MWI to design the National Water Strategy.
• Ensured that infrastructure was built using high quality materials and that construction
was carried out according to national standards.
• Built capacity of stakeholders in camps to manage water, sanitation and hygiene
within their communities.
• Created mechanisms for hygiene and water conservation promotion that operate under
the leadership of local stakeholders, such as WASH committees and the Lead Mother
model in Azraq.
• Improved water and sanitation infrastructure and services in some host communities
with high refugee populations to reduce tensions.
• Worked with the government to create Wash in School Standards to set a bar for
water, hygiene and sanitation in Jordan’s often overburdened schools. The standards
included a nation-wide assessment.
• Raised awareness regarding water conservation and promoted a leakage detection
app.
• Transferred a process for creating response and water strategies to the GoJ that they
can replicate in the future.
• Promoted the use of data for strategy development at the MoWI.
Evaluation Question 14 • Does the MWI have the necessary mechanisms in place and capacity to ensure the
water and sanitation needs of the most vulnerable people will be met as a result of
UNICEF’s support and advocacy, if another large scale humanitarian crisis occurs in
the future?
The evaluation team did not find evidence that MWI has the necessary mechanisms in place
for a future, similar crisis. One reason for this is that the government does not want to own
these crises or been seen as a refuge for at-risk populations. Additionally, UNICEF reports
that public support has dropped considerably compared to the situation in 2012.
Evaluation Question 15 • To what extent did UNICEF document its WASH interventions and make available to
the sector learning on water, wastewater, solid waste management and mobilisation?
Documentation of interventions for the purpose of sector learning was limited. Lessons
learned were shared at coordination meetings. The evaluation team did not find evidence that
the WASH programme documented its experience for the purpose of sector or institutional
learning. This evaluation is the first attempt to document UNICEF’s experience
implementing the WASH programme in Jordan. UNICEF reports that it created a new
position in January 2018 to document sectors learning and the position is working well.
76
Evaluation Question 16 • To what extent were environmental protection and sustainability taken into account in
the design and implementation of programmatic interventions?
Early in the programme, environmental protection was not taken into account sufficiently.
Lack of focus on environmental issues led to Za’atari’s construction over an aquifer that
accounts for 34 percent of Jordan’s water supply without proper facilities for managing
wastewater and preventing it from leaching into the ground. As described in earlier sections
of this report, the problem was exacerbated when open pits filled with graywater developed
as camp residents began to create private sanitation facilities in their housing units. UNICEF
corrected this issue in Za’atari by installing wastewater collection tanks, reaching 2,000 m3133
of collected wastewater per day, utilizing backwash from borehole water filters for
construction and cleaning purposes, and constructing a water treatment plant. UNICEF also
backfilled over 11,000 open pits. Water usage was controlled through UNICEF’s
enforcement of regulations and daily volume collection and distribution. Seasonal water
allocations are used to ensure sufficient levels, and in Azraq, the community has been
engaged to prevent wastage.
UNICEF applied the lesson it learned in opening a site location and water system for Azraq.
Prior to the construction of Azraq, UNICEF worked with the MoWI to conduct an
environmental impact assessment. The assessment identified a high risk of underground
water contamination. Also, the GoJ was concerned about increased abstraction from the well
field near Azraq. UNICEF incorporated protections in its design of Azraq’s WASH services.
It supported installation and operation of containerized wastewater units. As seen earlier in
this report, the treatment plant ultimately did not work out, but the evaluation team
recognizes that the decision to include it was appropriate at the time. The septage receival
and dewatering units would have permitted the system to receive wastewater from trucks and
bring down the cost of desludging operations. In 2016, a pond was also constructed at Azraq
camp to store wastewater that could be used for agriculture.
UNICEF also assisted the government in improving environmental protections as regards
WASH services. For example, UNICEF built the capacity of the MoWI to conduct
groundwater monitoring more effectively.
Other initiatives that UNICEF implemented that took environmental protection and
sustainability into account include:
• Working with the Royal Scientific Society to study the most environmentally
sensitive way to provide heating for camps. Providing heating with gas was chosen.
• Promoting water conservation in camps and host communities through
implementation partners.
• Celebrating world events, such as World Environment Day.
• Promoting recycling where feasible.
• Leading the construction of household connections in Azraq to reduce potential
contamination.
133 UNICEF reports this represents a doubling of wastewater volume collected.
77
Coverage
Evaluation Question 17 • Did UNICEF’s WASH response identify and reach the most vulnerable people? To
what extent did specific interventions strike an equitable balance between
interventions in camps/settlements and in host communities, to reach the most
vulnerable people?
Identification and outreach to the most vulnerable people
It is difficult to know if the response identified and reached the most vulnerable people over
the course of 2012 – 2017. At the initiation of the project it was assumed that refugees
housed in Za’atari camp were the most vulnerable, as refugees taking up residence elsewhere
had other options for WASH services, such as private sources or the government.134 The
evaluation team did not find documentary evidence of an attempt to identify the most
vulnerable people and reach out to them following the emergency response phase of the
project through July 2017. For example, while UNICEF has served inhabitants of some ITSs,
there is no comprehensive understanding of the number of ITSs or their population.
That said, UNICEF described its efforts to the evaluation team. These included:
• Sanitation assessments in Za’atari to identify households that needed the most
assistance. These households were supported by partner organizations.
• Identification of households that included people with a disability, and a subsequent
construction of ramps to meet their needs.
• Vulnerability assessments conducted in Ruwayshed followed by a prioritization of the
households which met vulnerability criteria for assistance.
• Identification of households in extreme in partnership with ACF, and conducting
interventions for these households.
• Continued work with especially vulnerable schools, ITSs, and host communities.
Following the emergency response phase, the WASH programme continued to devote most
of its resources to camps throughout the life of the programme. The programme did not
methodologically study the number and status of refugees outside the camps, and the
proportion of programmatic resources that went to people outside of the camps was small.
Though approximately 80 percent of the Syrian refugees in Jordan live outside of the camps,
only between 5 and 15 percent of the WASH programme budget went to host communities.
Finding resources to serve host communities more equitably would have meant reducing
standards in the camps.
Within the camps, UNICEF’s WASH programme has had a focus on ensuring access for the
most vulnerable. This includes, women, children, and people with disabilities. Please see the
section titled “Extent that UNICEF Achieved Equity Results” under evaluation question 4 of
this report.
Equitable balance between interventions in camps and host communities
The degree to which the programme is focused on the most vulnerable people is influenced
by the WASH programme first mandate, which is to maintain WASH service levels and
standards in the camps. While people entering the camps may have been the most vulnerable
134 Source: Interviews with UNICEF senior staff.
78
at the programme’s outset, the evaluation team believes that in later years occupants of ITSs
were more vulnerable and received less attention from the WASH programme. However, the
necessity of maintaining service in the camps combined with budgetary shortfalls has
prevented UNICEF from identifying and reaching that population.
Until July 2012, UNICEF Jordan did not have a WASH programme. It had WASH activities
within its health unit, but not a stand-alone programme. When the Syrian refugee crisis
began, the Jordanian offices of UNHCR and UNICEF came to an agreement that UNICEF
would provide WASH services in the camps, and that UNICEF would lead the WASH sector.
Thus UNICEF became tied to managing WASH in the camps as a first priority. As executive
staff at UNICEF Jordan put it “The camps had to be a starting point for funding allocation
because there was no other option for those people. 80,000 in Za’atari had no other way to
survive. Others had government and other options. The starting point is how do we maintain
services in the camps.”
As the number of refugees quickly increased, tensions between host communities and refugee
populations grew. Three months after Za’atari opened, Mercy Corps conducted a study to
document the growing tensions between Jordanians and the refugees. UNICEF began
allocating some resources to projects in the host communities, particularly Mafraq and
Ramtha, but its focused remained, appropriately, on the camps, which provided resources to
people with no other alternative. In the 2012 UNICEF Jordan Annual Report, UNICEF
Acknowledged, “here has been an additional challenge with the identification of the most
vulnerable children within the host communities,”
That challenge continued through 2013 and 2014 as the population of the camps continued to
grow. UNICEF implemented projects in host communities as well, but they were for the
purpose of quelling tension rather than serving the most vulnerable. REACH’s June 2014
report, “Access to Water and Tensions in Jordanian Communities Hosting Syrian Refugees,”
illustrates why UNICEF chose the projects it did in host communities, and why reducing
tensions was a worthwhile goal. A host community project in Irbid serves as a good example.
In Irbid, the WASH programme improved sewer lines and rehabilitated two boreholes. The
improvements were hugely important in Irbid, a governorate with a high refugee population
and degraded water infrastructure. The evaluation team’s inspection found that the
interventions added volume and efficiency to the water system.
As early as 2014 the WASH programme began to target indicators improving water access
for Jordan as a whole in its indicator framework. By 2016, though it was clear that the large
majority of refugees were outside the camps and that those living in ITSs were existing under
dire circumstances, still only 4 percent135 of the WASH programme’s budget went to host
communities.
Serving the most vulnerable in the camps
Within the context of the camps, the WASH programme has been conscientious of serving
vulnerable inhabitants. This evaluation report details initiatives to serve vulnerable
population in the camps in response to Evaluation Question 4.
135 Based on incomplete information submitted to the evaluation team.
79
Evaluation Question 18 • Did UNICEF as Sector Lead ensure that the needs of those in most need were
addressed in a coherent manner, particularly in comparison to alternative existing
systems at the local or regional level (i.e. UNRWA camps)? If not, how?
As described elsewhere in this report, the evaluation team found that UNICEF ensured the
needs of those in camps and at the settlements were addressed in a coherent manner.
UNICEF organized sector coordination efforts and coordination efforts at the camp and
settlement levels to design activities, assign responsibilities to avoid duplication, and set
standards to ensure equity of results as far as that was possible. Following that, UNICEF
worked in host communities and ITSs to the extent resources allowed. The evaluation team
does not have evidence that UNICEF used its position as sector lead to make sure those in
need were identified and that their needs were addressed. UNICEF reports that it shared
vulnerability mapping widely across the sector and encouraged partners to raise funds to
serve vulnerable people.
Coordination
Evaluation Question 19 • To what extent did UNICEF role as WASH sector lead enhance response processes
and results to avoid duplication and gaps in interventions?
Please see the evaluation’s response to question evaluation question 8 regarding avoiding
duplication.
Regarding enhancing response processes, a sample of WASH sector meetings from May
2016 through April 2018 show 13 organizations regularly attending. Action points and focal
points are thoroughly covered, and cover issues from the National Water Strategy to Schools
Assessments and issues in the camps.
UNICEF also established a number of reporting mechanisms through third party monitors
including water quality reporting, and security incident reporting. UNICEF established the
use of vulnerability Maps for Water and wastewater. The maps revealed Azraq governorate
as the most vulnerable place in both water and sewage. UNICEF has also worked with
partners to submit proposals for sources of funding to address specific issues.
UNICEF has also put tremendous effort toward preparation for winter storms including
stockpiling supplies, coordinating partner roles, and negotiating contingency plans. UNICEF
also provides ongoing monitoring during storms as well as clear communication with
partners.
UNICEF partners that the evaluation team interviewed revealed partner satisfaction with
UNICEF as sector lead. One partner said that, “UNICEF are very good coordinators. They
always have staff in Azraq.” A donor said that they were very happy with UNICEF’s
coordination and partnership. A staff member of one partner said, “UNICEF does a very
good job of controlling the work. 95 percent approval rating for UNICEF.”
80
Evaluation Question 20 • Were partnerships effective and leveraged to the maximum extent to assist the most
vulnerable population in camps/settlements, ITSs and in host communities?
The evaluation team believes that partnerships were effective and leveraged to the maximum
extent in the camps and settlements. As described in this report, UNICEF utilized its partners
to ensure full coverage in the camps and compliance with standards. Based on the limited
financial information that the evaluation team was provided, the team estimates that UNICEF
also leverages over $27, 250,000 in in-kind contributions from the partners it worked with136.
UNICEF also effectively worked with partners in host communities. Partners were relied on
to identify and manage key interventions. UNICEF had clear communication with partners to
identify ITS and school needs and to mobilize appropriate funds. WASH in schools programs
were also managed effectively through partners.
One aspect of working with partners that UNICEF could have improved was communicating
strategic decisions and medium-term vision for the project. The UNICEF partners that the
evaluation team interviewed reported that they often heard about drastic changes to plans or
operations with little notice or time to prepare. One partner noted “UNICEF take quick
decisions – the change from Mercy Corps to ACF in Azraq could have been done in a better
way. It was not a transparent process.”
Conclusions In July 2012, UNICEF was the only organization in Jordan that had the resources, capacity,
and institutional commitment to take leadership of the WASH response related to the Syrian
refugee crisis. UNICEF accepted responsibility for providing WASH services in a country
that was exhausted by previous refugee crises, water scarcity, a complex political
environment, and funding uncertainty. From July 2012 through July 2017, UNICEF provided
life-saving water and sanitation resources under these difficult conditions for the
approximately 400,000 people that benefited from WASH services137.
The programme that UNICEF developed as its WASH response to the crisis was to become
one of the largest programmes in UNICEF Jordan’s portfolio. The scope of the programme
was initially a mandate to provide services to refugees in camps. It quickly expanded to
include services for the water-deprived Jordanian host communities. These communities were
resource constrained before the crisis and were further burdened by the crisis. The WASH
programme’s scope also grew to include providing urgent WASH services for refugees living
in the border settlements at Hadalat and Rukban. The emergency response required quick and
flexible decision making and efficient utilization of scarce resources, which UNICEF
successfully performed.
The quick acceptance of responsibility in July 2012 and rapid action that followed led to
several missteps. The evaluation team could not find evidence that UNICEF WASH staff
utilized UNICEF’s internal resources that provide guidance in managing WASH during the
first phase of an emergency response. For example, UNICEF has led the Global WASH
Cluster for over a decade. In 2009 the organization published a handbook that described how
136 Estimate based on a review of PCA’s during the life of the WaSH Programme
137 Evaluation Terms of Reference, page 4.
81
to WASH services in an emergency like the Syrian Refugee Crisis (redrUK, 2009).
Utilization of the handbook might have assisted the WASH programme team in contributing
to the decision regarding the location of Za’atari camp, assessing refugee needs and
preferences, conducting ongoing monitoring, and allocating responsibilities to various
partners were made.
The UNICEF WASH programme was a conscientious steward of its donors’ contributions. In
the emergency response phase of the programme, UNICEF staff worked under difficult
conditions to bring WASH services to refugees on short notice. After the situation stabilized,
UNICEF was innovative in its analysis of WASH operations. This analysis led to consistent
reductions in operating costs, which allowed financial resources to serve many more people
at a higher standard than was otherwise possible. Examples include committing to build new
infrastructure that could be operated at a lower cost than emergency interventions, and the
“waste taxi” predictive model, which allowed WASH managers to anticipate desludging
needs and overflowing tanks.
Regarding hygiene, the evaluation notes an absence of reference to key infectious diseases
such as diarrhea and respiratory infections in the WASH programme’s documentation, with
the exception of the Hepatitis A outbreak in Azraq and the background information provided
in ACF’s PCAs. A notable feature of Za’atari and Azraq’s management was the strong
commitment to regularly implementing KAP surveys in both camps on an annual basis,
enabling the measurement of change over time. However, the evaluation team did not find
hygiene data with respect to host communities or those in the ITSs.
The information that the evaluation team collected through interviews and focus groups on
hygiene related items indicates that people received the items they needed. Sometimes,
however, quantities of certain items like soap powder were insufficient and at times arrived in
irregular intervals.
Hygiene was communicated in a number of different ways including through community
level hygiene promoters, children’s hygiene clubs, campaign days linked to key global days
such as global handwashing day and use of the mass media.
The evaluation found that consultation regarding preferences for WASH service options was
not always possible, and that meeting preferences was also not always possible. Nonetheless,
the WASH programme’s interventions remained highly relevant. The WASH programme set
standards early and used assessments to identify the best way to address needs given
available options and resource constraints. When the situation stabilized, the WASH
programme consulted more closely with camp residents to discover their preferences and
identify ways that operations could improve. The positive press that Za’atari has received,
(Jaafar, 2017) and Azraq camp residents’ reports of consistent improvement in services
speaks to the relevancy and effectiveness of the WASH programme’s main operations.
UNICEF’s work in host communities has reduced tensions and brought relief to hundreds of
thousands of people. Its work in host communities was well coordinated and of high quality.
One improvement UNICEF could make is rethinking its approach to serving host
communities and ITSs. In each year of the project, UNICEF estimated the number of people
they would need to serve to effectively achieve the WASH programme’s goal in host
communities, and every year the programme fell well short of the target. The reasons for
falling short certainly include lack of funding, as UNICEF stated in its annual reports. This is
certainly true as the international community has underfunded the humanitarian response
regionally, and much of UNICEF’s funding was earmarked for provision of services in
camps and settlements. However, missing the target several years in a row should have
triggered a strategy review and a reorienting of this programme component. In fact, though it
82
falls outside of the scope of this evaluation, senior staff interviews indicated that this strategic
reorientation may come in 2019, as UNICEF applies its innovative capacity and technical
knowledge to assist other donors that are more capable of funding large, municipal systems.
Serving refugees in ITSs should be an immediate concern for UNICEF. While in the early
days of the response, the most vulnerable people were in the camps, by December 2013 it
was clear that people living in ITSs had dire needs that were going unaddressed (REACH,
2013). Despite the vulnerability of people, especially children, living in ITSs, they have
remained remarkably underserved. UNICEF reports that it has worked with partners to raise
resources for ITSs and Schools. The evaluation team could find no recent information
documenting the number, locations, or status of ITSs in Jordan. The ITSs that the evaluation
team visited, which UNICEF serves, suffered from unsanitary and dangerous conditions,
such as unsatisfactorily covered latrine pits138. One can imagine that ITSs not benefitting
from UNICEF’s services are far worse.
Equity of coverage remains a challenging issue that the WASH programme consistently
seeks to address. The programme struggled early on with equity as public facilities in Za’atari
camp were difficult for women and children to securely access, and water trucking and
sanitation services were corruptible leading to some households receiving more water and
better desludging service than others. Building the water and wastewater networks in the
camps, and their management with innovative apps and third-party monitoring, solved the
initial problems, and improved the situation greatly. However, other new issues arose, such as
households in Za’atari receiving equal quantities of water regardless of the number of
household members. The water network in Za’atari was also misused as some households
rigged their water tanks to receive more of their share of water. UNICEF remains vigilant in
the camps and will continue to work to address these challenging issues.
Equity of coverage for women and children in the camps is also challenging. UNICEF
manages this challenge through ensuring that partners that provide camp services have
sufficient numbers of female hygiene promoters and social mobilizers on staff. Partners also
come up with innovative, woman centered mechanisms for programme management, such as
the Lead Mother model implemented by ACF in Azraq. UNICEF’s network expansion to
address graywater was also implemented to address inequity.
The WASH programme’s work at the national level has greatly assisted the government in
forming and implementing policy. National level capacity building has also assisted the
government in using data more effectively. The GoJ values UNICEF’s partnership and sector
coordination. UNICEF could further enhance its role in consulting with the government.
Areas of focus could include reducing NRW, advocating for an improved legal framework,
improving the efficiency of operations, and assisting with strategy implementation. The
evaluation’s interviews with senior UNICEF staff indicate that UNICEF may already have
taken steps in that direction.
The WASH programme completed a comprehensive nationwide assessment of WASH in
schools and created a plan for bringing those schools up to standard. However, achievement
against that plan was limited, and strong pilot projects carried out were not followed up on.
The limited achievement in the WIS area, as in host communities in general, reflects a lack of
strategic consideration when committing to objectives and setting targets.
138 See figure 9.
83
UNICEF’s stakeholders agree that that it has done a solid job in coordinating the sector.
UNICEF has established systems for avoiding duplications or service gaps in the camps. The
government appreciated UNICEF’s leadership in contributing to sector efficiency and
making it more data driven. Partner organizations that the evaluation team interviewed said
that UNICEF was always available to trouble shoot issues or provide useful technical input.
The WASH programme should improve its strategic management. Strategic management
includes the setting of objectives based on the consideration of the specific needs of a well-
defined population compared to an organization’s internal assets, internal shortcomings,
external opportunities, and external challenges. Strategic management during the course of
the evaluation period suffered because the programme’s rapid start-up meant that the WASH
programme team forwent standard assessments, strategic analysis, and programme design
activities. Still, the evaluation team was surprised that over the course of the programme,
monitoring and evaluation protocols and a monitoring data system were not established. The
WASH programme team noted that they do not document their activities adequately139.
Included in strategic management is the implementation a standard monitoring and evaluation
system, which would track results at all levels. Implementing an adequately managed
monitoring system is especially important given the regular, planned turnover among
UNICEF staff. Good record keeping prevents important knowledge from disappearing with
staff as they move to new assignments. It also allows for catching trends in data that may not
otherwise be apparent.
Also importantly, UNICEF’s WASH programme lacks an activity based accounting system,
which would allow it to track spending by year and programme component. A review of the
proposals and reports that UNICEF’s NGO partners demonstrate the standards required to
monitor and financially account for programmatic activities. Improving accounting standards
will increase in importance as UNICEF moves away from working through partner
organizations to directly managing contractors. Contractors will have a greater profit
incentive than NGOs and may recommend replacing equipment more frequently than is
necessary or other such profit generating activities. UNICEF will have to track depreciation
of assets, closely monitor costs, and have the ability to assign indirect costs to programme
activities to properly manage resources.
UNICEF has done everything possible to ensure that its interventions are long-lasting, and
that mechanisms are in place for their operation. Examples include organizing WASH
committees to oversee water access and hygiene promotion, constructing well designed
systems out of durable materials which meet the national standards, and implementing
messaging campaigns for hygiene promotion and water conservation. However, the ultimate
sustainability of the programme is a complex issue. It includes determining if and when
UNICEF will hand WASH management off to another entity, managing the transition of
refugees either back to their place of origin or a permanent settlement elsewhere, the
dismantling or integrating camp WASH infrastructure into Jordan’s municipal systems, and
dealing with Jordan’s medium-term water scarcity issues. Addressing these sustainability
issues will require deeper engagement from multilateral organizations and the agreement of
the Government of Jordan.
UNICEF’s WASH programme accomplishments are impressive and future challenges
complex. The programme’s experience offers much that future programme’s could learn
139 Source: WaSH programme staff interview – 10 October 2017
84
from. This report presents these lessons and recommendations for future programmes in the
following section.
Lessons learned and Recommendations This section presents lessons learned, followed by recommendations. At the end of the
section we include further recommendations based on the evaluation’s findings.
Lessons Lesson 1. Quick decision making, and decisive action saves lives. The decisions and
actions UNICEF took in the early days of the programme were crucial for initial inhabitants
of Za’atari. In the settlements, UNICEF provided water and sanitation services to respond to
the settlements’ exigencies. UNICEF moved quickly and effectively, with clear goals, such as
ensuring that each person had at least 15 liters of water per day. The WASH programme also
recognized that its intervention would have an impact on the larger Ruwaished host
community. To address that impact, UNICEF rehabilitated the community’s water and
sanitation facilities and provided support to identified vulnerable households.
Lesson 2. Installing quality water treatment and delivery equipment is crucial for the
long-term sustainability and use. UNICEF’s installed higher cost and higher quality water
infrastructure at Za’atari, Rukban and Azraq, including the treatment plant for Zaatari. In
each case, a robust design was followed by very good installation and excellent post
construction operation and maintenance. This was borne out through the evaluation team’s
on-site investigations. In a KII with the contractor in Rukban, the operators noted that
UNICEF always purchases high quality equipment. The result is operational efficiency,
equipment that is long-lasting in the harsh environments, and reduced maintenance costs,
downtime, and interruptions to supply.
Lesson 3. Innovative approaches to inefficiencies are scalable: Initially wastewater
management was hugely inefficient. For example, some contractors claimed to remove up to
40 percent more sludge than they actually removed. This stemmed from issues such as
carrying waste in to the camp from outside, which was subsequently charged to the UNICEF
operation. This inefficiency was fixed through the introduction of a voucher system where
communities in receipt of services had to sign for them. Another innovative approach was the
development of a third-party monitoring system, which paid off in improved service, greater
coverage and increased efficiency.
Lesson 4. When direct consultation is not possible, other means of assessment can
ensure programme relevance. In addition to direct consultation with beneficiaries, UNICEF
put a system of assessments and KAP surveys in place. These studies kept UNICEF staff and
sector participants informed of changes in the camps and other locations and assisted the
programme in staying relevant and responding quickly to eventualities.
Lesson 5. The division of labor system, which UNHCR determined and UNICEF
followed, in the camps worked well to determine roles, prevent duplication of services,
ensure that standards were set and enforced, and protect against gaps in coverage. The system
also assisted in understanding cost drivers and making improvements to efficiency.
Lesson 6. Setting strict standards for private sector engagement and holding contractors
to those standards improves efficiency. Emergency situations require engagement with the
private sector, but also present opportunities for gaming the system. Evidence of this
dilemma in the Jordan WASH programme includes some contractors water delivery and
desludging services giving preference to some people’s needs over others and claiming more
85
volume was moved than was accurate. It may also have occurred in ways that weren’t
spotted.
Lesson 7. Documents, tools, and methodologies are available to quickly assist UNICEF
staff in designing and managing WASH programmes in emergency contexts. There are many
lessons learned from the early days of the project regarding issues such as camp site
selection, means of conducting assessments, and methods of determining beneficiaries’
preferences. Most of these have been recorded and addressed in UNICEF guidebooks and
lessons learned documents. Many of these predate the WASH programme's initiation.
Lesson 8. A complaint hotline should not be run by the same organization that is
responsible for the delivery of services. Early in the programme, ACTED managed service
delivery and the complaint hotline. This arrangement was problematic. Complaints were
better addressed when delivered to the implementing partner via UNICEF’s third-party
complaint response mechanism.
Lesson 9. The programme lacks analytical rigor for determining the programme’s
objectives, indicators, or resource allocation, which has led it to overcommit to initiatives for
which it did not have resources or capacity, primarily in host communities. Examples include
the programme’s overestimation of the degree to which it could serve host communities, and
its funding of successful WIS pilots that did not receive follow up or resources to scale. The
WASH programme has operated without strategic planning, and operating theory of change,
or specific, measurable, timebound indicators. The lack of strategic programme design
protocols may have prevented the programme from leading strategically more appropriate
initiatives. These initiatives may have involved promoting wide scale NRW reduction,
rainwater water harvesting, water conservation, and groundwater recharge programs as real
strategic interventions that would yield long standing results.
Lesson 10. One of the greatest challenges faced by the programme was managing the
expectations of refugees and host community members. Refugees came from a relatively
water sufficient region that valued privacy and cleanliness. Host community members lived
in a water scare region and were concerned about newly arriving refugees depleting their
water supply, as well as receiving a free and higher standard of service than locals were
receiving. Creating an information campaign to keep both of these groups aware of the work
that UNICEF was doing, where sacrifices were required, and what they could expect in the
future might have prevented unrest and other difficulties
Lesson 11. The clearest path to sustainability for a programme like the UNICEF Jordan
WASH programme is to transfer activities to the government. However, in the Syria Refugee
Crisis, handing the programme to the government of Jordan was never an option. In taking
lead in the sector, UNICEF has indefinitely taken ownership over the equivalent of a
municipal utility in the form of water and sanitation service provision to Za’atari and Azraq.
Lesson 12. When it initially established protocols with the GoJ, particularly the MoWI,
UNICEF presented a budget and estimation of contributions to infrastructure for which it
hadn’t yet secured funding. The consequence was a misunderstanding with the government
that led to UNICEF not covering project costs because it didn’t meet fundraising targets, and
the GoJ taking infrastructure projects out of its portfolio because it thought they were
covered.
Lesson 13. International NGOs have different risk factors, management structures, and
commitments than UNICEF and other UN agencies. UNICEF often asked NGOs to make
rapid changes, perform out of scope work, or quickly transition in and out of camps. These
requests often felt like last minute decisions to UNICEF’s partners, and didn’t allow them to
prepare adequately in terms of notifying staff, changing commitments, or transitioning assets.
86
Lesson 14. The emergency response phase of the programme required quick decision
making and action. As the report mentioned, UNICEF staff said that there was five days
between UNICEF taking WASH sector lead and the opening of Za’atari. As a result,
UNICEF staff were not aware of UNICEF’s institutional resources that would have helped
response staff plan and manage the organization’s activities in the early days of the
programme.
Lesson 15. Decisions in implementing WASH programmes require trade-offs. For
example, deciding to locate a camp close to an urban population means easier access to
municipal infrastructure, but more tension between the camp and community; whereas
locating a camp in a distant location means less civil tension, but more expensive services
like desludging. Camp residents and community members see the effect of these trade-offs
and speculate as to why they were made.
Lesson 16. Beginning in 2012, UNICEF clearly desired to implement a comprehensive
programme that responded to all refugees needs. As the programme evolved, it has provided
excellent infrastructure and services for people in camps, and struggled to serve refugees
outside of the camps. Thus, commitments to the GoJ and host community and WIS fell short.
This struggle is partially due to a very difficult funding environment, but also a result of a
missing strategic development process.
Recommendations
Reference Lesson Recommendation
Lesson 6 - Private
sector engagement
and contractors
UNICEF should develop a system for contracting WASH services and
monitoring those services. The system should include regular report
objective measures back to the contractor as a means of increasing
efficiency.
Lesson 7 –
Utilizing
UNICEF’s tools
and methodologies
UNICEF should have a global or regional WASH manager that is
responsible for ensuring that emergency responders have the guidance
and tools they need at the beginning of a response. These could include
tools like the WASH Cluster Coordination Handbook.
Lesson 8 – Hotline
management
Ensure that future programmes set up a mechanism for beneficiaries to
report issues/complaints directly to UNICEF, not through an
implementing partner. If UNICEF is the direct implementer, an
accountability mechanism unconnected to the implementation of the
programme should be established.
Lesson 9 –
Improving
analytical rigor
and planning
The WASH programme should implement and monitoring and
evaluation system and comply with the strategic programme
development protocols and reviews that are detailed in the Jordan
Country Programme 2013-2017 document. The WASH programme
should hold itself to the same standard it holds its partners. UNICEF’s
PCA agreements require that partner organizations provide detailed
analysis of the reasoning behind proposed interventions and directly
87
connect activities to that analysis. It also requires that partners set
specific, carefully designed objectives that are strategically coherent.
UNICEF’s partners put monitoring systems in place to track progress
toward these indicators. Monitoring systems are also useful for making
decisions during programme implementation, as they allow managers
to detect initiatives that go astray.
Lesson 10 –
Managing
stakeholder
expectations
Future WASH emergency response efforts would benefit by
understanding how refugees’ standards in camps differ from what
they're accustomed to. When camps are near urban populations,
programmes should seek to understand how host community members
perceive WASH operations. Information campaigns and rallying
resources to support both populations could head off future tensions.
Lesson 11 –
Sustainability
UNICEF should decide if it has the capacity to serve as the manager of
WASH services to Za’atari and Azraq indefinitely, and if that is a
proper role for the organization. If it determines that it is not a proper
role, it must explore options for transfer. If it decides to stay in the role
for as long as required, it should upgrade its management and
accounting systems appropriately.
Lesson 12 –
Managing
government
expectations
In the future, UNICEF should make sure that it communicates clearly
with government partners the likelihood of protocol commitments
occurring. This issue is also tied to shortfalls in strategy and planning.
Lesson 13 –
Managing NGO
partners
UNICEF should seek to better understand their partners business
models and capacities. Forming an understanding would smooth
management of projects implemented through these partners.
Lesson 14 –
Managing
institutional
resources
UNICEF should pre-load a computer with the software and manuals
that emergency response staff need in the early days of a project. The
preloaded information should include up-to-date WASH emergency
response manuals or tools, a pre-developed results indicator
framework and logic model, and other strategic tools that will assist in
decision making and goal setting. It should also include management
tools. Such as a pre-set up version of QuickBooks or similar software
with classes and accounts need to track expenditures, and any contract
templates, human resources documents, financial information, or other
required management tools.
If UNICEF has these resources already available, a system for
ensuring that deployed staff have downloaded the materials would be
helpful. The staff should also have a conversation with someone in the
emergency so that they are prepared, and that there is HQ follow up
support for staff once deployed staff arrive at their post.
88
Lesson 15 –
Negotiating trade-
offs among
beneficiaries
Clear communication about why one decision was made over another
would help diffuse tensions in camps and communities. Where
possible, UNICEF should seek to explain why decisions were made,
such as the decision to put public latrines in Azraq, but private latrines
in Za’atari, so that people feel informed and their expectations are
managed.
Lesson 16 –
Achieving some
programme targets
while missing others.
UNICEF should formulate a realistic strategy for the services it can
provide given the needs of the population it serves and the resources it
has access to. An important part of any organization’s strategy is
knowing what it cannot provide. Once that is identified, UNICEF
could serve in an important leadership role to assist the international
community in rallying resources and capacity to address the uncovered
and vulnerable populations.
Other Recommendations
• UNICEF should address the disparity in water allocation in Za’atari. The water
network in Za’atari provides the same quantity of water to each household regardless
of the number of people in the household. Households can have from one person to
more than eight people. If UNICEF strives for 35 liters per person per day on average,
the wide variation in household size means that some people are getting very little,
while others receive much more than they are entitled.
• The trend toward cutting funds for hygiene promotion is disconcerting. UNICEF
should avoid cutting those funds if possible, or if they must, plan for the
contingencies that could result.
• Partner interviews revealed that UNICEF’s role in the Wash in Schools initiatives
may not be clear. UNICEF should clarify its role in this category of activity and
coordinate a stakeholder strategy.
89
Works Cited 1. ACF. (2017). Azraq Camp Knowledge Attitudes and Practices Survey Report 2017. Amman:
ACF.
2. (n.d.). ACTED Programme Cooperative Agreement 23-13.
3. ACTED, World Vision, ACT, RI, and Save the Children. (2016). Azraq Camp Knowledge
Attitudes and Practices Survey Report 2016.
4. Al Wazani, K. W. (2014). The Socio-Economic Implications of Syrian Refugees on Jordan.
Amman: Konrad Adenauer Stiftung.
5. Bocco, R. (2010). UNRWA And The Palestinian Refugees: A History Within A History.
Oxford Journals.
6. Financial Tracking Service. (2018, October). Appeals and response plans 2018. Retrieved
from
https://fts.unocha.org/appeals/overview/2018/sectors?f%5B0%5D=destinationPlanIdName%3
A%22629%3ASyria%20Humanitarian%20Response%20Plan%202018%22&f%5B1%5D=de
stinationPlanIdName%3A%22658%3ASyria%20regional%20refugee%20and%20resilience%
20plan%20%283RP%29%2020
7. Financial Tracking Service. (2018, October). Financial Tracking Service. Retrieved from
Syrian Arab Republic - Civil Unrest (from 2012) 2018:
https://fts.unocha.org/emergencies/600/summary/2018
8. Freeman, M., & al, S. M. (2014). Hygiene and health: Systematic review of handwashing
practices worldwide and update of health effects.
9. Hūls, V. (2017). UNICEF WASH Action in Humanitarian Situations; Synthesis of Evaluations
201-2016. New York: UNICEF.
10. International Monetary Fund. (2017). Jordan, Selected Issues. Washington, DC: The
International Monetary Fund.
11. Jaafar, S. (2017, April 5). This refugee camp in Jordan has turned into a frontier town for Syrians escaping war. Retrieved from PRI: https://www.pri.org/stories/2017-04-05/refugee-
camp-jordan-has-turned-frontier-town-syrians-escaping-war
12. (n.d.). JEN Programme Cooperative Agreement 25-14.
13. Kis, A. e. (2016). Water demand management in the context of water services: Jordan.
Regional Centre for Energy Policy Research.
14. Ledwith, A. (2014). Zaatari: The instant City. Boston: The Afforable Housing Institute.
15. Leestma, D. (2017, October 1). Draining a lake in Daraa: How years of war caused
Muzayrib to dry up. Retrieved from Middle East Eye:
https://www.middleeasteye.net/news/draining-lake-daraa-how-years-war-caused-lake-
muzayrib-largely-disappear-1202640340
16. Luck, T. (2013, April 21). In Jordan, tensions rise between Syrian refugees and host
community. The Washington Post.
17. Middle East and North Africa Out-Of-School Children Initiative. (2014). Jordan Country
Report on Out-Of-School Children. UNICEF.
18. Ministry of Planning and International Cooperation. (2014). Jordan Response Plan 2015 for
The Syria Crisis. Amman: The Hashemite Kingdom of Jordan.
90
19. Ministry of Water and Irrigation. (2016). National Water Strategy 2016 - 2025. Amman:
Hashemite Kingdom of Jordan.
20. OXFAM PCA 32-13 Executive Summary. (n.d.).
21. REACH. (2013). Informal Tented Settlements in Jordan: A Multi-Sector, Baseline
Assessment". UNICEF.
22. REACH. (2014). Access to Water and Tensions in Jordanian Communities Hosting Syrian
REfugees. Amman: REACH.
23. REACH. (2014). Syrian Refugees Staying in Informal Tented Settlements in Jordan. Amman:
REACh.
24. redrUK. (2009, January). Water, Sanitation, and Hygiene (WASH) Cluster Coordination
handbook.
25. ReliefWeb. (2017). Humanitarian Funding Update; December 2017. ReliefWeb.
26. Salman, M., & Mualla, W. (2008). Water demand management in Syria: centralized and
decentralized views. Water Policy, 549-562.
27. Seely, N. (2012, March 1). Jordan’s “open door” policy for Syrian refugees. Foreign Policy.
28. Seely, N. (2012, October 1). Most Jordanians say no to more Syrian refugees. The Christian
Science Monitor.
29. Terms of Reference for Service Contracting - Rukban. (n.d.).
https://www.unicef.org/jordan/RUKBAN_ToR.pdf.
30. The Sphere Project. (2018). Water supply standard 1: Access and water quantity. Retrieved
from Sphere Handbook: http://www.spherehandbook.org/en/water-supply-standard-1-access-
and-water-quantity/
31. The United Nations. (2013). Syria Regional Response Plan; January to December 2013. The
United Nations.
32. UNHCR. (2012, August 28). Syria: Doubling of refugees fleeing to Jordan. Retrieved from
UNHCR USA: http://www.unhcr.org/en-us/news/briefing/2012/8/503ca1c99/syria-doubling-
refugees-fleeing-jordan.html
33. UNICEF. (2012, October). In Jordan, huge water delivery and testing operation meets the life-saving water and sanitation needs of Syrian refugees. Retrieved from UNICEF Jordan:
https://www.unicef.org/wash/jordan_66157.html
34. UNICEF. (December 2012). Syria Crisis Bi-Weekly Humanitarian Situation Report.
UNICEF.
35. UNICEF Jordan. (2012). Jordan Country programme document; 2013 - 2017, pg 11.
Amman: UNICEF JOrdan.
36. UNICEF Jordan. (2012). SitRep 20 December 2012. Amman: UNICEF.
37. UNICEF Jordan. (2012). Syrian Refugee Response Overview. Amman : UNICEF.
38. UNICEF Jordan. (2013). SitRep 25 January 2013. Amman: UNICEF.
39. UNICEF Jordan. (2013). UNICEF Annual Report 2013 - Jordan. Amman: UNICEF.
40. UNICEF Jordan. (2016). UNICEF Jordan Annual Report 2016. Amman: UNICEF.
41. UNICEF Jordan. (2017). UNICEF Jordan Annual Report 2017. Amman: UNICEF Jordan.
42. UNICEF, ACF. (2017). Azraq Camp Knowledge and Attitudes and Practicies Survey Report
2017. Amman.
91
43. WASH Sector Working Group. (2012). Water, Sanitation and Hygiene (WASH) Sector Working Group Strategy for the Syrian Refugees Response in Jordan; December 2012 - June
2013. Amman.
44. Wildman, T. (2013). Water Market System in Balqa, Zarqa, & Informal Settlements of
Amman & the Jordan Valley - Jordan August - September 2013. Oxfam.
45. Wilkes, S. (2012). Jordan opens new camp for Syrian refugees amid funding gaps. Retrieved
from UNHCR: Jordan opens new camp for Syrian refugees amid funding gaps
46. World Vision Programme Cooperation Agreement 19-13 . (n.d.).
47. Za'atari Hygiene Promotion Working Grouip. (2015). Knowledge Attitudes and Practices
Survey; Za'atari Refugee Camp 2015. Amman.
48. Za'atari Hygiene Promotion Working Grouip. (2016). Knowledge, Attitudes and Practices
Survey Za’atari Refugee Camp 2015. Amman.
49. Za'atari Hygiene Promotion Working Groups. (n.d.). 2012 Baseline KAP Survey, Za'atari.
92
Annex A: Evaluation Terms of Reference STATEMENT OF WORK/TOR
Terms of Reference for the
Evaluation of UNICEF#s response to the Water, Sanitation and Hygiene needs in Jordan as a
result of the Syrian refugee crisis (July 2012 to July 2017)
Table of Contents
1. Background and context 1
2. Evaluation purpose and objective 2
3. Evaluation scope 3
3.1 Coverage and level of results 4
3.2 Geographical coverage 4
3.3 Time-period of the evaluation 4
4. Key evaluation questions 4
4.1 Relevance 4
4.2 Effectiveness 5
4.3 Efficiency 5
4.4 Sustainability 5
4.5 Coverage 6
4.6 Coordination 6
5. Evaluation Stakeholders 6
6. Methodology 6
6.1 Desk Review 7
6.2 Meetings with WASH section 7
6.3 Visits 7
6.4 Key informant interviews (Kll) 8
6.5 Focus Group Discussions (FGD) 8
6.6 Participation in at least 2 coordination meetings in Za#atari, Azraq and the border, and the
sector 8
7. Work assignment 8
7.1 Phase 1. Desk review 8
7.2 Phase 2. Prepare an inception report with project delivery plan 8
7.3 Phase 3. Data Collection 9
7.4 Phase 4. Draft evaluation report 9
7.5 Phase 5. Finalize the Final Evaluation Report 9
7.6 Phase 6. Prepare and present two PowerPoint presentations 9
8. Management of the Evaluation 9
9. Expected deliverables & delivery dates 10
10. Official travel involved 10
11. Desired qualifications 10
11.1 Team leader 11
11.2 Two WASH Experts 11
12. Estimated duration of the contract and payment schedules 11
13. Roles and responsibilities in the evaluation process 11
13.1 The Evaluation Manager 11
13.2 Evaluation Reference Group (ERG) 12
13.3 The Evaluation Team 12
14. Performance indicators for evaluation of results 12
15. Frequency of performance reviews 12
18. Equity, gender, human rights, including child rights 15
19. Ethical principles and premises of the evaluation 15
20. UNICEF recourse in case of unsatisfactory performance 15
1. Background and context
Jordan, is described as being an arid to semi-arid country with average annual rainfall of 200
mm/year over the country. On this basis, it is one of the most water scarce countries in the world,
and has been facing chronic challenges to ensure the provision of clean water and sanitation
services to its population for a number of years. The chronic water situation has been
compounded by successive waves of refugees from neighbouring areas, the most recent being
the Syrian crisis. Since the onset of the Syrian crisis in 2011, more than 659,000 Syrian refugees
have crossed the border into Jordan to seek assistance and refuge from the atrocities underway
in Syria. Since then, the population in some areas has risen dramatically, particularly those in the
93
north of the country (by more than 60% particularly in the Northern Governorates e g. Irbid). The
2015 Census reported that the total population of Jordan as 9.5 million, showing a considerable
increase over the recent years, and 30.6% of the population is non-Jordanian, including Syrian
refugees.
While an estimated 15% of the registered Syrian refugees live in refugee camps, the remaining
85% are living in host communities, which has caused tension between Jordanians and Syrians,
due to a perception that the Syrian refugees are absorbing the limited resources. This tension has
resulted in conflicts and disputes, particularly in the northern governorates, where the additional
pressure on resources has been most acutely felt.
According to the Joint Monitoring programme (JMP) 2015, access rates to a piped water network
in urban areas is estimated at 93%, and 80% in rural areas. However, in reality, most areas have
access to water once a week (in the main cities) reducing dramatically in rural areas, with some
areas reported to receive water once a month. Due to the limited resources, and associated
limited access, average household consumption is 71 l/p/d (National Water Strategy 2016 #
2025). As a result, most households have significant amounts of water storage facilities and when
necessary, supplement these with private water tankering services. For wastewater systems, the
situation is even more critical. Prior to the Syrian crisis, an estimated 62% of the population was
connected to a sewerage system nationally, however this figure is much lower in rural areas. In
areas where there has been a significant recent increase in the population, the existing
infrastructure has been unable to support the increase in the volume of wastewater which has led
to overflows in urban areas, causing significant environmental health concerns.
As a result of the influx, in addition to the growing expansion of the Jordanian economy, as well
as natural population growth, the demand for water and wastewater services has increased
significantly, which has contributed to over expansion of groundwater from aquifers. The Ministry
of Water and Irrigation (MOWI) estimates that average rates of over-pumping exceed 200
Mm3/year resulting in seven of the twelve groundwater basins being over-pumped at between
135 to 225% of the estimated safe yields of the aquifers. The critical situation is further
compounded by high levels of leakage, estimated at more than 60% in some of the northern
governorates, in the existing systems as well as non-revenue water.
In July 2012, UNICEF commenced its Water, Sanitation and Hygiene (WASH) programme,
initially focusing on refugee response and providing WASH services in Za#atari camp, and
expanding to a total of six settlements (currently five are active) over time. The initial focus on
refugee response, developed in line with the Syria Regional Response Plan (RRP), has gradually
expanded to cover resilience in line with the National Resilience Plan which was developed in
2013 to cover the period of 2014-2016 . Also since 2012, UNICEF has led the WASH Sector for
the humanitarian response.
UNICEF continues to be a key player in providing services to refugees living in the four main
camps of Za#atari, Azraq, King Abdullah Park and Cyber City; two settlements in Rukban and
Hadalat; as well as in Informal Tented Settlements (ITSs) and host communities. Currently, the
population of the five refugee camps/settlements is estimated to be about 170,000 people. It is
understood that the camp population peaked to 120,000 in Za#atari camp in late 2013 and the
population in Za#atari declined to approximately 80,000 thereafter, with Azraq opening in April
2014. The two settlements at the border, Rukban and Hadalat, expanded rapidly in 2016, to a
current estimated combined population of 55,000. As the movement to/from the camps has been
very fluid, there is no actual figure of the number of people who have benefitted from WASH
services since 2012, however, this is estimated to be more than 400,000.
Although UNICEF has had an active presence in Jordan since 1952, UNICEF was not active in
the WASH sector prior to the refugee crisis. As a result, the UNICEF Jordan Country programme
of Cooperation for 2013-2017, when originally developed in 2012, did not include WASH
programme. Seeing the rapid scaling-up of the refugee response in 2013, UNICEF Jordan revised
its results framework for 2013-2017 to include an output-level result focusing on Core
Commitments for Children in Humanitarian Action. This was further reviewed in 2014 to reflect
resilience more prominently, at the time of the Strategic Moment of Review and Reflection and in
light of the National Resilience Plan. The rapid onset and scale-up of the crisis has meant
revisions in the UNICEF programme Results Framework.
UNICEF WASH programme has evolved to work at four different levels as follows:
- Provision of water, sanitation and hygiene services, as well as the dissemination of key
messages to refugees on a daily basis in four camps and two settlements;
- Provision of support in the host communities (infrastructural support, support to
94
vulnerable households, informal tented settlements) including WASH in schools;
- Support to the WASH sector through technical support to the Ministry of Water and
Irrigation in the development of key planning documents (National Resilience Plan 2014, Jordan
Response Plan 2015, 2016-2018, 2017-2019), strategies (National Water Strategy 2016 to 2025)
and policies, as well as the nationwide WASH in Schools Assessment (2014/5) and the National
WASH in School Standards;
- Coordination of the humanitarian WASH sector at a camp and national level.
In terms of UNICEF Country programme in Jordan, the WASH programme has been one of the
largest in terms of scale and scope, constituting about 53% (USD 78 million) of the total budget in
2013, 54% (USD 100 million) in 2014, 31% (USD 56 million) in 2015,
27% (USD 61 million) in 2016 and 30% (USD 60 million) in 2017. The costs and logistics
required, as well as the monitoring associated with the provision of services in the camps, have
been enormous. To address this, and to improve the consistency, reliability, quality and equity of
services in the camps, UNICEF has been implementing a transitioning strategy through the
construction and operation of five boreholes and two wastewater treatment plants, with water and
wastewater networks currently under construction since 2015.
2. Evaluation purpose and objective
The purpose of the evaluation is to assess whether UNICEF WASH programme met its intended
results, and to generate lessons learnt which can inform WASH programmes in similar context.
The overall objective is to conduct an independent and critical assessment to the WASH
programme and its support to the sector and government institutions, using evaluation criteria of
relevance, effectiveness, efficiency and sustainability as well as coverage and coordination in the
target communities (camps, ITSs and host communities). The Impact criterion is excluded due to
the relatively short time period since the start of the interventions. Additionally, the evaluation is
anticipated to collect evidence and lessons learnt from
UNICEF#s WASH programme implementation to inform future programming, and to develop
recommendations to further strengthen delivery and planning of WASH services in similar
protracted humanitarian crisis in view of building resilience and sustainability. More specifically, this
evaluation will aim for the following key specific objectives:
a) To assess the relevance of the WASH programme (emergency response and resilience
components) in the context of Syrian Crisis in Jordan and its national priorities. To assess the
relevance of WASH programme design to achieve the expected results, taking into account
the appropriateness of the interventions to targeted communities.
b) To assess the effectiveness of the WASH programme and to measure to what extent the
programme has achieved its set results in an equitable manner.
c) To assess the efficiency of the WASH programme and to what extent the programme
has used resources (human, financial and others) and coordinated in an efficient manner over
time. To assess the cost of the response per unit of aggregation as compared to the cost being
incurred by similar WASH programme implemented in other countries.
d) To assess the sustainability of the WASH programme and its results, taking into account
the institutionalization and capacity building of national counterparts as well as a potential exit
strategy.
e) To assess the coverage of the most vulnerable population by the WASH programme and
provision of quality services in a coherent manner.
f) To assess the coordination of UNICEF as a sector lead to avoid duplication and gap and
enable effective partnerships to reach the most vulnerable population.
g) To document lessons learnt and good practices that will inform future programming,
replication in other countries in the region, suggesting different options for UNICEF and partner
agencies.
It is anticipated that the evaluation, and the expected Lessons learned and good practices, will
also benefit the Ministry of Water and Irrigation, implementing partners in addition to donors of
Jordan, as well as future UNICEF WASH programming in the region and globally.
3. Evaluation scope
This evaluation will be summative in nature. It should focus mainly on UNICEF#s WASH
programme as it evolved from refugee response to include resilience building from July 2012 to
July 2017. This includes the initial response which aimed to meet the urgent needs of Syrian
refugees, primarily residing in refugee camps, as well as resilience programming to support host
95
communities and government bodies to respond to the impact of the crisis. This evaluation will
assess UNICEF#s WASH programme which includes water, sanitation (WASH facilities, solid
waste management and desludging) and hygiene promotion implemented by UNICEF#s partners
and contractors. While UNICEF started providing life-saving WASH services at refugee transit
centres in March 2012, this evaluation will cover UNICEF#s WASH programme from the opening
of the Za#atari refugee camp in July 2012, and will look at programming in the four camps of
Za#atari, Azraq, Cyber City (currently closed) and King Abdallah Park, in addition to host
communities (and informal tented settlements), as well as Rukban and Hadalat at the
Syrian/Jordanian border. Although there are Syrian refugees registered in all 12 Governorates of
Jordan, 90% of the refugees are registered in Zarqa, Irbid, Mafraq and Amman Governorates,
with 99% of the refugees registered in ten Governorates.
In addition, this evaluation will assess support to the sector through technical support provided to
the Government, and UNICEF role in leading the sector at the camps and at the national level.
Stakeholder participation is important for this evaluation, and their views and inputs should be
reflected through means such as interviews and focus group discussions. Special emphasis will
have to be made on Ministry of Water and Irrigation (MWI), Ministry of Education (MoE), UNHCR,
WASH sector, local community leaders and International Partners (IPs) who were directly
involved in the UNICEF WASH response since its early stages. In addition, the voices of
caretakers, parents (both mothers and fathers) and children who benefited from UNICEF#s
WASH intervention should be captured during stakeholders consultations. Representatives from
the Ministry of Water and Irrigation will comprise a key component of the Evaluation Reference
Group.
3.1 Coverage and level of results
The evaluation will focus on four components of the UNICEF#s WASH programme in Jordan and
will include:
a) Provision of equitable WASH services and the dissemination of WASH messages in the four
camps (Za#atari, Azraq, King Abdullah Park and Cyber City) and two settlements in Rukban
and Hadalat, to the most vulnerable people namely children and women
b) Provision of gender equitable support to the most vulnerable people in host communities (in
all 12 Governorates) namely to children and women
c) Support to the WASH sector through technical support to the Ministry of Water and Irrigation
d) Coordination of the sector at camps and national level
3.2 Geographical coverage
The geographical scope of the evaluation will cover the following:
e) The four major camps of Za#atari, Azraq, King Abdullah Park and Cyber City; and two
settlements of Rukban and Hadalat. In addition, the evaluation will cover a number of Informal
Tented Settlements (ITSs) (in 2017, 111 ITSs are provided with WASH interventions) and will be
based on discussions with the consultant;
f) Host community interventions in all 12 Governorates. 3.3 Time-period of the evaluation The period July 2012 to July 2017 will be considered as the timeframe for this evaluation.
4. Key evaluation questions
The evaluation will apply Organization for Economic Cooperation and Development Assistance
Committee (OECD-DAC) criteria of relevance, efficiency, effectiveness, and sustainability as well
as relevant humanitarian criteria, taking into consideration the purpose, objectives and scope of
the evaluation. The Evaluation team should answer the following key questions, but not limited to.
They are requested to unpack them and should propose more detailed ones as part of the request
for proposal and inception report.
The humanitarian criteria of coverage and coordination are directly addressed and should be
investigated throughout the evaluation process.
4.1 Relevance
a) To what extent were affected people consulted on their preferences with regards to water and
sanitation service options and level of services? Were interventions appropriate in terms of
meeting their basic needs? Was there a feedback loop and monitoring system in place for
reflecting the learning and evidence to improve programming, especially for the humanitarian
response?
b) To what extent did the interventions consider the needs and the priorities of the MWI and
were they coherent with global references and the regional and national response to the Syria
96
Crisis?
c) How responsive was the UNICEF WASH programme over time to changes in the external
environment?
4.2 Effectiveness
a) To what extent did UNICEF#s response achieve its intended outcomes (as stated in
UNICEF#s Country programme Outcome and Outputs as given in Annex A)? To what extent
did UNICEF achieve equity results, especially for children and women specific interventions?
b) What were the strengths and weaknesses of the interventions in achieving results in the
camps/settlements, ITSs and host communities on the following:
- Water (Litre/day, supply, quality, consistency, reliability, accessibility, and equity)?
- Sanitation (adequacy, appropriateness, accessibility)?
- Hygiene promotion (appropriateness of the messages, timeliness, behaviour change)?
- WASH in schools?
- National level support?
4.3 Efficiency
a) To what extent did the programme use available resources in an economic manner to provide
WASH services and facilities in camps/settlements and host communities, to ensure child
rights, equity and gender equality?
b) To what extent were the costs to deliver water and wastewater services, as well as solid
waste management, rationalized and optimized in the camps and settlements to ensure Value for
Money? How do these compare to similar situations at the local or regional level? Could more
cost effective operations/interventions have been undertaken at an earlier stage?
c) To what extent did UNICEF try to minimize duplication/gap in WASH interventions in
camps? And how early did UNICEF move to more cost-efficient interventions?
d) To what extent were resources (financial and human resources) allocated for the range of
intervention types, with particular attention paid to the respective allocations for humanitarian
response and resilience, to achieve the intended results?
e) To what extent did UNICEF advocate with the Ministries to mobilise their available
resources to address the identified needs? And how efficient was UNICEF#s support to ministries
(i.e. MWI and MOE) to reaching most vulnerable people? Did the WASH programme maximise
the potential collaboration with other UNICEF programme sections and partners?
4.4 Sustainability
a) Was the infrastructure in the camps constructed in such a manner to ensure long term
functionality, and will continue to service the most vulnerable people even when UNICEF is no
longer directly supporting? This should be investigated in terms of Water and Wastewater
infrastructure.
b) To what extent did UNICEF take sustainability into account at the early stage for water,
wastewater and solid waste management services (e.g. transition to local government institutions
and local capacity)?
c) To what extent did UNICEF#s interventions increase the resilience of the Government
and respective target populations? To what extent will UNICEF-supported projects (schools, ITSs,
water and wastewater infrastructure, mobilisation) continue to be operational after the end of
UNICEF#s financial support
d) Does the MWI have the necessary mechanisms in place and capacity to ensure the
water and sanitation needs of the most vulnerable people will be met as a result of UNICEF#s
support and advocacy, if another large scale humanitarian crisis occurs in the future?
e) To what extent did UNICEF document its WASH interventions and make available to the
sector learning on water, wastewater, solid waste management and mobilisation?
f) To what extent were environmental protection and sustainability taken into account in the
design and implementation of programmatic interventions?
4.5 Coverage
a) Did UNICEF#s WASH response identify and reach the most vulnerable people? To what
extent did specific interventions strike an equitable balance between interventions in
camps/settlements and in host communities, to reach the most vulnerable people?
b) Did UNICEF as Sector Lead ensure that the needs of those in most need were
addressed in a coherent manner, particularly in comparison to alternative existing systems at the
local or regional level (i.e. UNRWA camps)? If not, how?
97
4.6 Coordination
a) To what extent did UNICEF role as WASH sector lead enhance response processes and
results to avoid duplication and gap in interventions?
b) Were partnerships effective and leveraged to the maximum extent to assist the most
vulnerable population in camps/settlements, ITSs and in host communities?
5. Evaluation Stakeholders
The stakeholders that were found relevant to this evaluation by the WASH programme are listed
in the below table 1. They were divided into 2 categories in terms of interest and influence.
6. Methodology
The evaluation methodology will mainly utilise qualitative methods, with a combination of tools for
data collection and analysis. The participatory approach is suggested to ensure participation of
various social groups including the most marginalized. Quantitative analysis will be carried out
through the desk review of the following major data sources, but not limited to:
Camps:
- Monthly, quarterly and final reports from partners
- Third party monitoring data (UNOPS) on beneficiary interviews, wastewater volumes, water
delivery and water quality
- Knowledge Attitude and Practice surveys (Za#atari 2013 & 2014/2015; and Azraq 2016 &
2017)
- Comprehensive Child Focussed Assessment (Za#atari 2015; Azraq 2015 and 2017)
- Partnership agreements and budgets
- Field visit reports
- Monthly Donor updates and Situation Reports
- Assessment reports
- Needs assessments -Transitioning Strategies
- Contracts
- Terms of Reference for the Groundwater Assessment for Za#atari and Azraq
- Water level and quality monitoring data
Settlements (border):
- Monthly Border updates and Situation Reports
- Partnership agreements
- Partner reports
- Contracts
- Third party monitoring data
- Security incident reports
- Distribution data
- Field visit reports
Host Communities:
- Nationwide WASH in Schools assessment
- Partnership agreements
- Partner reports
- Field visit reports
- Contracts
- Situation of Syrian children in host communities in Jordan (Frequent monitoring)
Table 1. Potential stakeholders Interest Influence
- MWI
- MoE
- Donors
- UNICEF
- UNHCR
- Sector
- NGOs - Women
- Children (girls and boys)
- Local Communities
- WASH Committees
- Schools (including parents, teachers and students)
98
ITS:
- Partnership agreements
- Partner reports
- Field visit reports -Transitioning Strategy
National-level support:
- Vulnerability assessments for water and wastewater (2014/2015; and 2016)
- Jordan Response Plan (2014; 2015; 2016/2017; 2017/2019)
- Regional Response Plans
- National Water Strategy (2016 # 2025)
The evaluation team is expected to propose a relevant set of tools and methods, as appropriate
to answer the evaluation questions. This should be detailed in the form of an evaluation matrix,
showing methods against proposed evaluation questions. Methodologies proposed by the
consultant will be subject to consultation with the UNICEF WASH and Planning Monitoring and
Evaluation (PME) sections, and WASH Working Group as appropriate. As part of methodology, it
is essential to articulate how ethical considerations will be taken into account in the design of the
evaluation and throughout the evaluation process. Methodology should include, but not limited to
the following:
6.1 Desk Review
Undertake a Desk Review: review of similar evaluations such as the recent Regional assessment
of UNICEF#s response, review of WASH Annual Reports, partner programme Cooperation
Agreements and programme Documents, contracts, funding tables, KAP surveys, third party
reports, UNOPS and partner reports, minutes and resources of Sector meetings, consultation of
UNHCR information platforms, review of Outputs (Water Strategy, vulnerability maps) and other
documents relevant to the evaluation.
6.2 Meetings with WASH section
At the start of the evaluation, there will be a telephone call for a briefing with key WASH and PME
sections personnel for an overview of the activities undertaken and the timeline of events.
Arrangements for additional meetings with the WASH section will be arranged as needed,
throughout the evaluation process.
6.3 Visits
Conduct visits to some of the main interventions in the camps and host communities:
a) Visits to the six camps/settlements of Za#atari, Azraq, King Abdullah Park, Cyber City, Rukban
and Hadalat;
b) Visits to the host communities.
6.4 Key informant interviews (Kll)
Conduct interviews with relevant stakeholders and beneficiaries as follow:
a) Duty bearers: This would include WASH section personnel, UNICEF Management, UNICEF
WASH Regional team for the Middle East and North Africa Regional Office, Ministries, NGO
partners (main partners in camps, WASH in Schools and ITSs) and UNHCR representatives.
b) Right holders: Camps, settlements, host community and ITS representatives
6.5 Focus Group Discussions (FGD
Conduct Focus Group Discussions with:
Beneficiaries in the five major active camps of Za#atari, Azraq, King Abdullah Park, Rukban and
Hadalat and the host communities, ITSs and schools. The FGDs should be disaggregated by age
and gender (women, men, children and youth);
6.6 Participation in at least 2 coordination meetings in Za#atari, Azraq and the border, and the sector
7. Work assignment
The key tasks to be undertaken under this evaluation are as follows (see summary in Table 2):
Table 2: Key tasks of the evaluation
Phase Activity
Phase 1 Desk review
Phase 2 Prepare an Inception Report
Phase 3 Data collection
99
Phase 4 Draft evaluation report
Phase 5 Final Evaluation Report
Phase 6 PowerPoint presentations
7.1 Phase 1. Desk review
Undertake a desk review of all relevant documents (e.g. the recent Regional assessment of
UNICEF#s response), review of WASH Annual Reports, programme Cooperation Agreements,
funding tables, consultation of UNHCR information platforms, review of Outputs (Draft Water
Strategy) etc.
7.2 Phase 2. Prepare an inception report with project delivery plan
The Inception Report should be developed in line with UNICEF global standards and should
include: (1) purpose and scope, (2) evaluation criteria and questions, (3) evaluation methodology,
(4) evaluation matrix, (5) a detailed evaluation delivery plan within the designated timeframe
(seven months), and (6) outline of the final report. Methodology should also include the evaluation
approach, and theory of change. The outline of the evaluation report should be in line with the
UNICEF-Adapted UNEG Evaluation Reports Standards, and UNICEF Global Evaluation Report
Oversight System.
The inception report will be used as an initial point of agreement and understanding between the
evaluation team and the evaluation managers and should be approved by the evaluation
Reference Group. Field visits and data collection can#t start before finalization of the inception
report.
7.3 Phase 3. Data Collection
Conduct the necessary interviews, field visits, consultations as agreed in the Inception Report,
which are expected to include:
a) Meetings with WASH section;
b) Conduct Field Visits to the six camps/settlements, randomly selected schools and ITSs, in
addition to host communities;
c) Kll with current and historic duty bearers (including ex-UNICEF and partner staff);
d) Kll with right holders;
e) FGD with rights holders and WASH partners;
f) Attend Coordination meetings in the camps.
7.4 Phase 4. Draft evaluation report
Perform the analysis and draft the evaluation report. It must be compliant with UNICEF-adapted
UNEG evaluation report standard (http://www.uneval.org/). The Evaluation report should
systematically answer the key evaluation questions included in the objectives sections of this
TOR. It should fairly and clearly represent the views of the different actors/stakeholders. It should
clearly give the findings, conclusions and recommendations in a way that is substantiated by
evidence. All recommendations included in UNICEF#s
evaluation require management response. It is pertinent that all recommendations are clear and
specific to what UNICEF Jordan could do or influence.
Validation process of the inception report will be as follow:
a) Disseminate the Evaluation Report to the WASH and Planning Monitoring and Evaluation (PME) Sections;
b) Collate feedback and integrate comments where agreed;
c) Finalize the draft evaluation report and submit to WASH and Planning Monitoring and
Evaluation (PME) Sections.
7.5 Phase 5. Finalize the Final Evaluation Report After review of the draft report, the
consultant will:
a) Collate final feedback made on the draft report and integrate all comments;
b) Finalize the final evaluation report and submit to WASH and Planning Monitoring and
Evaluation (PME) Sections.
The final report will be in English and Arabic of 60 to 80 pages in length (excluding annexes). It
will also contain an executive summary of no more than 4 pages that includes a brief description
of the programme, its context and current situation, the purpose of the evaluation, its methodology
and its major findings classified as per the evaluation objectives, conclusions and
recommendations.
100
7.6 Phase 6. Prepare and present two PowerPoint presentations
Prepare two PowerPoint presentations detailing the findings of the evaluation. They will be
presented during:
a) An open session for the Jordan Country Office;
b) And a second one for key sector partners.
8. Management of the Evaluation
To ensure independence, this evaluation will be co-managed by WASH Specialist and Monitoring
& Evaluation Officer at UNICEF Jordan Country Office. A reference group will be formed,
engaging subject matter experts from inside and outside of UNICEF, including the UNICEF
Regional Office for Middle East and North Africa. Members of the reference group will be engaged
and consulted at key milestones of the evaluation process such as review of the TORs, inception
report, and draft reports.
Selection of the evaluation team will be made through an open and competitive bidding process.
Review of technical proposal will be done by at least three members.
WASH Specialist will be responsible for technical oversight in relation to the WASH programing,
support to the evaluation process by providing necessary background information, data, contact
information, as well as management of contractual issues once the bidding process completes.
Monitoring & Evaluation Officer will oversee and quality assure the evaluation, ensuring that it
complies with the UNICEF global standards and norms.
The evaluation team will report to WASH Specialist and Monitoring & Evaluation Officer at
UNICEF Jordan Country Office who will serve as key contact points.
9. Expected deliverables & delivery dates The assessment team will be expected to submit the following deliverables as per the start date of 02.OCTOBER.2017 (table 3):
Table 3: Deliverables and timeframe # Deliverables Type and language Delivery Date 1 Inception
Report with Project Delivery Plan Soft
(within 3 calendar weeks of start date, or 21 working days)
2 Draft Evaluation Report Soft copy (in English) 04.MARCH.2017
(within 5 calendar months of the start date, or 126 working days)
3 Final Evaluation Report accepted Soft copy (in English and Arabic) 02.MAY.2018
(within 7 calendar months of the start date, or 180 working days)
4 Two PowerPoint presentations developed and presented at two sessions Soft copy (in
English and Arabic) 09.MAY.2018
10. Official travel involved
Travel to Amman and field locations in Jordan should be anticipated as per the evaluation
methodology and deliverables. All travel costs should be planned properly in financial proposal.
UNICEF will not provide transport support or field travel arrangements, thus the estimated cost of
travel should be included in the financial proposal. Please note that if selected, the contract can
be a supporting document to obtain entry visa (if necessary). UNICEF will be unable to secure
travel visas. Any applicable per diems should be included as part of the lump sum price proposal.
11. Desired qualifications
Given the broad scope, it is expected that the evaluation will be carried out by an institution or a
team of technical specialists, overseen by the Evaluation Team Leader. The evaluation team will
consist of at least one international expert as the Evaluation Team Leader, as well as two WASH
experts preferably with Arabic speaking skills. Any other capacities, as deemed required for the
evaluation such as an expert on Value for Money should be proposed by the evaluation team. It is
optional to include more than one expert to ensure that all aspects of water, sanitation and
hygiene (WASH) are adequately covered.
11.1 Team leader
101
- Desirable 15 years of experience in conducting evaluation with proven experience in leading
evaluations of similar scope, such as WASH evaluations
- Demonstrated experience undertaking comprehensive programme evaluations in
emergency contexts;
- Understanding of Cluster/Sector Coordination. Experience in supporting or working with
clusters and/or sectors an asset;
- Experience in working in refugee/IDP camps;
- Proven experience in projects related to gender equality and human rights, including child
rights;
- Proven experience in institutional and policy reviews in the field of water and sanitation;
- Excellent writing skills;
- Fluency in English;
- Preferable to have experience working in the Middle East and North African region;
- Knowledge of Arabic a considered advantage.
11.2 Two WASH Experts
- At least five years of experience implementing water supply interventions in development or
emergency contexts (at least one expert should have this level of experience);
- At least five years of experience implementing sanitation interventions in development or
emergency contexts (at least one expert should have this level of experience);
- At least two years of experience implementing hygiene promotion/community mobilisation
interventions in development or emergency contexts (at least one expert should have this level of
experience);
- Experiences in evaluating WASH programmes an advantage;
- Preferable to have experience working in the Middle East and North African region;
- Fluency in Arabic would be a considered advantage
If the team members do not speak Arabic, then arrangements should be made by the successful
bidder to ensure translation/interpretation services are provided, and are included in the costs in
the financial proposal
12. Estimated duration of the contract and payment schedules
The contract is expected to commence on 02.OCTOBER.2017 and will be undertaken over a
maximum of seven months, and should be completed by 09.MAY.2018.
Payment is contingent upon approval by the UNICEF Evaluation Manager and will be made in four
instalments. The following payment schedule is proposed:
- 30% of the total contract value will be paid upon acceptance of the Inception Report
with Project Delivery Plan;
- 40% of the total contract value will be paid upon
acceptance of the Draft Evaluation Report;
- 30% of the total
contract value will be paid upon finalisation and acceptance of the final evaluation report
and two PowerPoint
presentations (one to the Jordan Country Office and a second one for key sector partners) # the
presentations will take place in the UNICEF Jordan Country Office.
13. Roles and responsibilities in the evaluation process
13.1 The Evaluation Manager
- Manage the evaluation process throughout the evaluation (design, implementation and
dissemination and coordination of its follow
up);
- Convene the evaluation reference group meetings;
- Facilitate the participation of those involved in the evaluation design;
- Coordinate the selection and recruitment of the evaluation team;
- Safeguard the independence of the exercise and ensure the evaluation products meet
quality standards;
- Connect the evaluation team with the wider programme unit, senior management and key
102
evaluation stakeholders, and ensure a fully inclusive and transparent approach to the evaluation;
- Facilitating the evaluation team#s access to all information and documentation relevant to
the intervention, as well as to key actors and informants who should participate in interviews, focus
groups or other information-gathering methods;
- Provide the evaluators with overall guidance as well as with administrative support;
- Oversee progress and conduct of the evaluation, the quality of the process and the
products;
- Approve the deliverables and evaluate the consultant#s/team#s work in consultation with
Evaluation reference group and will process the payments after submission of the deliverables that
respond to the quality standards;
-Take responsibility for disseminating and learning across evaluations on the various programme
areas as well as the liaison with the National Steering Committee;
- Disseminate the results of the evaluation.
13.2 Evaluation Reference Group (ERG)
The ERG will comprise the representatives of the major stakeholders including Evaluation
Manager, Chief of the WASH Section, Planning, Monitoring and Evaluation Officer, and a
representative from the Ministry of Water and Irrigation. The Regional WASH Advisor and the
Regional Monitoring and Evaluation Advisor will also provide a quality assurance of ToR,
inception report, draft and final evaluation report and participate at ERG meetings as necessary.
The ERG will:
- Provide clear specific advice and support to the evaluation manager and the evaluation
team throughout the whole evaluation process;
- Review the ToR, inception report and draft evaluation report and ensure final draft meets
the UNICEF-Adapted UNEG Evaluatio
103
Reports Standards; - Review and provide comments and feedback on the quality of the evaluation process as well as on the evaluation products (comments and suggestions on the TOR, draft reports, final report of the evaluation).
13.3 The Evaluation Team
The evaluation team will report to Evaluation Manager and conduct the evaluation by fulfilling the
contractual arrangements in line with the TOR, UNEG/OECD norms and standards and ethical
guidelines. This includes developing of an evaluation plan as part of the inception report, drafting
and finalising the final report and other deliverables, and briefing the commissioner on the
progress and key findings and recommendations, as needed.
14. Performance indicators for evaluation of results
Proposed timelines for completion of activities are met and deliverables submitted on time with
good quality and as per the standards described in the TORs as well as UNICEF/UNEG global
standards. Overall performance at the end of the contract will be evaluated against the following
criteria: timeliness, responsibility, initiative, communication, and quality of the products delivered.
15. Frequency of performance reviews
Performance review will be conducted after completion of each deliverable. Bi-weekly meeting or
meeting at a frequency mutually agreed will be held between the evaluation team and managers.
Equity, gender, human rights, including child rights
The evaluation process will need to follow -but not limited to- guidelines and procedures on equity,
gender and human/child rights made by UNICEF and the United Nations found in the Zip File
"Annex 1: Equity, gender, human rights, key DOCs".
Main Instruments or policies on human rights, including child rights and gender equality, that should
guide the evaluation process are:
- Child Rights and International Legal Framework, https://www.unicef.org/tdad/index_56373.html
- Convention on the Rights of the Child (CRC), convention and subsequent comments,
guidelines, declarations: https://www.unicef.org/crc/
- Sustainable Development Goal (SDG), namely SDG 5 "Achieve Gender Equality and
Empower All Women and Girls",
https://sustainabledevelopment.un.org/post2015/transformingourworld, and
http://www.unwomen.org/en/news/in-focus/women-and-the-sdgs/sdg-5-gender-equality
- Transforming our world: the 2030 Agenda for Sustainable Development,
http://www.un.org/ga/search/view_doc.asp?symbol=A/RES/70/1&Lang=E
- UNICEF strategic 2014-2017; https://www.uniceforg/strategicplan/index_68123.html
- CEDAW, http://www.un.org/womenwatch/daw/cedaw/
- UNICEF Gender Action Plan 2014-2017
- Security Council Resolution 2122 on Sexual Violence in Armed Conflict (2013),
http://unscr.com/en/resolutions/doc/2122
- International Conference on Population and Development (1994), http://www.unfpa.org/icpd
- Security Council Resolution 1325 on Women and Peace and Security (2000),
http://www.un.org/womenwatch/osagi/wps/
Ethical principles and premises of the evaluation
The evaluation process will need to adhere to the United Nations evaluation norms and standards
(https://www.unicef.org/supply/files/ATTACHMENT_IV-
UNICEF_Procedure_for_Ethical_Standards.PDF), and the UNICEF ethical standards found in the
Zip File "Annex 2: UNICEF Ethical Standards". Inception report, data collection tools, and final
report will be reviewed on ethics by Institutional Review Board and need to be cleared.
104
Annex B: Anonymized List of Interviews Code GENDER ORGANISATION Role Date
01 Male Unicef Jordan Management July 24
02 Female Unicef Jordan Technical Specialist July 24
03 Female Unicef Jordan Technical Specialist July 25
04 Male Unicef Jordan Technical Officer July 25
05 Male Unicef Jordan Information
Management Officer July 25
06 Male Mercy Corps Technical Specialist July 25
07 Male Mercy Corps Technical Specialist July 25
08 Male UNICEF Jordan Technical Specialist July 26
09 Male Unicef Jordan up to
Sept. 2017 Technical Specialist July 26
10 Male Oxfam Jordan Management July 26
11 Male Oxfam Jordan Project Leader July 26
12 Female Norwegian Refugee
Council Education Specialist July 29
13 Male Norwegian Refugee
Council Project Manager July 29
14 Female ACTED Management July 29
15 Male UNICEF Monitoring and
evaluation July 30
16 Female UNICEF Management July 30
17 Male Ministry of Water and
Irrigation Official July 30
18
Male
Water Authority of
Jordan Official July 30
19 Male World Vision Project Manager July 30
20 Male UNICEF Quality Assurance July 31
21 Female ACF Management July 31
22
Male ACF Technical Officer July 31
23 Male Unicef Ruwaished,
Jordan Teachnical Officer July 31
105
24 - 27 4 Males Contractor – four
participants July 31
28 Male UNICEF Management July 31
29 Male Kfw Management Aug 1
30 Male Unicef Regional
Office, Amman Technical Advisor Aug 1
31 Male US State
Department/BPRM Official August 1
32 - 34
2 males
and 1
female
Future Pioneers (local
NGO)
Management
Monitoring and
evaluation officer
Technical Specialist
Aug 2
35 Female Unicef Officer Aug 5
36 Female Unicef Technical Officer Aug 5
37 Male Unicef Technical Officer Aug 5
38 - 43
2 males
and 4
females
Oxfam Project Staff Aug 5
44 - 46 3 males Oxfam Project Staff Aug 5
47 Male UNHCR Management Aug 6
48 Male Unicef Technical Specialist Aug 6
49 Male UNICEF Management August 6
50 Male UNOPS Management Aug 24
51 Female ACTED Management
52 Male Mercy Corps
Management
Sept 4
Annex C: List of Sites Visited
1. Irbid Governorate Host Community programme
2. Ramtha
3. ITS site 702
4. ITS site 709
5. Ruyaished Host Community
6. Rukban Settlement
7. Za’atari Camp
106
8. Azraq Camp
9. King Abdullah Park Camp
10. WIS Schools Visit in Irbid
Annex D: Documents Consulted
Camp Assessments
UNICEF_REACH_Comprehensive Child Focused Assessment Azraq Refugee Camp
UNICEF_REACH_Comprehensive Child Focused Assessment Za'atari Refugee Camp
107
Camp Contracts
6.2.2 Aref & Mohamed Lafi Contract for repair septic tanks
f6.2.3 Bab El-Amood Contracts
6.2.7 IMDAD
6.2.8 Irbid District Electricity Company contract 43141693
t6.2.9 Open Hands - Monitoring of Refugee Camp wastewater Disposal Contract
6.2.10 Team Connect - Contract # 43143909
6.2.12 Safe Gaz PO # 43133530
6.2.13 Royal Scientific Society contract #43132050
6.2.18 GIS Contract - Ghazi Gsouss
6.2.21 EEC
6.2.25 Site Group
6.2.34 Loyality
6.2.35 New Day
6.2.37 Engicon
6.2.38 ACE
6.2.39 GITEC
6.2.40 ITC
6.2.41 Aquatreat
6.2.42 Health Bureau
6.2.44 Nimer al lawzi
6.2.47 Desluding Zaatari 2015-16
6.2.51 Specalized company for trade service - Hygien Kits - LLTA 42407750
6.2.53Bakhos
6.2.59 JEA
6.2.60 Al Mehwar
6.2.61 GAMA
6.2.64 Al Faris
6.2.65 Al Manaseer
Field Trip Reports
ACF Report - March2015.docx
programme visit and trip report - ACF-Dec 2016-0&M.
programme visit and trip report - ACF-Dec 2016-soft
programme visit and trip report - ACF-dec-O&M.docx
108
programme visit and trip report - ACF-July 2017-0&M
programme visit and trip report - ACF-June 2017-0&M
programme visit and trip report - ACF-June-V2.
programme visit and trip report - ACF-Nov-GTD
programme visit and trip report - ACF-Oct-GHWD
Trip report - Q4 - ACF1.
Trip report - Q4 - ACF2
Trip report - Q4 - ACF3
15 06 ACTED Prog Visit.pdf
16 01 20 Za programme visit and trip report.docx
16 06 08 Prog Review visit to Azraq.pdf
16 07 27 programme visit and trip report-ACTED-28.07.2016
Prog Review visit to Azraq
17 0411 Prog Review visit to Zaatari
17 06 20 Prog Review visit to Azraq.pdf 83 ACTED Report April 2017
Jen Reports
Q1 2015
Q1 2015
Q1 2016
Q1 2017
Q2 2015
Q2 2015
Q2 2016
Q2 201
Q3 2014
Q3 2014
Q3 2015
Q3 2015
Q3 2016
Q4 2014
Q4 2014
Q4 2015
Q4 2016
Oxfam Reports
programme visit_trip report-0xfam20140917
109
programme visit_trip-report-Oxfam20141231
OXFAM-Q1-Prog-Visit_Trip-Report-20150323
Notes from programme Visit Meeting_Oxfam_290615
OXFAM-Q2-Prog-visit-trip-report-20150629
Notes from programme Visit Meeting.Oxfam 300915
OXFAM-Q3-Prog-visit-trip-report-20150930
OXFAM-Q4-Prog-visit-trip-report-20151208
World Vision
Prog-Visit_Trip-Report-World_Vision-20141109
programme visit and trip report - World.Vision and IMC-November -SAL
programme visit and trip report - World.Vision-December -SAL
programme visit and trip report - World_Vision-Feb-SAL
programme visit and trip report - World.Vision-January -SAL
programme visit and trip report - World_Vision-Mar-SAL
programme visit and trip report -WVI-july-SAL
programme visit and trip report -WVI-June-SAL
programme visit and trip report -WVI-sep
Trip report - Q2 - WVI
Trip report - Q3 – WVI
Program Cooperative Agreements
WV PCA19-13 Prog visit report Azraq - 9 Nov 2014
WV PCA 19-13 Prog visit report Azraq - 9 Nov 2014
WV PCA 19-13 Prog visit report Azraq -11 Sep 2014
WV spot check 2014 0119.pdf
WVI PCA 19-13 Prog visit report Azraq -19 Feb 2015
WVI PCA 19-13 Prog visit report Azraq -19 Feb 2015
lWVI PCA 19-13 Prog visit report Azraq - 25 Mar 2015
WVI PCA 19-13 Prog visit report Azraq - 25 Mar 2015
WVI PCA 19-13 Prog visit report Azraq V5 & J stations - 25 Jan 2015
WVI PCA 19-13 Prog visit report Azraq V5 & J stations - 25 Jan 2015
l§J WVI PCA 19-13 Prog visit report Azraq-23 June 2015
L WVI PCA 19-13 Prog visit report Azraq-23 June 2015
KAP Surveys
2017 AZRAQ Camp Knowledge Attitudes and Practices Survey Report Final.
110
KAP 2013 Final Report
KAP 2014 Final Report 15.03.15
KAP 2015 Final Report 21.03.16
KAP 2015 Executive Summary 21.03.16
ACF
ACF PCA14-PD 1-2015-Azraq
ACF PCA 26-14-Houshold-(Ajloun-lrbid)
ACF-PCA-14-PD 02 Irbid
ACTED
ACTED PCA 02 PD 02 2017
ACTED PCA 02-PD 01-2015
ACTED PCA 23-13
JEN
JEN PCA 01-PD 01-2015
JEN PCA 01-PD 03-2015 (amend to PCA 25-14)
JEN PCA 01-PD 04 - 2017 Za'tari and Mafraq
JEN PCA 9-13
JEN PCA 18-15
JEN PCA 25-14
JEN PCA 34-13
Oxfam
OXFAM PCA 13-PD 1-2015
OXFAM PCA 13-PD 2-2016
OXFAM PCA 13-PD 3-2017
OXFAM PCA 32-13
SRIPCA 33-13
World Vision (WV) May 2013
Factsheets on Syrian Refugees October 2012 - December 2017
Monthly, Quarterly, and Annual Reports from Partners
Partnership Agreements, Budgets, and Ammendments, 2012 - 2017
Other Documents
Sitrep Reports December 2012 - August 2018
Third Party desludging, water and solid waste Monitoring data (UNOPS)
Water monitoring data, Yarmouk; 2015-2017
Jordan Nationwide Assessment in Public Schools for Strategic Planning
111
Running on Empty: The situation of Syrian Children in Host Communities in Jordan
ITS Field Visit Reports
Host Community Field Visit Reports
AZRAQ Camp Knowledge Attitudes and Practices Survey Report 2016_HJ_HP unicef
comment
KAP Report 2017(non-public) REACH_JOR_factsheet_CCFA_Feb2017_WASH
REACH_JOR_UNICEF Zaatari Camp Assessments List_Oct 2017
REACH_UNICEFJTS_MS_AUGUST2014_FINAL
UNICEF_REACH_Comprehensive Child Focused Assessment Azraq Refugee
Camp_J.._.pdf p UNICEF_REACH_Comprehensive Child Focused Assessment Za'atari
Refugee Camp UNICEF_REACH_Factsheet_Zaatari_CCFA_WASH
2017 UNICEF Jordan Audit
Access to water and tensions 2014
ACTED Gender Task Force
UNICEF Jordan Annual Reports 2012 - 2017
Controling Groundwater Thesis.html L' Coordination Study.pdf & Cyber City Map
EVALUATION_OF_THE_WASH_PROGRAMME_WITHIN_THE_UNICEF
COUNTRY_PROGRAMME_IN_LEBANON_2013-2015_2017-00
Manual for maintenance cleaning and hygiene promotion in schools - English
Explanation of Gama Contract
Groundwater Jordan
Handicap International Advice
Market Systems 2013
History of Za'atari
History of the Palestinian Refugees in Jordan
IT Humanitarian Funding Update_GHO_31DEC2017
IMF Jordan Report 2017
Syrian refugees in Jordan, A Protection overview January 2018
C Jordan, 2012 Article IV Consultation; IMF Country Report 12_119; March 23, 2012
Jordan_Water_Strategy_2008-2022
Jordanian attitudes toward refugees 2012
MoWI Annual Report 2016
National Water Strategy( 2016-2025)-25.2.2016
Equity, scalability and sustainability in UNICEF WASH programming: A thematic meta-
analysis of UNICEF’s WASH evaluations 2007-2015
Water, Sanitation, and Hygiene (WASH) Cluster Coordination Handbook; January 2009
112
WASH Sector Coordination; Minutes of weekly Za’atari camp WASH sector coordination
group.
Annex E: Data Collection Tools
Key Informant and Key Stakeholder Interview Guide
Introduction for the Interviewer ISG is conducting an evaluation of a programme managed by UNICEF. The objective of the research
is to understand how UNICEF has helped people get better access to water and sanitary facilities, and
the results of their hygiene activities where you live. ISG will conduct its research via key informant
interviews, key stakeholder interviews, and focus group discussions. The interviews will include male
and female stakeholders to ensure differences in uses and needs are captured. This interview guide
serves as a manual for data collectors conducting the key informant interview portion of the research.
ISG’s researchers will conduct face to face interviews WASH stakeholders. While this guide is
intended to assist the interviewer in setting key areas for research and orienting the discussions, the
interview should happen as a conversation, allowing the interviewer to ask follow up questions that
may lead to discovering new and important information.
INSTRUCTIONS FOR THE INTERVIEWER
• Follow the informed consent process below.
• You should ensure that you cover all the major issues noted in this guide. However, some
questions may elicit long discussions and others short answers depending on the relevance of
individual topics for discussion to individual respondents. Exploring the most relevant areas
(and others that may be important to the respondent, but not necessarily covered in this guide)
is more important than covering every question in the guide.
• Ensure that women participate in the interview, especially if the interview is taking place in a
group.
• The WASH programme’s activities operate in a dynamic and complex environment. It is
likely that there are key issues that the author of this guide did not anticipate. The interviewer
should feel free to ask questions that the interviewer feels are fruitful whether or not they are
included in the guide.
• Please take notes in a separate document.
113
• Make sure to record your notes at the end of each day. Too much time between the interview
and recording notes can lead to loss of salient information that will have a negative impact on
the rest of the research.
Informed Consent Process This section outlines the consent process for the UNICEF Jordan WASH programme Evaluation.
While this document provides illustrative language, it is important to tailor both language and
presentation to each stakeholder to ensure understanding of the study and his/her involvement.
The people that the evaluation team will interview have the right to know that they are participating in
a study, that the evaluation team is collecting their opinions as part of a research project, and to be
told about the purpose of the research and the possible risks and benefits of the project and their
participation. The interviewer will introduce himself/herself and explain the purpose of the interview
as detailed in the Informed Consent Form below. Following the explanation, the researcher will ask
the respondent to sign the Informed Consent Form:
INTERVIEWER INTRODUCTION
Hello! Thank you for participating in this interview. My name is _________ and I am working on a
project for International Solutions Group, a research organization based in the United States. ISG is
conducting research on behalf of UNICEF.
The purpose of our research is to assess the effectiveness of UNICEF’s Water, Sanitation, and
Hygiene related activities where you live. You may not know about UNICEF’s specific activities.
However, UNICEF is interested in understanding if the program has achieved its intended goals,
which include improved access to water and sanitation and improved hygiene for people who live in
your area. I will ask you questions about your life and experience in relation to UNICEF’s water,
sanitation, and hygiene promotion activities. Although you may not directly benefit from participating
in this study, we hope that the findings of this study will contribute to the improving UNICEF’s
Water, Sanitation, and Hygiene programme’s for people like you.
The interview today will last about an hour. During the interview, I’ll take notes so I can remember
what you said.
I want to mention a few important points before we start.
• There are no ‘right’ or ‘wrong’ answers to any of the questions. We are interested in hearing
your opinions, whatever they are.
• Your participation in this interview is voluntary. If you don’t want to answer any question,
you don’t have to. The responses that you provide are confidential: we won’t mention your
name in any report we write and we will never connect your name with anything you say.
Your responses will be combined with responses from other people.
Do you have any questions before we begin?
Now that I’ve told you about the topics we’ll discuss and answered your questions, are you
comfortable proceeding with the interview?
1. IF NO: ASK IF YOU COULD PROVIDE ANY OTHER INFORMATION THAT WOULD
MAKE THE RESPONDENT MORE COMFORTABLE. IF RESPONDENT PREFERS NOT
TO CONTINUE, TERMINATE INTERVIEW.
2. IF YES: CONTINUE TO THE NEXT SECTION.
114
Household Level Questionnaire
Basic Information [NOTE TO INTERVIEWER: Fill out before the interview begins.]
1) Consent form number:
2) Date of Interview:
3) Gender (if apparent):
4) Primary and Secondary location of interview:
5) Interview Number:
Water Supply 6) Is water supply sufficient for your needs? [If no Goto 6a]
a) Which needs are unmet [for example: drinking, cleaning, latrine, other]
b) How much more water you would need?
7) Where do you most often access water [if outside the home, Goto 7a]
a) How often do you fetch water?
b) How far do you go?
c) How long does it take?
d) Are there alternative sources from where you fetch or buy water?
8) Do you buy additional water from other providers? [If yes, Goto 8a. If no, Goto 9]
c) Why do you buy additional water? Is it expensive?
d) Does the water have good quality?
9) Have you ever experienced problems accessing water? [if yes, Goto 9a]
a) What caused the problem?
b) How was the problem solved?
c) Has it reoccurred?
Water Quality 10) Is the quality of the water sufficient for all of your needs [for example: drinking, cleaning, latrine,
other]? [if no, Goto 10a]
a) For which activities is in insufficient (for example: drinking, cleaning, latrine, other)?
11) Have you ever become sick because of the water? [If yes, Goto 11a. If no, Goto 12]
a) What was the illness?
b) Was it just one person or many people in your family?
c) How often did it happen?
115
d) How was it treated?
Facilities 12) Does your family have access to a latrine and bathing facility?
a) Is the bathing facility and latrine private or communal?
b) Is it satisfactory?
c) Are there any improvements you would make to it (if communal, is distance reasonable, is it
safe, is it lighted sufficiently, does it lock)?
13) Is there enough privacy for women and children?
a) Are sanitation and bathing facilities segregated by gender?
b) Is there enough distance between female and male WASH facilities?
14) [if latrines are inside in the house Goto 14a; if outside of the house Goto 15]
a) Does the household have access to latrines/septic tanks or mainline sewerage? [if septic tanks,
go to 14b]
b) Is there any service provider that comes to empty the latrines/septic tanks? Is the service
timely and reliable?
e) Have you ever paid for the maintenance of your WASH facilities? If so, whom did you pay?
Is it the money worth the service?
15) Have you ever experienced problems accessing latrines or bathing facilities? [if yes, Goto 15a]
a) Have they ever been closed or not usable for short or long periods of time? If so, why? Was
the problem solved? How often did it happen?
16) If people generally have trouble with water or sanitary facility access, or the facilities don’t work,
what do they do?
17) Have you or any other member of your community experienced any threat while using the WASH
facilities? [if yes, Goto 17a]
a) Was it at day or night?
b) What do you think can be done to avoid similar situations?
c) Has security improved since the event?
18) Does the camp/community have latrines and bathing facilities accessible to people with limited
mobility? Where are they? (follow observation)
a) Are they sufficient? If not, what could be done differently?
19) Is there any area of this camp/community that has a harder time accessing WASH facilities?
a) If yes, which areas? Why?
20) Are there any vulnerable people that have a more difficult time accessing WASH facilities?
a) If no, which areas? Why?
Operations and Maintenance 21) Who cleans the WASH facilities that you use? Have you ever cleaned them? Is there a system in
place to maintain the facilities clean?
22) Is there any committee in charge for the maintenance of the water or sanitation facilities or
infrastructure?
23) Is there anyone in your community or immediate circle able to fix minor issues with the WASH
facilities? [if yes, Goto 23a]
a) If so, is this person paid?
b) Does he/she do a good job?
c) Is he/she reliable?
24) Do you know if this person was trained by UNICEF or by other organizations? Which
organization?
Final Questions 25) Were you consulted about your preferences regarding water and sanitation services? If so, how?
a) Were the different options clearly explained to you?
116
b) Did you have the opportunity to provide feedback on the services provided? If so, how? Did
the feedback you provided have any outcome? (Were the services
adjusted/corrected/improved according to your feedback?)
26) Have you noticed major changes related to access to water in your camp since you have been
here? [If yes, Goto 26a]
a) If so, what kind of changes?
b) How did they happen?
c) Who made the changes?
d) Did the changes improve your situation?
27) Is there anything else that you would like us to know about Water, Sanitation, or Hygiene where
you live?
Focus Group Discussion Guide
Introduction for the Focus Group Leader ISG is conducting research on behalf of UNICEF. The objective of the research is to understand the
impact of the UNICEF’s Water, Sanitation, and Hygiene programme. Particularly, we want to know
how the program has worked to meet you expectations, where it has succeeded, and where it could
improve. ISG’s research will be conducted via a survey, focus group discussions, and key informant,
key stakeholder interviews. This focus group guide serves as a manual for the focus group leader and
note taker conducting focus groups.
ISG will conduct focus groups with residence of camps, ITSs, host communities and stakeholders in
schools:
1. The degree to which project activities served the needs of beneficiaries.
2. The attitude of community members toward ownership and maintenance of infrastructure and
asset building projects
3. The effectiveness of program in achieving water, sanitation, and hygiene results for all
stakeholders, including vulnerable groups.
4. The sustainability of the programme’s activities and facilities.
The purpose of these focus groups is to present broad issues and to let participants generate ideas and
build on each other’s ideas. The focus group leader’s goal is to encourage participation and ask
questions that build on participants thoughts. This guide provides follow up questions that the leader
may use. The leader may ask specific participants to compare their opinion or thoughts to those of
other participants through prods such as “Participant B, how does your experience compare to the
experience of participant A that we just heard?” or “Participant C, what would you add to participant
B’s comments?” The interaction that the focus group leader encourages will provide depth to the
information we are gathering through the survey and interviews.
Preparation
• Make sure that informed consent forms are available for each participatng.
• Make sure the room is set up with flip charts or whatever materials you need.
117
• Set up refreshments if appropriate.
• If available, set up small gifts to thank the focus group participants.
Agenda
Introduction, Ground Rules, Logistics and Sign-in Sheet
[FOCUS GROUP LEADER: Introduce yourself and the note taker.]
Thank you all for participating in this focus group. My name is _________ and I will lead the group,
and this is ____________ who will take notes. Our discussion will last no more than an hour and a
half. Before we get started, let me mention that there are drinks and refreshments available (point
them out), and the bathroom is (indicate location of bathroom). Please feel free to get up and move
around or help yourself to refreshments during the discussion.
The purpose of this focus group is to understand the impact of the UNICEF Water, Sanitation, and
Hygiene programme in your community (or school). Particularly, we’d like to understand:
• How project has served your community’s needs.
• Your opinion of the project’s facilities and services.
• What you like best about the project.
• What you would most like to improve about the programme
Infront of you is an informed consent form. This form let’s you know about the purpose of the
research and the types of questions we will ask. Please read the form and let me know if you have any
questions. If you agree to participate in this focus group, please sign the form at the bottom where
indicated.
I wanted to mention a few important points before we start.
• The purpose is for us to learn from you, both your positive and negative experiences and
opinions.
• We’re gathering information, not trying to achieve consensus. Please contribute your
experience and opinion. It’s ok for there to be disagreement or different ideas.
We have a few ground rules that we hope you’ll agree with:
• Please participate. Your contribution is valuable and will improve the program’s activities in
the future.
• Please agree to keep the information provided in the focus group confidential. We won’t
discuss any sensitive topics. Still, people may share ideas that they’d rather not be discussed
privately.
• Stay with the group and please don’t have side conversations.
• Turn off cell phones if possible.
Are there any questions before we get started?
FOCUS GROUP LEADER: Go around the table and ask people to introduce themselves.
118
Relevance 1. Do you think the interventions responded to your basic needs related to water, sanitation and
personal hygiene?
a. What’s the most important way your community has improved in terms of water
access, sanitation, or hygiene practices?
b. What needs does the community have that aren’t being addressed? Why do you think
they haven’t been addressed?
Effectiveness 2. If you could improve your access to water or sanitation facilities, what would you improve?
3. Is there any group in your community that have not been reached or that has not benefitted
from UNICEF like the others?
4. What do people do if they have trouble accessing water, or the sanitary facilities don’t work?
a. What authorities are responsible? Are they responsive?
WATER
5. How has access to water improved for this community recently? What is the most important
improvement?
6. What changes would you make to the community’s ability to access water?
SANITATION
7. Do you use toilets or latrines?
a. Are they public or private?
b. How are they kept clean?
c. Do they work all the time?
d. Are they safe?
8. What changes would you make to sanitary facilities?
HYGIENE
9. Has the hygiene knowledge or practices changed in the community in the last five years?
How?
a. Do people ever get
SUSTAINABILITY
10. What would happen if UNICEF would stop its activities tomorrow? Do you think your
community/school would be able to maintain the water and sanitation infrastructures as they
are?
11. Is the wastewater and solid waste handled properly? Do you have any concern for the
environment?
119
Annex F: Team Description
Mr. Jason Wares; Team Leader
Role: Mr. Wares led the team l, organized the completion of deliverables, conducted data
collection activities, and arranged team logistics. He also interfaced with the UNICEF at all
stages of the assignment.
Mr. Luca Palazotto, Mr. David Robbins, and Mr. Niall Roche; Co-Evaluators
Role: The Co-Evaluators will assist the Team Leader during the tools development,
sampling, desk review, data collection, and reporting aspects. Dr. Bassam will lead in the
translation of tools and reporting, as he will serve as the Country Expert for this assignment.
interdisciplinary research consultants (id:rc)
Role: id:rc organized logistics for the evaluation in Jordan and provided local WASH and
monitoring and evaluation consultants. Id:rc participated in data collection and analysis, and
ensured that the evaluation was grounded in the Jordanian context.
120
Annex G: Evaluation Matrix
EQ1: To what extent were affected people consulted on their preferences with regards to water and sanitation service
options and level of services? Were interventions appropriate in terms of meeting their basic needs? Was there a
feedback loop and monitoring system in place for reflecting the learning and evidence to improve programming,
especially for the humanitarian response?
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
JC 1.1: The needs of the
population, in particular
those of vulnerable groups,
were well considered during
the programming process
and adjusted during
programme implementation.
- Evidence of an process to
identify needs prior to the
design and implementation
of interventions.
- The extent to which chosen
interventions were consisted
with the needs that
assessments identified.
- The choice of interventions
prioritized activities that
would serve vulnerable
groups.
- The existence of a feedback
loop and monitoring system
that evolved with the
programme.
- Starting at WASH Working
Group paper
- WASH Cluster Coordination
Handbook
- Jordan Regional Response
Plan 5
- 2015 KAP survey for
Za’atari
- WASH Sector Coordination;
Minutes of weekly Za’atari
camp WASH sector
coordination group. Sunday
24 November 2013
- 2017 Azraq KAP Survey
Report
- 2014 WASH Sector Gender
Analysis Document
- Specific data analysis
- World Vision Staff Interview
30 July 2018
- UNICEF Staff interview – 26
July 2018
- Focus Gropu Discussion
number 1 through 4– Azraq
- Official Interview – MoWI
- Mercy Corps Staff Interview
– 4 September 2018
- On site assessments and
interviews in Za’atari, Azraq,
KAP, and Rukban.
- Interview with Gama
Engineering
- Host community Interviews
in Irbid and Ruwashaed
121
- Evidence that UNICEF used
evidence for learning.
- REACH 2014 ITS
Assessment
- World Vision PCS 19-13
- ACF 2017 Azraq KAP
Survey Report
- Rukban emergency plan
- UNOPS Staff Interview
- UNICEF Jordan Country
Programme Document
- UNICEF WASH Actions in
Humanitarian Situations:
Synthesis of Evaluations,
2010 - 2016
In the intial phase of the Programme, UNICEF determined refugee’s WASH needs through talks with partners such as The WAJ,
Mercy Corps, Acted, and Relief International, and based on the assessments and information that they’d gathered. The tactics that
they employed took into consideration the Convention of the Rights of the Child (CRC) and the Convention on the Elimination of
All Forms of Discrimination against Women (CEDAW). Through these means, UNICEF created estimates of the best way to
provide WASH services for the newly arriving refugees. UNICEF also consulted humanitarianresponse.info for guidance on issues
such as accountability and feedback mechanisms.
The evaluation team did not find evidence that UNICEF consulted internal resources for guidance on WASH response in an
emergency situation.
Another consequence of the need for quick decision making was that, in the initial emergency response phase, the camp’s effect on
Jordanians living near the camp and in other host communities with high concentrations of refugees was not taken into
consideration adequately.
In later phases, UNICEF used assessments and consultations to understand the needs of beneficiaries inside the camps. Azraq
residents benefited from lessons learned in the establishment of WASH services in Za’atari and contributed their preferences to
WASH set up.
After the emergency response phase, UNICEF worked with its partners to adapt WASH facilities and services to the observed needs
and behavior of camp inhabitants.
122
Time pressure and the desperation of the Settlements’ inhabitants made consulting them about their preferences impractical.
UNICEF provided water and sanitation services to respond to the settlements’ exigencies. They moved quickly and effectively,
with clear goals, such as ensuring that each person had at least 15 liters of water per day. The WASH programme also recognized
that its intervention would have an impact on the larger Ruwaished host community, so they undertook to rehabilitate sanitation
facilities there.
The WASH programme has maintained weekly updates since the initiation of work. The updates monitor operations and needs. In
2017, surveys and assessments became part of the monitoring mechanism. These tools ensure that WASH programme activities
remain relevant and effective.
UNICEF’s improvement of the wells and infrastructure in host communities is significantly important. However, the degree to
which it addresses the needs generated by the refugee crisis and the needs of the most vulnerable is uncertain.
UNICEF reports that they have served 114 ITSs, but the total number of ITSs is unknown, and the level of that service is unclear.
The evaluation team could not find evidence that The WASH programme is structured as a learning organization. It does not appear
to have taken guidance from UNICEF’s experience in WASH programming at the outset of the programme. While it has certainly
made marginal improvements to its activities in terms of efficiency and responsiveness to people’s needs, it did not build
monitoring networks or feedback loops that fed into a system that would allow for learning or informing strategic management
EQ2: To what extent did the interventions consider the needs and the priorities of the MWI and were they coherent with
global references and the regional and national response to the Syria Crisis?
123
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
JC 2.1: Degree to which
UNICEF took government
policy and strategy into
consideration when
determining programme
priorities.
JC2.2: Degree to which the
MWI and WAJ were
satisfied that UNICEF
responded to GoJ needs
- Evidence that MWI priorities
were considered in
intervention design and
implementation.
- Government officials feel
that UNICEF’s interventions
were in line with policy and
regulations.
- National Water Strategy
2016 – 2025
- Jordan Response Plans
- Senior Staff Interview –
WAJ 30 July 2018
- Senior Staff Interview –
MoWI 30 July 2018
- Specific focus groups
UNICEF’s work in WASH is in line with the priorities of the National Water Strategy, for which The key reference document is the
National Water Strategy 2016 – 2025. UNICEF is focused on the sustainable operation of boreholes to supply water and create
resilience. UNICEF also recognizes the government’s concern that Jordan’s northern governorates have not benefited from the
government’s efforts to increase water supply (Ministry of Water and Irrigation, 2016). In response, UNICEF has focused much of
its host community interventions on improving WASH in the northern governorates.
Both the MoWI and WAJ reported in interviews that the GoJ-UNICEF partnership is successful. The WAJ praised UNICEF’s
success at sector coordination despite budget cuts140. The MoWI appreciated UNICEF’s role in coordinating the sector141.
140 WAJ Senior Staff Interview – 30 July 2018
141 MoWI Senior Staff Interview – 30 July 2018
124
Interviews at MoWI and WAJ demonstrate the Government of Jordan’s struggles with its strategy and UNICEF’s response. Senior
staff at the MoWI praised UNICEF’s responsiveness inside the camps, but sees that project’s host communities are more dependent
on available funding. MoWI said that when the WASH programme first took on host community projects, there was a
miscommunication regarding the meaning of the protocol UNICEF established with the MoWI. The protocol included a mandate
and budget for UNICEF to improve infrastructure in governorates. The ministry didn’t realize that the budget was prospective, and
moved those items off of its budget for the year. When fundraising fell short, those governorates were underserved.
EQ3: How responsive was the UNICEF WASH programme over time to changes in the external environment?
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
JC 3.1: UNICEF
demonstrated flexibility in
- Evidence of adjusting
programme design and
- Jordan Annual Reports - Senior Staff Interview – 6
August 2018
125
decision making and the
ability to reorient resources
as the external environment
changes.
operations to changing
conditions
-
- Terms of Reference for Service
Contracting - Rukban.
- Focus group discussions in
Azraq, Za’atari, and KAP
In the initial phase of the programme, the GoJ and INGOs in Jordan prepared for a temporary camp. This preparation included
procuring tents, portable sanitary facilities, and trucked in water supplies. While these interventions were effective, their expense
made them unsustainable. When it became apparent that the need to house refugees was more than temporary, UNICEF devised
solutions that required investment, but would be cheaper and more sustainable to operate in the medium to long-term. These
interventions included WASH Blocks, piped in water from nearby boreholes, and established hygiene campaigns. UNICEF also
made these adjustments in the settlements, establishing water infrastructure that was both resilient enough to withstand difficult
conditions and use, but also reduced operating funding needed for maintenance142. As Senior UNICEF Staff noted, “Transitioning
from humanitarian response to systems and infrastructure…is incredibly challenging, particularly in environments like the berm
(Rukban).143”
One type of transition in which the WASH programme struggles was in implementing host community projects. As an interview
respondent at the MoWI suggested, UNICEF did not originally plan on implementing host community projects, but turned to them
in the face of external pressure from municipalities that felt that refugees were putting stress on local water resources144. The
WASH programme boldly took up the challenge and set high goals for itself such as increasing the percentage of the population in
Jordan that has access to adequate sanitation facilities (2014), or increasing the number of inhabitants with safe water and storage
facilities by 1.4 million (2015)145. While UNICEF missed these targets in each year, it continued to set high targets in this
component. As mentioned in the Funding Environment section of this report, UNICEF blamed shortfalls in fundraising for missing
host community targets in each year of the programme. However, the evaluation team did not find evidence that the WASH
programme considered its funding outlook when planning these targets, or, given the regular shortfall in funding, how it might
better spend these funds. We discuss this topic further in the Effectiveness and Efficiency sections.
142 UNICEF Staff Interviews – 10 October 2017
143 UNICEF Senior Staff Interview – 6 August 2018
144 MoWI Senior Staff Interview – 30 July 2018
145 See results framework, Annex H.
126
EQ4: To what extent did UNICEF’s response achieve its intended outcomes (as stated in UNICEF’s Country
Programme Outcome and Outputs as given in Annex A)? To what extent did UNICEF achieve equity results, especially
for children and women specific interventions?
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
JC 4.1: The degree to which
UNICEF set specific,
measurable and realistic
targets.
JC4.2: The degree to which
the evaluation team could
find evidence to support
UNICEF’s achievement
measurements.
JC4.3 The degree to which
targets were met.
- Indicators based on results
frameworks.
- RAM reports 2013- 2016
- UNICEF Jordan Annual
Reports
- Za’atari Hygiene Working
Group KAP reports
- The Sphere handbook
- Interviews with partner
organizations
- Interviews with UNICEF
staff
- Reviews of PCAs
Component 1
Under this component, the WASH programme achieved high levels of effectiveness for the targeted population across all years.
127
Component 2
As detailed with citations below, the WASH programme consistently struggled to achieve targets under this component. Two issues
created difficulties: 1) funding, 2) lack of strategic planning
As described in the Funding Context section of this report, UNICEF Jordan struggled with funding issues. Maintaining services in
the camps necessarily received first priority for funds at the expense of achieving targets in host communities.
That said, after several years of missing targets, UNICEF doesn’t appear to adjusted its strategy to make it more effective or
realistic. Also, indicators aren’t specific or timebound under this component, so it’s difficult to know what the programme is trying
to achieve. For example, UNICEF’s Outcome 4, Indicator 2 target is that 1.5 million host community residents use improved
drinking water as per Jordanian standards. In its 2015 RAM report, the WASH programme stated that “In Host Communities, 41%
(634,858) of the target population of 1,550,500 benefitted from improved access to water facilities and systems as a result of
rehabilitation of water infrastructure, as well as support to vulnerable households.” In its 2016 RAM report, for the same indicator
the WASH programme reported, “In Host Communities in four governorates (Mafraq, Madaba, Balqa and Irbid), 237,360
additional people were reached in 2016. Since 2013, an estimated 1,482,402 people have improved access to water services
…which equates to 96% of the Country programme target.” It is unclear if the target is that UNICEF serve 1.5 million per year
under this outcome as the 2015 report implies, or 1.5 million cumulative people as the 2016 report implies. It is also important to
consider the improvement. For xample, In Irbid UNICEF made improvements to the water distribution system. This should not
imply that everyone in Irbid had better water services as a result of the intervention. Water services are still very limited, and there
are still serious problems with the systems distribution network, water safety, and availability.
Component 3
Government officials at the MoWI and WAJ reported satisfaction with the WASH programme contribution to planning documents,
strategies, and policies146. The MoWI said that, as head of the WASH Taskforce, UNICEF always has a good plan, and has a good
partnership with UNICEF on technical committees. The Ministry also praised the work done by the consultant that UNICEF
seconded to Ministry, and wished he could have stayed longer.
146 Source: Interviews at MoWI and WAJ
128
The ministry staff interviewed by the evaluation team mentioned that there were some unintended consequences to the work that the
WASH programme has done in host communities. A specific example was the downstream effects of increasing water supply
through a network. Increased water supply means quicker degradation on infrastructure, leading to the Ministry having to replace
equipment sooner than expected. One interview respondent mentioned the opportunity cost of providing water to refugees, in that
the water could have gone to Jordanian workers or industry. He estimated that for each refugee the government spends $650 on
water and $7,700 on indirect costs included lost opportunities.
Component 4
Stakeholders and UNICEF partners felt that UNICEF was effective as WASH sector lead. One contractor that serves Rukban said
“UNICEF staff are super cooperative and available anytime of the day or night”.147
The evaluation team also spoke with two of the WASH programme’s donors, The Bureau of Population, Refugees, and Migration
(BPRM) at the United States State Department, and the German owned development bank Kreditanstalt für Wiederaufbau (KfW).
BPRM commented that UNICEF was effective in leading the sector, and excellent to work with as a donor. BPRM felt that they had
benefited from UNICEF’s guidance in setting their own strategy for addressing the crisis. KfW appreciated the guidance UNICEF
could provide in constructing infrastructure in the camps and assistance in host communities.
As sector lead, UNICEF has led WASH activity coordination in the Camps, including responses to the occasional emergency, such
as the 2015 outbreak of Hepatitis A in Azraq. As part of camp coordination, UNICEF leads a bi-weekly Za’atari coordination
meeting that has been effective in setting standards and avoiding duplication of effort. UNICEF also improved the MoWI’s use of
data to create water management strategies, and led the sector’s WASH in schools initiatives.
One area in which UNICEF could improve its coordination is in strategic planning and communication with partner organizations.
Several partner organizations, particularly INGOs, mentioned that they were surprised by UNICEF’s decisions in the camps, often
147 Source: Interview with Gama Engineering
129
requiring that they make major adjustments on short notice, or conduct work for which they hadn’t planned or budgeted148.
Examples included sudden decisions to hand operations over to other organizations or contractors with short notice, without
guidance on the handover process. Other partners mentioned contracting processes that took a long time only to have a requirement
that initiatives start days after a contract is signed. Decisions such as these, that do not appear as part of a strategically designed
plan, may force risk and costs on to partner organizations. It may also create risks for UNICEF that its monitoring and accounting
systems cannot bear. These risks could also be mitigated by an annual strategic planning process that would set expectations and
guidelines based on clear analysis and reduce last minute decision making149.
EQ5: What were the strengths and weaknesses of the interventions in achieving results in the camps/settlements, ITSs
and host communities on the following:
1. Water (Litre/day, supply, quality, consistency, reliability, accessibility, and equity)?
2. Sanitation (adequacy, appropriateness, accessibility)?
3. Hygiene promotion (appropriateness of the messages, timeliness, behaviour change)?
4. WASH in schools?
5. National level support?
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
148 Interviews with Mercy Corps, World Vision, ACF, and Oxfam Staff
149 The UNICEF Jordan Country Programme Document 2013-2017 mentions a two year rolling workplan which is measured through annual component reviews and a mid-
term review in 2015. The evaluation team did not see evidence that the WaSH Programme had complied with this mandate.
130
JC 5.1: The degree to which
UNICEF met its own and
international standards.
JC 5.2: Beneficiary opinions
on stregths and weaknesses
of interventions.
- International WASH
standards
- WASH in Schools targets vs
accomplishment
- Jordan response plan
- Global WASH Cluster
handbook
- Sphere handbook.
- UNICEF Reports
- Camp Assessments
- Hygiene Promotion Working
Group reports.
- Interviews with Partner Staff
- Interviews with UNICEF
staff
- Site inspections of UNICEF
intervenions
Water
For both Za’atari and Azraq, the interventions were effective. Residents of both camps people at least 35 liters of water per day,
well in excess of the 15 liters per day that is the Sphere standard
Sanitation
The system in Za’atari, described below, is an innovative approach that works well and is cheaper to install than a traditional sewer
system. The same is true for the boreholes and water systems at both Zaatari and Azraq, where high quality, well designed systems
are being run by local operators from the camps, which contributes to sustainability.
In Azraq, The WASH programme has implemented gradual improvements of basic service. In 2014, shelters in Azraq were not
connected to a greywater network, meaning greywater and mud tends to accumulate in the ditches surrounding the shelters.
UNICEF with partners constructed a plot-level grey water network, and provided income generating opportunities within the camp
contributing to camp cleaning and sustainable waste management practices.
Communal toilets remain unpopular in Azraq.
The absence of assessment data with respect to the early stages of the response makes it difficult to determine what the key public
health issues were in relation to hygiene, and the behaviours that needed to be targeted. As noted earlier in the report, WASH
131
programme partner ACF noted a high occurrence of waterborne diseases in Za’atari in 2012/2013. A general observation from the
evaluation team is that the no official documents referenced these key infectious diseases. The exception was the Hepatitis A
outbreak that afflicted Azraq. The first reported measurement on the issue of Hygiene was the Baseline KAP survey in Za’atari in
November 2012, four months after the camp opened. A notable feature for Za’atari and Azraq was the strong commitment to
repeating KAP surveys in both camps on an annual basis enabling one to measure change over time.
The evaluation team did not have access to hygiene data with respect to Host Communities, those in the ITSs, or the settlements
(Hadalat and Rukban). However, the evaluation team understands that given the circumstances it was not feasible to gather hygiene
data in those areas.
Over time the KAP survey results for Za’atari showed an improvement.
In the area of hygiene items, interview respondents indicated that they did receive hygiene items, but not enough of certain items.
The intended results at the settlements were to provide water to the Rukban and Hadalat Refugee Camps as effectively as possible
while achieving value for money. The evaluation team inspected the equipment and operations of the water system at Rukban. The
borehole and related equipment are of high quality, the design is appropriate, the construction is of high quality, and the operations
and maintenance activities are performed properly and on schedule. Staff working for Gama, the UNICEF hired to supply and
operate the system at Rukbank reported in an interview that UNICEF always purchases high quality equipment.
At Rukban, the treatment system is a sophisticated and costly, but warranted and necessary due to the concentration of certain
constituents of the raw water.
Hygiene promotion in the settlements seems to be limited to the distribution of hygiene kits on a periodic basis.
132
EQ6: To what extent did the programme use available resources in an economic manner to provide WASH services and
facilities in camps/settlements and host communities, to ensure child rights, equity and gender equality?
EQ7: To what extent were the costs to deliver water and wastewater services, as well as solid waste management,
rationalized and optimized in the camps and settlements to ensure Value for Money? How do these compare to similar
situations at the local or regional level? Could more cost effective operations/interventions have been undertaken at an
earlier stage?
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
JC 6.1: The degree to which
the programme could
demonstrate it had used
resources in an economic
manner.
JC 6.2: Documentary
evidence that costs were
rationalized and optimized.
- Demonstrated declining
operations costs.
- Demonstration of value for
money.
- Existence of an adequate
system.
- Review of PCA agreements.
- Review of cost reports
-
- Compilation of budget data.
- Complication of cost data.
- Interviews with financial
management staff.
- Interviews with UNICEF
staff
There is no indication that the cost of providing water services in the early stages of the intervention was rationalized.
UNICEF took steps to bring down the unit cost of providing water to refugees in the camps. As figure 10 below shoes, the
evaluation team estimates that between 2013 and 2016, the cost of delivering water to refugees in the camps dropped from 3.34
Jod/m3 to 0.24 Jod/m3.
133
There is some indication that trucking water into Za’atari camp went on longer than it should have, draining the programme’s
resources.
Initially wastewater management was inefficient. Issues such as contractors claiming to remove up to 40 percent more sludge than
they actually removed burdened the programme. Drivers claimed trucks were capable of holding more volume than they actually
did and sometimes carrying waste in to the camp from outside, which was subsequently charged to the UNICEF operation
The wastewater system in Za’atari evolved from unimproved pit latrines to a system of shared septic tanks, a wastewater collection
network, and eventually a wastewater treatment and reuse system.
In Azraq, the wastewater treatment plant is not functioning and has been abandoned.
An innovation that UNICEF developed was the “Waste Taxi” also known for a time as Uber for Waste. It was a predictive model
designed to address with the various different volumes of wastewater tanks that were introduced when people began constructing
their own private washing and toilet facilities in Za’atari.
EQ8: To what extent did UNICEF try to minimize duplication/gap in WASH interventions in camps? And how early did
UNICEF move to more cost-efficient interventions?
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
134
JC 8.1: The existence of a
management plan to
minimize duplication and
gaps.
JC 8.2: The degree to which
the plan was effectively
implemented
- Evidence of full coverage in
beneficiary locations
- Satisfaction with coverage
from beneficiary populations.
- 3
- REACH Assessments
- UNICEF camp management
documents.
- Working group meeting
minutes
- Focus groups among
beneficiary populations
- Interviews with UNICEF and
Partner staff.
-
UNICEF minimized duplication or service gaps in camps by dividing the camps into sectors and assigning different partner
organizations responsibility for WASH services in each sector. UNICEF’s role as sector lead allowed it to unify service standards
among the different partners.
In Azraq, a similar coordination structure was followed as Za’atari. ACTED, World Vision, and THW provided WASH services
with Relief International providing hygiene promotion and WASH in schools. However, coordination among the three entities
appears to have been difficult. There is far less documentation on coordinated efforts in Azraq.
in Zaatari, the WASH programme has reduced costs and realized efficiency by reducing the numbers of partners that they are
managing.
135
EQ9: To what extent were resources (financial and human resources) allocated for the range of intervention types, with
particular attention paid to the respective allocations for humanitarian response and resilience, to achieve the intended
results?
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
JC 9.1: The degree to which
UNICEF could demonstrate
allocation of resources
- Financial information
showing distribution of
resources
- Partner organizations PCAs
and budgets.
-
- Interviews with UNICEF
management staff.
UNICEF Staff estimated that approximately 5 percent of UNICEF’s budget goes to host community interventions. The evaluation
team’s estimate, which as mentioned is based on incomplete information150, is that from January 2013 - -July 2017, about 63
percent of project funds went to camp operations, about 15 percent is spent on host communities, and another 15 percent is spent in
settlements, Rukban and Hadalat (figure 13).
EQ10: To what extent did UNICEF advocate with the Ministries to mobilise their available resources to address the
identified needs? And how efficient was UNICEF’s support to ministries (i.e. MWI and MOE) to reaching most
vulnerable people? Did the WASH programme maximise the potential collaboration with other UNICEF programme
sections and partners?
150 The financial numbers used in this report were derived from UNICEF’s estimates of expenditure on contracts and PCA agreements provided by UNICEF, which were
often in a format that was difficult to read. PCA numbers were budgeted amounts, not actual expenditure. These figures also do not include any indirect costs that may
not have been allocated proportionally by beneficiary group.
136
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
JC 10.1: Existence of
documentation
demonstrating advocacy
efforts.
JC 10.2: Degree to which
GoJ officials agreed that
UNICEF’s support assisted
them in meeting most
vulnerable people.
JC 10.3 Degree to which
maximum potential
collaboration was
demonstrated
- Satisfaction levels of
government officials
- Existence of reports detailing
support initiatives.
- Partners report positive and
maximum collaboration
- Programmatic reports.
- GoJ strategy documents
- Interviews with partners
- Interviews with UNICEF
staff
- Interviews with government
officials.
Over the time frame that this evaluation covers, UNICEF’s primary engagement with the MWI was through a consultant UNICEF
hired to assist the government in developing strategies and response plans. UNICEF also interacted with the government to support
host communities.
The evaluation team saw no documentation of WASH programme advocacy with the MoE during the period described by the scope
of the evaluation. UNICEF Jordan has education programmes that may have had primary engagement with that government
ministry. Still, a UNICEF partner reported that there is more demand for education services than the MoE can handle. UNICEF
could work more to support the capacity of the MoE and should consider building those activities into its future strategy in this area.
137
EQ11: Was the infrastructure in the camps constructed in such a manner as to ensure long term functionality, and
will continue to service the most vulnerable people even when UNICEF is no longer directly supporting? This should
be investigated in terms of Water and Wastewater infrastructure.
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
JC 11.1: Demonstration that
infrastructure in camps was
constructed with long term
functionality as a priority.
JC 11.2 Degree to which
UNICEF balanced quality
and price in construction
decisions.
- Evidence of quality material
use.
- Evidence that long term
sustainability was taken into
account.
- Review of Partner PCAs
- Review of Contractor
documentation
- Onsite inspections
- Review of construction plans
and reports.
- Interviews with engineering
staff
- Interviews with UNICEF
contractors
Following the emergency response phase of the programme, The infrastructure in the camps was constructed to ensure long-term,
sustained functionality. The GoJ prohibited the construction of permanent structures at the beginning of the crisis, forcing UNICEF
to make due with mobile sanitation and trucked in water. When it became apparent that the refugee crisis would not abate quickly,
the government recognized the need for longer-term solutions and UNICEF provided lasting, sustainable solutions.
UNICEF chose highly mechanized treatment systems for both Za’atari (trickling filter and MBR) and Azraq (the MBBR American
system, which eventually failed). These systems are expensive, but the decision to use these systems was sound and valid. The
national laws of Jordan specify stringent effluent quality from municipal wastewater sources. Even though camps are surrounded
by vast unused parcels of government land, wastewater systems must comply with environmental standards. Also, the 4-month
rainy season that makes it more difficult for passive wastewater treatment systems to produce quality effluent year round.
For Za’atari, the common septic tank and small diameter sewer is an appropriate and innovative wastewater collection model.
138
For Azraq, the installation of the MBBR system, translocated from Afghanistan and provided by US donors, turned out to be a poor
choice, although the rationale for selecting it was valid. Unfortunately, the system did not meet the needs of Azraq’s inhabitants or
the program in general. Wastewater strength was higher than anticipated. Following an expensive attempt to make it work, the
system was eventually abandoned. Wastewater is now pumped and transported at great expense. There is a greywater management
scheme that has yet to come on line fully, and until there is some centralized wastewater treatment option, septage will require
hauling some 90 KM away. It is surprising that the Azraq camp would have more difficulty in establishing a wastewater system as it
is neatly laid out, compared to Za’atari, which has developed an elegant and lower cost solution, which has evolved into something
that could become sustainable over the long term as economic development in the camp increases.
The water systems in the camps are designed, installed, operated and maintained to provide long term functionality, even if
UNICEF were to leave, which is unlikely. The quality of the equipment is high quality, which adds to the longevity of the
systems.
The upgrades to the boreholes and pump station in the host community of Irbid are well designed and constructed and should
continue to operate give the required O&M for many years. While the UNICEF interventions are sustainable, the overall water
situation in Irbid however, is not. The extreme water scarcity, depleting aquifer, population growth, and high NRW rates are all
causes for concern. They suggest areas for next steps should donor funding be available.
EQ12: To what extent did UNICEF take sustainability into account at the early stage for water, wastewater and solid
waste management services (e.g. transition to local government institutions and local capacity)?
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
JC 12.1: Evidence of
UNICEF’s Jordan
- Strategic plans for
sustainaibility.
- Workplanning
documentation.
- Interviews with UNICEF
staff.
139
promotion of Sustainability
Initiatives.
- - Interviews with GoJ officials.
The evaluation team has no evidence that UNICEF has a transition plan to government or any other institution in place. The GoJ at
all levels has made it clear that they will not take ownership or spend public funds on the crisis. The government understands that it
is providing an international public service in using internal resources such as water to sustain the refugees and expects the
international community to provide for the rest of their needs.
UNICEF’s approach to sustainability has three aspects.
• Build high quality infrastructure: Building systems and infrastructure that have long term potential to serve Jordanians
and the national government.
• Leverage other, non-governmental resources to contribute to ongoing operations of the camps: This approach includes
building the capacity of refugees to manage their own WASH issues and partnering with national entities to engage in
management.
• Become as efficient as possible: UNICEF continues to find ways to bring down costs so that resources are used as
efficiently as possible, and available as long as refugees need WASH services.
EQ13: To what extent did UNICEF’s interventions increase the resilience of the Government and respective target
populations? To what extent will UNICEF-supported projects (schools, ITSs, water and wastewater infrastructure,
mobilisation) continue to be operational after the end of UNICEF’s financial support
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
140
JC 13.1: Evidence of planned
resilience in UNICEF
Interventions
- - Jordan Response Plan
- Partner descriptions of
projects in PCAs
- Interviews with beneficiaries
- Interviews with partner staff
- Focus group discussions
This evaluation question is difficult to answer because it appears that target populations will require UNICEF’s financial support
well into the future. UNICEF increased the resilience of supported populations and the GoJ in the following ways.
• Designed projects in accordance with the JRP, linking refugee assistance in Za’atari and Azraq with resilience oriented
projects that rehabilitate or extend systems in urban areas with large refugee populations.
• Instituted consistent standards and policies around water access in camps to promote equitable access and reduce conflict.
• Ensured that infrastructure was built using high quality materials.
• Built capacity of stakeholders in camps to manage water, sanitation and hygiene within their communities.
• Created mechanisms for hygiene and water conservation promotion that operate under the leadership of local stakeholders,
such as WASH committees and the Lead Mother model in Azraq.
• Improved water and sanitation infrastructure and services in some host communities with high refugee populations to reduce
tensions.
• Worked with the government to create Wash in School Standards to set a bar for water, hygiene and sanitation in Jordan’s
often overburdened schools.
• Transferred a process for creating response and water strategies to the GoJ that they can replicate in the future.
• Promoted the use of data for strategy development at the MoWI.
EQ14: Does the MWI have the necessary mechanisms in place and capacity to ensure the water and sanitation needs of
the most vulnerable people will be met as a result of UNICEF’s support and advocacy, if another large scale
humanitarian crisis occurs in the future?
141
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
JC 14.1: Evidence of
mechanisms in place at the
Ministry level.
-
The evaluation team did not find evidence that MWI has the necessary mechanisms in place for a future, similar crisis. One reason
for this is that the government does not want to own these crises or been seen as a refuge for at-risk populations.
JC 14.2:
EQ15: To what extent did UNICEF document its WASH interventions and make available to the sector learning on
water, wastewater, solid waste management and mobilisation?
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
JC 15.1: Evidence of the
documentation of
interventions and their use
for learning.
- Documentary evidence of
lessons learned
- Evidence of propogation of
materials.
- Reports from annual reviews
as per the Jordan Programme
Document.
- - Interviews with UNICEF
staff.
- Interviews with partner
organizations.
- Interviews with sector
coordination group
participants.
142
Documentation of interventions for the purpose of sector learning was limited. Lessons learned were shared at coordination
meetings. The evaluation team did not find evidence that the WASH programme documented its experience for the purpose of
sector or institutional learning. This evaluation is the first attempt to document UNICEF’s experience implementing the WASH
programme in Jordan.
EQ16: To what extent were environmental protection and sustainability taken into account in the design and
implementation of programmatic interventions?
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
JC 16.1: Evidence that
environmental protection
assessments were carried
out.
JC 16.2: Evidence that
environmental protection
- Completed environmental
assessments.
- Ongoing environmental
protection efforts.
- Programme Reports
- PCA Environmental
protection descriptions
- Interviews with partner staff.
- Interviews with donors.
- Onsite reviews
143
was built into programme
designs.
Early in the programme, environmental protection was not taken into account sufficiently. Lack of focus on environmental issues
led to Za’atari’s construction over an aquifer that accounts for 34 percent of Jordan’s water supply without proper facilities for
managing wastewater and preventing it from leaching into the ground. As described in earlier sections of this report, the problem
was exacerbated when open pits filled with graywater developed as camp residents began to create private sanitation facilities in
their housing units. UNICEF corrected this issue in Za’atari by installing wastewater collection tanks, reaching 2,000 m3 of
collected wastewater per day, utilizing backwash from borehole water filters for construction and cleaning purposes, and eventually
constructing a water treatment plant.
UNICEF applied the lesson it learned in choosing a site location and water system for Azraq. Prior to the construction of Azraq,
UNICEF worked with the MoWI to conduct an environmental impact assessment. The assessment identified a high risk of
underground water contamination. UNICEF incorporated protections in its design of Azraq’s WASH services. It commissioned
construction and operation of containerized wastewater units. It also attempted to install a treatment plant and septage receival and
dewatering units. As seen earlier in this report, the treatment plant ultimately did not work out, but the evaluation team recognizes
that the decision to include it was appropriate at the time. The septage receival and dewatering units permitted the system to receive
wastewater from trucks and bring down the cost of desludging operations. In 2016, a pond was also constructed at Azraq camp to
store wastewater that could be used for agriculture.
UNICEF also assisted the government in improving environmental protections as regards WASH services. For example, UNICEF
built the capacity of the MoWI to conduct groundwater monitoring more effectively.
Other initiatives that UNICEF implemented that took environmental protection and sustainability into account include:
• Working with the Royal Scientific Society to study the most environmentally sensitive way to provide heading for camps.
Providing heating with gas was chosen.
• Promoting water conservation in camps and host communities through implementation partners.
• Celebrating world events, such as World Environment Day.
• Promoting recycling where feasible.
144
EQ17: Did UNICEF’s WASH response identify and reach the most vulnerable people? To what extent did specific
interventions strike an equitable balance between interventions in camps/settlements and in host communities, to reach
the most vulnerable people?
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
JC 17.1: Sector satisfaction
with UNICEF’s performance
as sector lead.
JC17.2: Evidence that
UNICEF took the lead in
sector coverage and ensured
the most vulnerable’s needs
were addressed.
- Degree to which the most
vulnerable’s needs were
covered.
- Sector satisfaction with
UNICEF’s achievements as
sector lead.
- - Interviews with WASH
sector actors.
As described elsewhere in this report, the evaluation team found that UNICEF ensured the needs of those in camps and at the
settlements were addressed in a coherent manner. UNICEF organized sector coordination efforts and coordination efforts at the
camp and settlement levels to design activities, assign responsibilities to avoid duplication, and set standards to ensure equity of
results as far as that was possible. Following that, UNICEF worked in host communities and ITSs to the extent resources allowed.
The evaluation team does not have evidence that UNICEF used its position as sector lead to make sure those in needs were
identified and that their needs were addressed.
145
EQ18: Did UNICEF as Sector Lead ensure that the needs of those in most need were addressed in a coherent manner,
particularly in comparison to alternative existing systems at the local or regional level (i.e. UNRWA camps)? If not,
how?
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
JC 18.1: Evidence that
UNICEF enhanced
addressed needs in a
coherent manner
-
As described elsewhere in this report, the evaluation team found that UNICEF ensured the needs of those in camps and at the
settlements were addressed in a coherent manner. UNICEF organized sector coordination efforts and coordination efforts at the
camp and settlement levels to design activities, assign responsibilities to avoid duplication, and set standards to ensure equity of
results as far as that was possible. Following that, UNICEF worked in host communities and ITSs to the extent resources allowed.
The evaluation team does not have evidence that UNICEF used its position as sector lead to make sure those in needs were
identified and that their needs were addressed.
EQ19: To what extent did UNICEF role as WASH sector lead enhance response processes and results to avoid
duplication and gap in interventions?
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
146
JC 19.1: Evidence of
processes that UNICEF
enhanced.
- Identification of process
improvements through
documentation and
interviews.
- Vulnerability maps
- National WASH inSchools
strategy.
- Interviews with WASH
programme stakeholders.
Regarding enhancing response processes, a sample of WASH sector meetings from May 2016 through April 2018 show 13
organizations regularly attending. Action points and focal points are thoroughly covered, and cover issues from The National Water
Strategy to Schools Assessments and issues in the camps.
UNICEF also established a number of reporting mechanisms through third party monitors including water quality reporting, ad
security incident reporting. UNICEF established the use of vulnerability Maps for Water and Sewage, though these have only been
in use since 2017. The maps revealed Azraq as the most vulnerable place in both water and sewage.
UNICEF partners that the evaluation team interviewed revealed partner satisfaction with UNICEF as sector lead. One partner said
that, “UNICEF are very good coordinators. They always have staff in Azraq.” A donor said that they were very happy with
UNICEF’s coordination and partnership. A staff member of one partner said, “UNICEF does a very good job of controlling the
work. 95 percent approval rating for UNICEF.”
EQ20: Were partnerships effective and leveraged to the maximum extent to assist the most vulnerable population in
camps/settlements, ITSs and in host communities?
Judgement Criteria Indicators Sources of Documentary
Information
Methods and tools for data
collection
JC 20.1: Degree to which
partners provide in kind
contributions.
- Amount of partner in kind
contribution.
- Degree to which reports
show effective collaboration.
- Financial documents.
- PCAs
-
- Partner interviews
- Host community interviews
147
JC 20.2 Degree to which
vulnerable populations were
served.
l
The evaluation team believes that partnerships were effective and leveraged to the maximum extent in the camps and settlements.
As described in this report, UNICEF utilized its partners to ensure full coverage in the camps and compliance with standards. Based
on the limited financial information that the evaluation team was provided, the team estimates that UNICEF also leverages over
$27, 250,000 in in-kind contributions from the partners it worked with151.
UNICEF also effectively worked with partners in host communities. Partners were relied on to identify and manage key
interventions. WASH in schools programs were also managed effectively through partners.
One aspect of working with partners that UNICEF could have improved was communicating strategic decisions and medium-term
vision for the project. The UNICEF partners that the evaluation team interviewed reported that they often heard about drastic
changes to plans or operations with little notice or time to prepare. One partner noted “UNICEF take quick decisions – the change
from Mercy Corps to ACF in Azraq could have been done in a better way. It was not a transparent process.”
1.
151 Estimate based on a review of PCA’s during the life of the WaSH Programme
148
149
Annex H: UNICEF WASH programme Results Framework Outcome Indicators
Objective Objective
Outcome Indicators Baseline Target Outcome Indicators Baseline 2015 - 2017 Target
Indicator 1: Existence of a
comprehensive national
monitoring system for school
readiness to inform policy
decision on child disparities
Monitoring system
needs improvement
Monitoring
system upgraded
Indicator 1: Existence of
strategic planning capacity in
MoWI and sector and other
relevant institutions in
Emergency preparedness
and response
no clear strategy strategy in place
Indicator 2:Existence of a
child and neonatal
information system in
support of child and
maternal deaths audits
No information
system existent
Information
system in place
Indicator 2: Proportion of
population at camps and
host communities including
schools using improved
drinking water as per Jordan
standards
100,000 at camps, 225,000 at host
communities and 188,000 students
Target: 100% of 2015
target 140,000 camps,
1,550,000 host
community, 200,000
students
Indicator 3: Existence of
updated and functional MoH
guidelines to focus on neo-
natal mortality and high risk
pregnancies
Not updated Updated
Indicator 3: Proportion of
population at camps and
host communities including
schools using improved
sanitation facilities
100,000 at camps, 575,000 at host communities and 180,000 students
100% of 2015 target:
140,000 camps, 545,000
host community,
200,000 students;
Indicator 4: % of health
facilities providing neonatal
care services with the new
MoH guidelines
0 30%
Indicator 4: Proportion of
population practicing
hygiene promotion at
targeted camps, host
communities and schools
Baseline: 100,000 camps, 35,000 host communities, and 188,000 students
Target: 100% of 2015
targets 140,000 camps,
150,000 host
community, 200,000
students
2013/2014 2015-2017
National Institutions provide improved
health and neonatal care services and
quality early childhood care with a focus
on disadvantaged groups
Quality WASH facilities are sustained and utilised, and
hygienic behaviours practiced, by the most vulnerable while
ensuring the protection of the environment
150
Output Indicators
151
2013/2014 2015 2016/2017
Output Indicators Baseline Target Output Indicators Baseline Target Output Indicators Baseline Target
Indicator 1: Existence of
comprehensive WASH
Master plan aligned to
National priority, NRP and
integrate Syrian Crisis
Impact
Fragmented
WASH donor
plans, National
plan in place
(2022 strategy)
not include
Syrian refugee
impact
Comprehensive
WASH master
plan in place
aligned to
National plan
(2022) and
incorporate
Syrian crisis
Impact
Indicator 1: Availability
of effective planning,
monitoring and
coordination
mechanism for
engagement of
government
WASH
working
group,
scattered
concept and
preliminary
idea
Master
Plan/Strategy on
Humanitarian for
development in
WASH, GIS/MIS
Indicator 1: Number of
people living in vulnerable
Households with access to
improved household
WASH facilities and
services
2016:
18,000
2017:
44,000
Indicator 2 : % Reduction
of water wastage/leakage
to ensure targeted host
population accessing
sufficient quantity of
water of appropriate
quality for drinking,
cooking and personal
hygiene (2 cubic)
50 percent of
the water
supplied by the
Public water
systems is Non-
revenue water.
Rreduce water
waste/leakage
by 20%
Indicator 2: Number of
population benefiting
from safe water and
storage facilities
improvement
150,000
refugees in
HC and
225,000
people in
host
communities
150,000 refugees
in HC and
1,400,000 host
communities
Indicator 2: Number of
people with access to
improved municipal water
services
2016:
600,000
2017: 3.2
million
Indicator 3: % Increase of
targeted host population
accessing sufficient using
appropriate sanitation
facilities
Sanitation
Coverage in
Jordan overall is
around 62 %
while in
Northern
Governorates it
is at the level of
37 %
5% increased
specifically in
the Northern
Governorate
Indicator 3: Number of
students provided by
WASH services
188,000
students in
HC
200,000 students
in HC
Indicator 3: Number of
people with access to
improved municipal
sewerage services
2016:
400,000
2017: 2.1
million
Indicator 4: No of targeted
population accessing
hygiene promotion and
hygiene education in
schools
KAP survey
available for
Northern
Governorate
and camps
Host community
(Nationally start
in 2
Governorates)
and 324 schools
Indicator 4: Number of
population at host
communities benefiting
from sewage systems
improvement
575,000 at HC 150,000 Syrians in
HC, 395,000 in HC
Indicator 4: Number of
people with access to
improved WASH facilities
in institutions
2016:
60,000
2017:
114,000
Indicator 5: Availability
of Cost analysis Study
conducted (Y/N)
None Evaluation study
Output 1.6: New Output: Support the Ministry of Water and
Irrigation and Sector partners to address the impact of
refugee influx as well as chronic sector needs
Output 4.1: Government and WASH sector partners plan
and optimally utilize WASH resources and services for
equitable access
Output 4.2: Institutions, utilities and
organisations provide equitable access to
sustainable water and sanitation services for the
most vulnerable
152
2013/2014 2015 2016/2017
IR Indicators Baseline Target Output Indicators Baseline Target Output Indicators Baseline Target
Indicator 1:Availability
of effective working
group coordination
mechanism
WASH
Working
group
functional
and ToRs
developed
Formation of a
working group
for the sector
response plan
Indicator 1: Number refugees
provided with safe access to
sufficient water for drinking
and domestic use through
water trucking
100,000
refugees
140,000
refugees
Indicator 1: Number of
people in camps
reached through
social/community
mobilisation on key
WASH messages and
water conservation
2016:
50,000
2017:
50,000
Indicator 2:Number of
emergency affected
population (male and
female) provided with
access to drinking and
domestic water
250,000
345,000
refugees (in
camps and off
camp) and
180,000
affected
Jordanians.
Indicator 2: Number of
refugees provided with safe
waste water disposal through
dislodging of septic tanks
100,000
refugees
140,000
refugees
Indicator 2: Number of
people in institutions
in camps reached
through social/
community
mobilisation on key
WASH messages and
water conservation
2016:
20,000
2017:
20,000
Indicator 3: Number of
Emergency affected
population (male and
female) provided with
access to soap and
other hygiene items
345,000
refugees (in
camps and off
camp) and
180,000
affected
Jordanians.
Indicator 3: Number of
refugees provided with safe
waste water disposal through
provision of Waste Water
Network in Zaatari Camp
0 refugees 140,000
refugees
Indicator 3: Number of
people in the host
community reached
through
social/community
mobilisation on key
WASH messages and
water conservation
2016:
18,000
2017:
44,000
Output 4.3: The most vulnerable women and
children are aware of and practice key hygiene
behaviours and practice water conservation
IR 1.3: Relevant Core Commitments on WASH are met
in all humanitarian settings
Output 4.3: Urgent needs of vulnerable people
especially children are met through quality and
lifesaving WASH services
153
IR Indicators Baseline Target Output Indicators Baseline Target Output Indicators Baseline Target
Indicator 4: Number of
Emergency affected
population (male and
female) with access to
appropriately designed
toilets and sanitation
services
345,000
refugees (in
camps and off
camp) and
180,000
affected
Jordanians.
Indicator 4: Number of
refugees provided with safe
solid waste management
facilities
0 refugees 140,000
refugees
Indicator 4: Number of
people in institutions
in the host community
reached through
social/community
mobilisation on key
WASH messages and
water conservation
2016:
60,000
2017:
114,000
Indicator 5:Number of
population (male and
female) covered
through face to face
(including HH and
Group sessions)
hygiene promotion
messages
19,300
345,000
refugees (in
camps and off
camp) and
180,000
affected
Jordanians.
Indicator 5: Number of
refugees provided with safe
access to sanitation facilities by
operating and maintaining the
existing WASH Blocks through
WASH Committees and/or
community involvement
100,000
refugees
140,000
refugees
Indicator 5: Number of
people in informal
settlements reached
through
social/community
mobilisation on key
WASH messages and
water conservation
2016:
15,000
2017:
45,000
Indicator 6:Number of
children (girls and boys)
provided with access
to safe water,
sanitation and hygiene
facilities in their
learning environment
and in child friendly
spaces
72,985 230,633
Indicator 6: Number of refugee
Boys and Girls in camps
provided with WASH in schools
19,108
students
45,000
students
Indicator 7:Number of
children (girls and boys)
not practicing open
defecation in the camp
settings and in host
communities
NA 100%
154
2016/2017
Output 4.4: The urgent WASH needs of the vulnerable women and children are met through safe, predictable and sustainable water and sanitation services and facilities
Output Indicators Baseline Target
Indicator 1: Number of people in camps supplied with water (tankering or network)
2016: 140,000 2017: 140,000
Indicator 2: Number of people in camps receiving wastewater services (tankering or network)
2016: 140,000 2017: 140,000
Indicator 3: Number of people in camps receiving solid waste collection services
2016: 140,000 2017: 140,000
Indicator 4: Number of people in informal settlements with improved access to WASH facilities and services
2016: 15,000 2017: 45,000
Indicator 5: Number of people in institutions benefitting from WASH facilities and services
2016: 20,000 2017: 20,000
155
Annex I: UNICEF Provided Financial Estimates Note: The following documents were provided after the evaluation period.
Overview of examples of increased efficiency in the
UNICEF Jordan WASH programme
(June 2016, updated July 2016, updated October 2016)
Ongoing WASH services in Camps
UNICEF has been providing WASH services to camps since July 2012. Once critical WASH
services were established, UNICEF has consistently worked to reduce the cost associated
with the provision of these life-saving interventions through the construction of infrastructure
to reduce external tankering, establishment of third party monitoring mechanisms to ensure
that only actual services delivered were paid for (verification of truck volumes, reduced
recycling of wastewater etc) and the progressive rebidding of services to ensure that the unit
costs are progressively reduced.
- Unit costs have been progressively reduced as outlined in the table (since 2012 or
from the start of the service):
Price
Difference
(JD/m3) % Reduction
Water Tankering - Za - internal 0.17 14%
Water Tankering - Za - external 0.41 16%
Water Tankering - Az - internal 0.12 8%
Water Tankering - Az - external 0.2 6%
Water Tankering - KAP -
external 0.78 31%
Water Tankering - CC - external 0.78 31%
Desludging - Za - internal 0.02 1%
Desludging - Za - external 0.55 17%
Desludging - Az - internal
Desludging - Az - external 0.61 10%
Desludging - KAP - internal 0.73 29%
Desludging - CC - internal 0.73 29%
Solid waste unit costs - Za 2.9 67%
Solid waste unit costs - Az 8.46 85%
156
Solid waste unit costs - KAP 2.7 63%
Solid waste unit costs - CC 2.7 63%
- Transitioned from external water sources to internal tankering (three wells in Za’atari
and one well in Azraq)
As the four boreholes are now operational, the water is supplied for most of the year
from the internal boreholes with no need for external tankering. In Za’atari, Borehole
1 became operational in 2012, Borehole 2 became operational in 2013 and Borehole
3 in May 2015. In Azraq, the internal borehole started providing water to the camp in
October 2015. Through these key infrastructural projects, the cost of supplying water
has been reduced considerably For Za’atari, the difference between the highest
external water tankering unit cost and the current price is 1.48 JD/m3. Likewise, for
Azraq, the price difference is 2.05 JD/m3. On average, this equates to a monthly cost
difference of 150,000 JD for Za’atari and 73,800 for Azraq (excluding the operational
costs of the boreholes)
- Improved Operation and Maintenance of the three boreholes has resulted in reduced
downtime of the boreholes and their more consistent operation due to regular
monitoring and maintenance (there was no daily external water trucking from
November 2015 to May 2016).
- Over the period July to November 2014, the average daily demand (including
institutions) was 3,704 m3, of which 780 m3 (21%) was from Borehole 1, 1,160 m3
(31%) from Borehole 2 and the remaining 1,757 m3 (48%) was from external
boreholes. Prior to 2015, there were no seasonal allocations and issues were
regularly faced with Borehole 1 and turbidity. As a result, it was often necessary to
further supplement the water supply with external water. Over 2015, the water needs
at a block level were regularly recalculated (based on updated population numbers as
well as a seasonal alteration to take account of increased needs in summer and lower
needs in winter), and as a result, the volume of water required was recalculated and
the volume reduced, over an annual period
- UNICEF’s positive response to the water strike in May 2015 (alternative modes of
delivery and the accelerated operation of BH3, as well as the construction of a mini
network within a few days) sent a very strong signal to the Contractors and their
ability to negotiate
A mini-network was installed and this enabled 200 to 300m3/day to be delivered
directly by pumping and as a result, there has been a reduction (of approx. 270
JOD/day, 8,100 JOD/month, 129,600 to date – the network cost less than 5,000 JOD
to construct)
- Construction of a connecting line to the main pumping distribution line in Za’atari as
a Contingency supply (with the permission of the Minister of Water and irrigation)
which can be activated in the event of an emergency
- Mobilisation on water conservation to parents and children (to reduce children
playing with the water) and reducing the losses – refugees have been engaged to
disseminate messages at tapstands in Azraq where children often played with the taps,
which has resulted in a considerable reduction in the water consumed, as well as the
amount of standing water
- Replacement of items which were prone to vandalism e.g. the ventilation pipes (triple
bolted) and taps (taps removed and a basic design installed) which has significantly
reduced the cost of replacement
157
There was a reduction of 58% in the costs for the replacement of taps, and 24% for
the replacement of the ventilation pipes in Azraq
- Transitioned from external desludging in Za’atari to the internal treatment plant and
nearing completion of the Azraq wastewater treatment plant (unit transportation costs
will reduce to less than one third of the external)
The cost to manage 1 m3 of wastewater in Za’atari Camp has decreased from an
estimated with the WWTP completion in February 2015 (the figures are currently
being updated)
- Extensive third party monitoring to accurately determine the volume of wastewater
reported (physical inspection of the truck when it arrives and before it is offloaded)
and to reduce the scope for ‘recycling’ of the waste
See the figure below:
- Development of an Operational Plan to ensure the most efficient use of the desludging
trucks (to ensure a smooth supply of wastewater to the treatment plant and to reduce
the scope for external desludging)
See the figure below:
158
- Expansion of the storage capacity at the Za’atari Wastewater Treatment to reduce the
scope for external tankering
The works to expand the storage facility (by 400 m3 storage + 200 m3 treatment
capacity) cost approximately 250,000 JOD – this led to an additional 19,500 m3 of
wastewater/month which could be treated at the plant (and not transported
externally) which has led to a saving of c20,800 JOD/month on the tankering costs
- Independent physical verification of the volume of the wastewater trucks – brought to
a weighing station as often modifications have been made to the trucks (the registered
volume was different to the actual measured volume)
The actual measured figure was 27% but a compromise figure of 23% was agreed
with the Contractor – this equates to a 23% saving in the monthly desludging costs
for Za’atari
- Construction of a weighbridge for the volume of the compacted solid waste in Azraq
- Regular rebidding of contracts for improved unit rates as well as UNICEF’s Long
Term Arrangements (as given in the first table)
- Progressive reduction in Cash For Work (gradually reduced and in consultation with
all agencies) while supporting alternative job opportunities (recycling)
The first scale down of CFW took place in April 2016, which reduced the cost by
about 20% with a second (20%) scheduled to take place from 9 October 2016 – these
two reductions will reduce the monthly cost by approximately JOD 10,000
- Progression from partner implementation to UNICEF’s direct engagement of
Contractors (networks, desludging, borehole operation and maintenance, construction
etc)
- The decommissioning of public WASH blocks has significantly reduced the cost of
repairs, operation and maintenance. The installation of private toilets with no O&M
costs for UNICEF and much appreciated by the refugees (significant cost reduction
(over 50%) in the installation of private toilets due to the recycling/or reuse of
materials from decommissioned public WASH blocks)
159
As of early October, 346 WASH Blocks have been decommissioned (out of 355) in the
camp – the materials from the blocks are being recycled for Household toilets for
vulnerable families, where possible
- The phasing out of O&M of WASH facilities in schools and the subsequent
engagement of another agency (by Education section) have led to reduced operational
costs for WASH
- Reduced cost of water quality monitoring due to internalized water supply
- Functional water supply contingency plan have led to price stability; prices could
have been most likely increased if we were more vulnerable
It is hard to quantify this but the fact that alternative modes of transport and delivery
were up and running within one day of the strike sent a very strong message, as did
the construction of the tanks for the network – we consider that this ensured that the
prices for the internal and external water did not increase in the subsequent bidding
period
- Reduced power consumption due to the installation of VFDs at the boreholes
(installed on all four boreholes and already making a difference)
This has just recently been introduced but is anticipated to have two impacts; a) on
the energy consumed (c 7.5% for the reduction in the output) and b) more continuous
water supply with less interruptions and shutting down/restarting
- Construction of water reuse pond where contractors can access water for construction
works at no cost. Otherwise, UNICEF would have been charged for construction
water. The water was also used for cleaning of the water tanks, again contributing to
reduced water needs
During the peak of wastewater network construction, an estimated 56m3 (six trips of
8m3 truck) of water was used for construction, as well as for other purposes – on
average, approximately 60m3 per day of water is collected and used, which has saved
approximately 2,700 JOD per month
- Reuse of water from the Wastewater treatment plant for agricultural purposes in
Za’atari and under construction for Azraq
Approximately 1000 to 1200 m3 of treated wastewater is produced by the Za’atari
wastewater treatment plant, each day – if this had to be tankered to the site, the cost
would be 25,000 to 49,500 JD/month, depending upon the source
- Efficiency review of the energy requirements of all of the infrastructure (an initial
review of the energy consumption of the Za’atari Wastewater Treatment Plant has
been carried out)
- Partner contribution to cooperation agreements (up to c10% but up to 70% in some
agreements)
Za’atari Network Construction:
UNICEF will directly engage the Contractors for second phase of Water and Wastewater for
the Za’atari Networks
- This will significantly reduce the number of staff maintained by partners and
associated indirect costs and overheads, as well as reduce duplication in roles and
responsibilities
- UNICEF will combine the phase II water and wastewater network design and
construction tender documentation, and will directly engage construction supervision
consultants and construction contractor(s). This strategy will improve cost
effectiveness through streamlining of the project management structure and reduction
160
in construction contractor costs through joint delivery of the networks. Joint delivery
of the networks will also offer significant benefits to beneficiaries through reduced
construction impact on communities, reduced construction safety and coordination
risk, and improved construction quality outcomes
It is estimated that by revising the project management structure and combining the
water and wastewater work into distinct packages, the cost will be further reduced by
an estimated 15-22% due to construction and management efficiencies.
- The works have been designed to facilitate the operation of the water network in
districts which will facilitate a gradual reduction in the costs of tankering
It is planned that three districts will be connected per quarter from June 2017 (25%
reduction per quarter) from when the pumping stations start to be commissioned –
this will reduce the Za’atari costs by over 87,000 JD per quarter (for each three
districts)
- The costs to supply water across the camp via the network will eliminate the need for
water tankering and as a result dramatically reduce the cost of water supply – directly
(no costs for tankering) as well as indirectly (reduced monitoring, support staff,
transportation, water quality monitoring etc)
The elimination of tankering from Za’atari and Azraq will result in savings of over
US$ 4M a year (over US$ 7M compared to external tankering)
- Although there will still be a need for desludging of the wastewater network, the
frequency and scale will be significantly reduced
It is anticipated that there will be a reduction in the frequency of desludging from
household tanks from once per week to once every 6 months (desludging from the
communal tanks of the WASH blocks was daily)
- Procurement of desludging trucks for the network desludging (once or twice a year)
which can be done using our own trucks which we can procure
The cost of desludging will be reduced by over US$ 1.5M/year
- Bulk procurement of tanks for Phase 1 of the wastewater network (across all three
partners)
- Pipeline for Azraq – elimination of trucking
- UNICEF’s Engineers oversight of the designs and revision of the BoQs
Reduced reliance on partners and increased oversight of the technical requirements
and strategic planning of progressive construction, commissioning and
operationalization of the networks, will ensure that ongoing operational costs are
progressively reduced as the camp transitions to a longer term sustainable operations
and maintenance framework
- Reduce the cost of the chemical used to treat the wastewater at Za’atari Wastewater
Treatment Plant (ZWWTP)
Review of the ZWWTP hydraulic and process efficiency in coordination with WAJ
will ensure that plant energy and chemical usage requirements are optimised while
meeting camp wastewater treatment requirements. UNICEF Engineers are working
collaboratively with other agencies to investigate the potential for ZWWTP sludge to
be used in a trial biogas generation facility that will process generated sludge
(removing the need for external trucking and disposal of the sludge) as well as
generate renewable energy for export to the local power grid and reduction in camp
operational power costs.
Support to Host communities:
161
- UNICEF support the Water and Sanitation Vulnerability mapping to ensure that areas
in most need were prioritised (based upon agreed criteria)
- UNICEF supports the replacement of leaking water lines and inefficient equipment to
reduce the operational costs in water systems
- UNICEF supports the capacity building of operators of the utility companies to
undertake maintenance and strengthened oversight of the equipment
- UNICEF supports the repair, unblocking and replacement of sewer lines to reduce
sewerage overflows
162
Annex J: Additional Financial Information
The table below summarizes additional financial information provided by UNICEF in
December 2018, after the evaluation period concluded.
The table below summarizes UNICEF WASH programme reported expenditure by category
for the years 2012 through 2017. The table include categories that accounted for at least 1
percent of total expenditure. These categories account for 96 percent of total expenditure. The
top expenditures were for the categories “Sanitation and Solid Waste Management, and
“Provide Access to Safe Drinking Water,” accounting for 35 percent of the total, though there
are many categories with similar titles on the list.
Items USD Percent
SANITATION AND SOLID WASTE MANAGMENT
$47,113,983 18%
PROVIDE ACCESS TO SAFE DRINKING WATER $44,568,288 17%
PREPAREDNESS & CONTINGENCY PLANNI $22,224,107 9%
PROVIDE REFUGEES IN CAMPS WITH SAF $15,547,325 6%
CONSTRUCT/EXTEND WATER NETWORK IN $13,650,381 5%
PROVIDE DRINKING WATER-CAMPS $10,147,397 4%
CONSTRUCT WASTEWATER NETWORK IN ZA $11,457,662 4%
SEWAGE NETWORK-CAMPS $7,409,768 3%
HYGIENE EDU AND MATERIALS $6,011,048 2%
SOCIAL MOB IN CAMPS (GENERAL) $5,590,211 2%
WASTEWATER DISPOSAL-CAMPS $4,210,105 2%
SOLID WASTE MANAGEMENT-CAMPS $4,418,779 2%
WASTEWATER COLLECTION (DESLUDGING) $4,806,943 2%
COLLECT AND DISPOSE SOLID WASTE $4,997,696 2%
TECH ASSIST FOR SUSTAINABLE W $4,732,505 2%
PROVISION OF SANITATION AND HYGIENE $2,074,047 1%
MERCY CORPS PCA/0/12 $1,625,658 1%
ACCESS TO SAFE WATER $1,637,291 1%
WASH SUPPLIES $3,408,290 1%
WASH IN SCHOOLS $3,561,058 1%
WATER SYSTEM REHABILITATION $1,715,156 1%
SEWAGE NETWORK REHAB-HOST COMM $1,615,982 1%
HYGIENE PROMOTION-CAMPS $3,152,868 1%
WATER SUPPLY NETWORK-CAMPS $3,607,696 1%
WASTEWATER TREATMENT-CAMPS $1,322,810 1%
OPERATE/MAINTAIN WASH FACILITY $2,732,411 1%
O&M WASTEWATER TREATMENT PLANTS IN $2,788,510 1%
PROVISION OF WASH SERVICES IN ITS $3,037,358 1%
WASH FACILITIES IN INSTITUTIONS I $2,339,079 1%
WATER INFRASTRUCTURAL SUPPORT $3,060,999 1%
WASH FACILITIES IN SCHOOLS, CFS, C $1,627,130 1%
Total $246,192542 96%
163