Meta-Evaluation of the 2010 Earthquake Response in Haiti

Stichting Samenwerkende Hulporganisaties (SHO)

Meta-evaluation Report

Ede, the Netherlands, 20th of October 2015

Final report Meta-Evaluation of the 2010 Earthquake Response in Haiti

Stichting Samenwerkende Hulporganisaties (SHO)

Irma Alpenidze Sjoerd Zanen

Ede, the Netherlands, 20th of October 2015

1

Table of Contents

Abbreviations 2

Executive Summary 3

1 Introduction 5

2 Quality assessment of the SHO evaluation reports 7

2.1 Quality assessment: process and methodology 7

3 Assessment of the results of SHO's use of the funds 16

3.3 Overview of performance by criteria 17

Relevance/appropriateness 18

Effectiveness and impact 19

Efficiency 20

Sustainability and Connectedness of the SHO earthquake response interventions 21

Adherence to standards/Coherence 22

Adherence to cross-cutting themes 23

Coverage 24

Strengthening of responses 25

Cooperation 25

3.4 Summary of findings 27

3.5 Evaluation and Learning 28

3.6 Final remarks 30

4 Concluding remarks, lessons and recommendations 31

4.1 Lessons 31

4.2 Result-presentation by SHO 32

Annex 1. Evaluation Report Quality Assessment Grid 35

Annex 2. Overview of overall scores for all assessed SHO evaluation reports 37

Annex 3. Persons consulted 39

Tables and Figures Table 2-1. Explanation of the overall scoring for the reports .................................................................................. 7 Table 2-2 Distribtion of reports by score and clusters ............................................................................................. 8 Table 2-3. Distribution of reports by scores and budget categories ....................................................................... 9 Table 2-4 Distribution of reports by category, cluster and budget ......................................................................... 9 Table 2-5 Comments and recommendations for sub-criteria, quality evaluation grid.......................................... 12 Table 3-1. Overview of participation in main clusters by SHO-partners ............................................................... 17 Figure 1-1 Map of affected areas of Haiti earthquake _______________________________________________ 5 Figure 3-1 Infographic published by SHO on Haiti Earthquake Response _______________________________ 19

2

Abbreviations

AA ALNAP

ACT Alliance - Action by Churches Together for Development Active Learning Network for Accountability and Performance in Humanitarian Action

ARK ECD

Algemene Rekenkamer (Dutch for National Court of Audit) Early Childhood Development

GoH Government of Haiti HAP HC HQ

Humanitarian Accountability Partnership Humanitarian Coordinator Head Quarters

IASC Inter-Agency Standing Committee ICCO Interkerkelijke Coördinatie Commissie Ontwikkelingssamenwerking (Inter-Church

Coordination Commission for Development Cooperation) IFRC IOB KiA LdH NGO NLRC OECD DAC RNDDH RTE

International Federation of Red Cross and Red Crescent Societies Policy and Operations Evaluation Dept of the Ministry of Foreign Affairs Church in Action Leger des Heils (Salvation Army) Non-Governmental Organisation Netherlands Red Cross Society Organisation for Economic Co-operation and Development Development Assistance Committee Réseau National de Défence des Droits Humains Real Time Evaluation

SHO Stichting Samenwerkende Hulporganisaties (Cooperating Aid Agencies) ToR UN

Terms of Reference United Nations

UNDP United Nations Development Programme UNICEF United Nations International Children's Emergency Fund VNG Netherlands Association of Municipalities WASH Water, Sanitation and Hygiene WHO WV

World Health Organisation World Vision

3

Executive Summary

The meta-evaluation of the 2010 Earthquake Response in Haiti was commissioned by SHO and was executed by MDF T&C, from August-October 2015. It consisted of two stages: first - an assessment of the quality of the evaluations carried out by the individual SHO members with regard to their implemented activities in Haiti, and second - an assessment of the results of the use of the SHO subsidy, including key lessons learned. The first part – an assessment of the evaluation reports quality was done with the use of three criteria: Reliability, Validity and Usefulness, detailed in 17 sub-questions. To focus the meta-evaluation efforts, these reports were further categorised into four groups. All 40

1 reports were of acceptable quality, i.e. not a single

one fell into the category with lowest scores. All reports qualified as reasonably reliable, valid and useful documentations of the SHO Response. 31 out of 40 (75% of all) reports, from which 12 (30% of all) with higher scores, were analysed further during the second stage of the meta-evaluation. Overall, an analysis of the quality of all reports per criteria revealed a gradual improvement in all three areas: Reliability, Validity and Usefulness: with exception of year 2011, different aspects of the reports reliability and validity improved while their usefulness remained stably high. No significant correlation was found between the quality scores and clusters on which projects worked or between the quality scores and total budgets of the evaluated projects. An analysis of scoring patterns for each sub-question resulted in a recommendation on slightly adjusting the quality evaluation grid. The assessment of the results of the SHO 2010 Haiti Earthquake Response projects was done on the basis of the selected evaluation reports, and on a number of consultations: individual interviews, focus group interviews and one briefing session. The reports were assessed based on nine main criteria: (1) Relevance/Appropriateness, (2) Effectiveness and Impact, (3) Efficiency, (4) Sustainability and Connectedness, (5) Adherence to standards/Coherence, (6) Adherence to cross-cutting themes, (7) Coverage, (8) Strengthening of responses and (9) Cooperation. The assessment of results, based on the evaluation reports, shows that interventions funded by SHO convincingly met the immediate needs of the local earthquake victims, especially in the emergency and relief phase. The projects which were best tailored to local needs and enhanced ownership were those with a high degree of involvement and participation of the Haitian people, communities and government, those who disposed of good knowledge of the context and of the needs of victims and those who communicated well with beneficiaries and institutions. In general, the projects were effective. Flexibility has contributed to enhanced effectiveness. The biggest challenge for the projects was the urban character of the disaster for which most were ill prepared. Although rural areas have been overlooked in the early stages of humanitarian response, this has been balanced out later, resulting in the satisfactory geographical coverage of the response projects. Efficiency was assessed as positive/neutral. Amongst main factors contributing to it were the use of local resources and flexibility in approach; amongst the factors hampering efficiency were external factors such as rising prices and procurement procedures. Sustainability of results was related to the planning of development support in the wake of emergency relief, as did five SHO member organisations that continued their work till the end of 2014. Crucial supportive factors for sustainability were the issues of ownership such as anchoring the response in government structures, communities and culture, local capacity development and advocacy. All organisations adhered or attempted to adhere to international humanitarian standards. When not all standards were respected this was due to work pressure, limited space, resources and experience of local staff. Gender and vulnerability were taken into consideration. The issue of "rape epidemic” in camps might have been exaggerated, but gender violence was a factor to be seriously reckoned with, which many SHO projects did. The SHO members have cooperated with government institutions and local organisations and did so most often quite successfully. Most of the SHO 1 Meta-evaluation based its findings on 42 reports, but two were removed from the quality assessment as they are

recordings of lessons learned, therefore directly qualify for the second part – an assessment of results.

4

members worked in UN-clusters, despite some of the clusters not functioning optimally. The SHO members embraced a more efficient way of working in "baby clusters", with the participation of local government. The SHO organisations appeared to agree with most of the conclusions of the locally commissioned evaluations, which have also served to enhance the capacities of local partner organisations. The SHO organisations internally dispose of evaluation and learning mechanisms, either formal or informal. While some organisations have formally integrated lessons in their manuals and policies, others seem to have capitalized knowledge or applied lessons to a lesser degree. The global international humanitarian earthquake response in Haiti has been criticised for exclusion of Haitian participation at all levels, poor coordination, weak needs assessment and risk-averse leadership (cf Lord Ashdown, 2011). One evaluation of the SHO response, however, characterizes one SHO member project as a rupture with the classical aid approach (such as criticised above) and calls the approach a model in which capacities are transferred to men and women so that they can take their future in their own hands. The (meta) evaluators believe that most SHO responses have adhered to such models and have been successful in the majority of cases. Regarding the issue whether the presentation of the results achieved in Haiti could have been done in a more uniform/harmonized manner, we believe that, since the SHO Haiti Earthquake Response was not designed in such a way, the outputs were not easily comparable. Diversity is a characteristic of the SHO approach, and should rather be considered as its strength than as a weakness: it reflects the identity of the Dutch humanitarian organisations. The information in SHO reports published on web sites is easily accessible to interested readers and the media. The presentation to the general Dutch public of the achieved results by the SHO Haiti Earthquake Response in the form of a well-designed SHO infographic has been adequate. For the future, it is recommended to SHO to prepare a short film about its disaster responses, to be broadcasted on TV.

5

1 Introduction

On January 12, 2010 Haiti was hit by an earthquake with magnitude 7.0 in the densely populated region near its capital Port-au-Prince. About 220,000 persons lost their lives, 300,000 persons were wounded and initially about 1.5 million persons became displaced. Practically the whole government machinery and public services came to a standstill due to loss of lives and infrastructural destruction. The same applied to a number of international organisations. Figure 1-1 Map of affected areas of Haiti earthquake

Immediately after the earthquake, the SHO (Stichting Samenwerkende Hulporganisaties) started a co-ordinated effort for the victims of the earthquake in Haiti. The SHO is a partnership of 10 Dutch aid organisations that collectively raise funds after major humanitarian disasters to provide aid to victims. Since its creation in 1989, SHO staged 41 campaigns to raise funds for humanitarian responses. Other major initiatives concerned for instance hurricane Mitch in Central America (1998); the tsunami in Asia (2004), the earthquake in Pakistan, India and Afghanistan (2005), typhoon Haiyan in the Philippines (2013) and the earthquake in Nepal (2015). With 111 million euro raised, The Haiti campaign is the second largest in SHO history. The regular participating organisations in the Haiti Earthquake Response were Cordaid (Mensen in Nood), ICCO & Kerk in Actie, Oxfam Novib, Netherlands Red Cross Society (NLRC), Save the Children, Tear Fund, Terre des Hommes, UNICEF and World Vision. In addition, 6 guest organisations took part: CARE, Dorcas, Habitat for Humanity, Plan Nederland, Leger des Heils and VNG International. Funds were granted to the regular participants according to a beforehand fixed distributive code. Guest organisations received smaller amounts based on specific project proposals and operated under the responsibility of a regular participant. Fundraising by SHO was the only coordinated effort. Participants were responsible for the planning, implementation and evaluation of their own projects and programmes. The funds raised were spent on emergency assistance as well as reconstruction (e.g. protection, disaster management, health care, shelter, food security, WASH, schools/education). The initiative officially ended in December 2014. Since the total amount of funds raised for the Haiti Earthquake Response consisted in part of funds from the Ministry for Development Cooperation

2, SHO, in accordance with its contractual obligation, organised this

independent meta-evaluation, which evaluates the total results of the use of the SHO Haiti resources. 2 More than €41 million

Source: United Nations Office for the Coordination of the Humanitarian Affairs (UN OCHA)

6

In the years between 2010 and 2014 each of the participating SHO members commissioned independent evaluators to evaluate their responses (projects). SHO members used their own formats for the ToRs and evaluation reports, and, this way, allowed for the evaluation reports of the individual participating organisations "to have different objectives, methodologies, focus, lay-outs and depth"

3. Therefore, this meta-

evaluation was tasked to find commonalities and overall learning from all available reports. The meta-evaluation consisted of two main stages, respectively responding to two objectives: 1) an assessment of the quality of the evaluations carried out by the individual SHO members, and 2) drawing of general conclusions on the total results of the use of the SHO subsidy, including key lessons learned. For the assessment of the quality of the SHO members' evaluation reports (stage one) a quality evaluation grid was developed by MDF, which was applied after approval by the Reference group for this meta-evaluation (consisting of representatives of Oxfam Novib, NLRC and Cordaid). In drawing general conclusions of and lessons learned from the SHO subsidy total results (stage two), the meta-evaluation takes into account the OECD DAC criteria for disaster response evaluations, in accordance with ALNAP

4 quality criteria; the Sphere minimum standards in Humanitarian Response during the intervention, the

Code of Conduct for the International Red Cross and Red Crescent Movement and NGOs in Disaster Response Programmes during the intervention. Apart from the two stages mentioned, this meta-evaluation has a third objective that concerns the SHO as a whole rather than the individual projects in Haiti directly, namely "to consider whether SHO could have done more during implementation and closure of the projects to present the results achieved in Haiti in a more uniform/harmonized manner or whether this is not desirable given the nature of SHO funding, differences between organizations, and context of the emergency"

5.

Following the introduction, Chapter 2 describes the process and results of the quality assessment of the evaluation reports provided by SHO. Chapter 3 summarises and synthesises the answers on key questions related to ten issues in accordance with the above- mentioned criteria

6. The Chapter 4 comprises of concluding

remarks. The report closes with annexed documents.

3 Ref "SHO ToR Meta-evaluation of the 2010 Earthquake Response in Haiti", p.2. 4 Active Learning Network for Accountability and Performance in Humanitarian Action. 5 ToR meta-evaluation, p. 2 and 3. 6 I.e.: (1) Relevance/Appropriateness, (2) Effectiveness and Impact, (3) Efficiency, (4) Sustainability and Connectedness, (5) Adherence to standards/Coherence,

(6) Adherence to cross-cutting themes, (7) Coverage, (8) Strengthening of responses and (9) Cooperation.

7

2 Quality assessment of the SHO evaluation reports

2.1 Quality assessment: process and methodology

MDF received 134 documents that were narrowed down to 43 unique reports (singling out the reports with the

same content that were sent in by different participating agencies). Further, one of the reports was removed

from the assessment list completely as it is written in Spanish. Moreover, two reports were removed from the

assessment phase 1 as they are recordings of lessons learnt and, although valuable for part two of the meta-

evaluation, would not meet the set criteria to pass the assessment threshold. At the end, the final list of

evaluation reports assessed amounted to 40, from which 24 were in English and 16 - in French.

The check of the evaluation reports was carried out to identify their quality and determine which of the studies

should be included in the formulation of general conclusions and lessons learned. The reports were evaluated

based on the quality assessment grid that combines criteria from two checklists for the evaluation reports

quality: one from the Policy and Operations Evaluation Department (IOB) of the Ministry of Foreign Affairs of

the Netherlands and another – from the Active Learning Network for Accountability and Performance (ALNAP).

The sub-questions for the criteria from both documents were combined and a final version of the quality

evaluation grid for this meta-evaluation agreed with the Reference Group. An overview of the selected criteria

comprising the final assessment grid as well as the links to both checklists is attached as Annex 1.

Final assessment of the evaluation reports quality was done with the use of 3 criteria: Reliability, Validity and Usefulness, detailed in 17 sub-questions. Each question was scored from 1 to 4 where 1 – Poor, 2 – Fair, 3 – Good and 4 – Excellent. Summary of cumulative scores that defined the overall quality judgement of the report, explained in the Table 2-1 below. The first column of this table lists the categories of reports, second refers to the range of points that qualify reports to fit into a category and the last column explains the cut-off point for each category, i.e. the highest possible score that makes a report to qualify for a given category. Table 2-1. Explanation of the overall scoring for the reports

Overall score Range of points Explanation of the cut-off point

Category A 56 – 68 points All questions scored as “excellent” (17 * 4) = 68

Category B 44 – 55 points Half of the sub-questions scored “good” plus another half with 3,5 (17/2 * 3 + 17/2 * 3.5) = 55

Category C 27 – 43 points Half of the sub-questions scored “fair” plus another half “good” (17/2 * 2 + 17/2 * 3) = 43

Category D 01 – 26 points Half of the sub-questions scored “poor” plus another half scored “fair” (17/2 * 1 + 17/2 * 2) = 25,5

All 40 reports that were part of stage one were of acceptable quality. They qualified as reasonably reliable, valid and useful documentations of the SHO Haiti Earthquake Response. Following the methodology for this meta-evaluation, to focus the meta-evaluation efforts, the reports with higher scores were taken on board to the second stage of the meta-evaluation. After the assessment 31 out of 40 reports (75% of all) were selected for this

7, from which 12 (30% of all) reports were reports with highest

scores. The overall budget of the projects that have been evaluated in the meta-evaluation covers 62 % of the total amount of SHO Haiti funds. From this, the projects of 31 selected reports with highest scores account for 56% of the total amount of SHO Haiti funds. Note: the actual total is higher, however as some evaluation reports did not indicate the total project amount, they could not be included in this calculation. We advise to streamline specifying the overall budget together 7 Please refer to Annex 2 for the selection and overall list of the reports

8

with specific project information (such as duration, place, type of evaluation) in the evaluation reports as a standard.

2.2 General overview of quality assessment results: all reports

Overall, an analysis of the quality of all reports per criteria shows a gradual improvement in all 3 areas: Reliability, Validity and Usefulness. The graph below illustrates the yearly breakdown, per criteria, of averages for low scores (1 and 2) and high scores (3 and 4). As observed by the difference in heights of the columns, the graph illustrates that with exception of year 2011, different aspects of the reports reliability and validity, measured by sub-questions, on average became higher. Meanwhile the usefulness is rather stably high.

No significant correlation was found between the scores and clusters the projects cover, as illustrated by the Table 2-3 below. However we observed that projects that covered multiple clusters had produced higher quality reports (86 % of total in categories A and B) than projects focusing on a single cluster (70% of total in categories A and B). Table 2-2 Distribution of reports by score and clusters

Clusters Category A Category B Category C Grand Total

Multiple 6 12 3 21

Single among which 4 5 4 13

- Education 1 2 1 4

- Health

1 1 2

- Shelter and NFI 1

1 2

- WASH 2 2 1 5

Undefined 2 2 2 6

Grand Total 12 19 9 40

Figure 2-1 Reports with low/high reliability, validity and usefulness, by year

9

Nor was a significant correlation found between the scores and total budgets of the evaluated projects as illustrated by the Table 2-3. In a similar fashion, with little ground to be conclusive, we observe that the projects with overall budget between 1 and 5 mln have a relatively high proportion of evaluation reports with higher scores (75 - 100 % of total). Table 2-3. Distribution of reports by scores and budget categories

Budget categories, based on total budget of evaluated projects Category A Category B Category C Grand Total

Up to 1 mln 3 5 1 9

1 to 5 mln 2 4 5 11

6 to 30 mln 3 3 1 7

More than 30 mln 1 3

4

Unknown 3 4 2 9

Grand Total 12 19 9 40

Finally, looking at the distribution of reports quality by both: budget category and clusters (see Table 2-4 below), two observations could be added: a) that majority of higher quality reports with smaller budgets (up to 1 mln) cover multiple clusters, and b) the majority of lower quality reports with lower-middle budgets (from 1 to 5 mln) cover a single cluster. Table 2-4 Distribution of reports by category, cluster and budget

Budget/Cluster Category A Category B Category C Grand Total

Up to 1 mln 3 5 1 9

Multiple 2 4

6

Single cluster: Education 1

1

Single cluster: WASH

1

1

Undefined

1 1

1 to 5 mln 2 4 5 11

Multiple clusters 1 1 1 3

Single cluster: Education

1 1 2

Single cluster: Health

1 1 2

Single cluster: Shelter and NFI

1 1

Single cluster: WASH 1 1

2

Undefined

1 1

6 to 30 mln 3 3 1 7

Multiple 1 2 1 4

Single cluster: Education

1

1

Single cluster: Shelter and NFI 1

1

Single cluster: WASH 1

1

More than 30 mln 1 3

4

Multiple 1 3

4

Unknown 3 4 2 9

Multiple 1 2 1 4

Single cluster: WASH

1 1

Undefined 2 2

4

Grand Total 12 19 9 40

10

For the lower quality reports with budgets from 1 to 5 mln that cover a single cluster, further analysis of the scores per sub-question revealed that their main focus is on usefulness (i.e. higher scores on clarity of evaluation goals, that are answered fully by clearly presented conclusions) and less on validity (i.e. lower scores on the use and adherence to international standards, selection of appropriate evaluation methods and description of the evaluation process). A hypothesis could be put forward that projects with a focus on a single cluster might produce more reports that assume rather than explain methodological choices and therefore, need adjustments when assessed with the quality evaluation grid (see concrete suggestions in Table 2-5). This hypothesis cannot be tested within this meta-evaluation but is potentially flagged for further research/follow up meta-evaluations.

2.3 Overview of scores

Category A From the 12 reports in this category, five had contribution up to 1mln Euros from SHO into their total project budget, two – between 1 and 5 mln Euros, one – around 15 mln Euros. Six out of 12 cover multiple clusters, four cover a single cluster: one Education, one Shelter and NFI and two – WASH. More than 75% of these reports scored highest on the following sub-questions: Sub-question (criteria) Basis of high scores

Q 16: (Usefulness) Applicability of lessons or recommendations

Specific, practical recommendations clearly within the sphere of influence of the INGO that commissioned the evaluation

Q 15 (Usefulness): Connections/ linkages/ coherence The reported is well structured to follow the logic of recommendations link to the main conclusions

Q 12 (Reliability): Substantiating of conclusions with findings

Report presents conclusions with sufficient evidence, both qualitative and quantitative to support them

Q 2 (Validity): Research questions and evaluation criteria (EHA)

An adequate selection from standard criteria of humanitarian action evaluation as defined by OECD/DAC is made and translated into research questions

Around 25% of these reports scored lowest on the following sub-question: Sub-question (criteria) Basis of low scores How to improve

Q 3 (Reliability): Positioning of the subject of the evaluation in the policy/institutional context

The report neither introduces relevant political context, nor does it provide analysis of the affected area and population

Briefly describe the main facts about disaster and response are, including analysis of the research done previously, if any

Category C Four out of nine reports in this category had a contribution between 1 and 2 mln Euros from SHO into their total project budget, three others - below 1 mln Euros. Three out of nine cover multiple clusters, four – a single cluster: one WASH, one Shelter and NFI, and Health and one Education. 55% of these reports scored higher on the following sub-questions: Sub-question (criteria) Basis of high scores

Q 16 (Usefulness): Applicability of lessons or recommendations

All recommendations are within the influence of INGOs; they are practical and applicable

Q 13 (Usefulness): Clarity of the research goals The purposes of evaluation are clearly explained

Q 2 (Validity): Research questions and evaluation criteria (EHA)

An adequate selection from standard criteria of humanitarian action evaluation as defined by OECD/DAC is made and translated into research questions

Q 6 (Validity): The needs and livelihoods assessments that informed the intervention

A comprehensive and clear analysis of the humanitarian crisis, including key events and facts is presented

Q 12 (Reliability): Substantiating of conclusions with findings

Conclusions are well supported by the findings

More than 30% of these reports scored lowest on the following sub-questions:

11

Sub-question (criteria) Basis of low scores How to improve

Q 8 (Reliability): Explanation of research process

No description of the process Describe and explain the evaluation process, including potential adjustments to the initial set-up

Q 11 (Reliability): Quality assurance through internal or external supervision

No mention of such any group (i.e. reference or steering) that was involved in quality assurance

Describe what arrangement are made for the internal or external quality check of the design and / or implementation of the evaluation

Q 4: (Validity): Appropriateness of the overall evaluation methods

Only a list of methods without description nor explanation for their selection

Justify selection of used research methods, describe their appropriateness, relative to the evaluation's primary purpose, focus and users

Q 5 (Validity): Evaluation constraints No constraints mentioned in the report

Outline key constraints to carrying out the evaluation and their effects, e.g. available time, respondents, sample size, other

Q 17 (Usefulness): Accessibility of the research results

The report structure is confusing, errors in titles, no consistent use of language (French text inch), terms are not defined, therefore remain unclear

Clearly define the core of the research, especially the main findings. Provide a short summary of main chapters

2.4 Overview of scores: quality evaluation grid

Finally, an overview of the highest and lowest score for all evaluations is presented below The sub-questions the consistently of which got high scores: Q 2: Research questions and evaluation criteria Q 6: The needs and livelihoods assessments that

informed the intervention Q 12: Substantiating of conclusions with findings Q 14: Research questions answered by the

conclusions Q 16: Applicability of lessons or recommendations

The sub-questions consistently of which got low scores: Q 5: Evaluation constraints Q 9: Representativeness of a sample

and/or case studies Q 11: Quality assurance through internal or

external supervision

On top of this short overview, detailed explanations and recommendations for each of the questions are provided on the following page. The latter is concluded with a recommendation on how to adjust the quality evaluation grid.

12

Table 2-5 Comments and recommendations for sub-criteria, quality evaluation grid

Criteria and sub-questions Reports with low scores

8

Reports with high scores

9

Evaluators’ comments on the question Recommendation

A Validity (Does the research measure what it intends to measure?)

1 Positioning of the subject of the evaluation in the policy/institutional context

10 30 Most of the time the purpose of the evaluation is implicitly defined and the context is briefly but adequately described. However, in most of the reports the uses and users are not clearly defined and communicated

Keep the question and aid the use by specify in the ToR that an evaluation report should define users and uses of the evaluation. The reports would read with two main elements being more explicitly described

- the purposes of the evaluation (e.g. lessons for similar projects, accountability, recommendations for improved continuation of the project, dialogue with national and international stakeholders, feedback to donors, other) and

- the users of the reports (e.g. donors, national or international NGOs involved in humanitarian aid, beneficiaries, management, other)

2 Research questions and evaluation criteria (EHA)

6 34 Most of the reports, including ones in Category C use selected standard criteria of humanitarian action evaluation (defined by OECD/DAC). Relatively fewer reports translate criteria into evaluation questions or link the questions clearly to them

Keep the question and aid use of criteria by defining the selection from the standard criteria in ToR. Use of criteria and respective questions in ToR might ensure that conclusions and recommendations are structured around them and, that way, helps with the overall quality of the report

3 The use of and adherence to international standards

12 28 Most of the reports do not assess the projects against international standards such as the Red Cross/NGO Code of Conduct but imply it. This question is consistently scored low in Category C reports. We observed that explicit explanation of how the standards are used is particularly lacking in evaluation that cover a single cluster.

It can be argued that usage of these standards makes assessments easier and reports more comparative. However, as this meta-evaluation team doesn’t specialize in these standards, we cannot judge how imperative use of them should be. We recommend to keep the question, if its combined with ToRs explicit requests to make assessments based on int. standards

4 Appropriateness of the overall evaluation methods

11 29 Most of the reports, including excellent ones pay less attention to selection of used research methods; almost none reflect on their appropriateness in relation to the purpose, focus and users of the evaluation. In addition, similar to the question 4 above, we observed that methodological aspects are assumed rather than

It is a common practice to define evaluation methods in general e.g. focus group discussions, surveys, in-depth interviews, rather than specify them e.g. Appreciative Inquiry, Outcome Harvesting, etc. Although elaboration on the methods used is useful, comparative analysis of the selected choice might not

8 Scored as “poor (1) and “fair” (2) 9 Scored as “good” (3) and “excellent” (4)

13

specified particularly in evaluation covering a single cluster.

be adding value. Therefore, we recommend keeping the question and combine it with information on the sampling/case studies used (Q 9)

5 Evaluation constraints 14 26 The majority of reports, including excellent ones scored poorly on this aspect.

Although this element is important to understand the context where evaluation took place, it is most relevant for a larger-scale research that makes use of samples and generalizes the conclusions. We recommend keeping it but in combination with the explanation of the evaluation process (Q 8)

6 The needs and livelihoods assessments that informed the intervention / Intervention objectives

6 34 Most of the reports describe concise and adequately the context, history of the disaster, main facts and figures and challenges faced. Only part of them pays attention to institutional context, relevant to the sector and mentions previous evaluations/research undertaken by the government or other relevant actors

Keep the question and reinforce it’s quality by requiring the following:

- The use of previous evaluations/research as a background (desk study)

- An analysis of institutional set-up in the sector

- A balanced picture of challenges and progress so far

7 Cross-cutting issues 9 31 While assessing this aspect, this meta-evaluation considered all cross-cutting issues together: a) gender equality, b) protection, c) capacity building, d) advocacy and e) vulnerable and marginalised groups (elderly, disabled, children, HIV/AIDS sufferers, other). Whereas these issues were not tackled to the same extent.

We recommend keeping the question and expanding it further in sub-questions, relevant to the evaluated project throughout the intervention and the effect on the intervention.

B Reliability (How rigorous are the evaluation results?)

8 Explanation of research process 12 28 Most of the reports, including excellent ones pay less attention to describing the evaluation process and adjustments to its initial set-up and arrangements

An important aspect, however better be combined with Q 5, see comment above

9 Representativeness of a sample and/or case studies

17 23 Most of the reports scored relatively poorly on this aspect, understandably as only few reports used sampling techniques and none - case studies.

An important aspect, however specific to the evaluation type and therefore, best to be combined with Q 4, see comment above

10 Verification of findings/triangulation

11 29 No major observations Keep the question

11 Quality assurance through internal or external supervision

17 23 This aspect is one of the two questions that scored poorly in majority of reports, including excellent ones. On a few occasions, the quality assurance set-up was described in ToRs, but not followed up in the report. We assume there was more quality assurance than documented, as description of this element is not a

Remove the question. Assure quality with external or internal quality supervision and describe it in accompanying documents such as ToR and SHO quality assessment form (Verslaglegging Evaluaties SHO Projecten)

14

traditional part of the evaluation report. In addition, similar to the question 4 and 5, we observed that quality assurance is assumed rather than specified particularly in evaluation covering a single cluster.

12 Substantiating of conclusions with findings

5 35 One of the highly scored elements in majority of reports, including ones in Category C.

Keep the question

C Usefulness (how applicable are the evaluation results?)

13 Clarity of the research goals 7 33 Highly scored in excellent reports, we can assume that clarify helps overall quality as it aids the reader in following the evaluators logic of narration

Keep the question

14 Research questions answered by the conclusions

4 36 Almost all reports performed well on this sub-criteria, no further comments

Keep the question

15 Connections/linkages/coherence 9 31 This element is scored well even in poor report Keep the question

16 Applicability of lessons or recommendations

3 37 Almost all reports performed well on this sub-criteria, no further comments

Keep the question

17 Accessibility of the research results

7 33 Most of the reports reflected well the core of the research, especially the main findings. Although excellent reports scored relatively low on this, the sub-criteria is relevant to keep

Keep the question

In conclusion, to better suit the SHO / humanitarian nature of projects and evaluation, the meta-evaluators recommend to adjust the list in the following way:

A Validity (Does the research measure what it intends to measure?) 1 Positioning of the subject of the evaluation in the

policy/institutional context The report refers to relevant political context, assumptions and institutional settings of the subject of evaluation. It provide analysis of the affected area and population (including relevant historical, social, economic, political and cultural factors)

2 Research questions and evaluation criteria (EHA) The report outlines research questions and defines which standard criteria of humanitarian action evaluation (defined by OECD/DAC) is applied to assess the evaluation subject

3 Explanation of evaluation process including constraints

The report describes the evaluation process, including potential adjustments to the initial set-up and arrangements as well as outlines key constraints to carrying out the evaluation (e.g., lack of time, difficult travelling conditions, lack of baseline data, poor agency monitoring systems, lack of access to key information sources, difficulties setting up control groups, use of translators), and the effect of these constraints.

15

4 The needs and livelihoods assessments that informed the intervention / Intervention objectives

The report provides an analysis of the needs and livelihoods assessment practices and their effect on the intervention as well as a clear analysis of the crisis, including key events (and a chronology where appropriate). The report assesses the relevance of the intervention objectives to the contextual analysis and needs/livelihoods assessments

5 Cross-cutting issues The report analyses consideration given to the following cross-cutting issues: a) gender equality, b) protection, c) capacity building, d) advocacy and e) vulnerable and marginalized groups (elderly, disabled, children, HIV/AIDS sufferers, other) throughout the intervention and the effect on the intervention.

B Reliability (How rigorous are the evaluation results?)

6 The use of and adherence to international standards The report assesses the intervention against appropriate international standards such as the Red Cross/ NGO Code of Conduct

7 Appropriateness on the overall evaluation methods, including sampling and/or case studies

The report defines employed research methods and techniques, reflects on their appropriateness relative to the evaluation's primary purpose, focus and users and explains (with the use of known sampling rules and techniques) how conclusions from the researched sample and/or case studies are valid / generalized for the entire researched population

8 Verification of findings/triangulation The report outlines how data have been checked and cross-checked with the use of different methods and (secondary) sources

9 Substantiating of conclusions with findings Degree to which the conclusions were substantiated with findings

C Usefulness (how applicable are the evaluation results?)

10 Clarity of the research goals Clarity of purpose of the evaluation, for which the research results will be or are used

11 Research questions answered by the conclusions Comprehensiveness to which all evaluation questions were addressed. Together all aspects were covered.

12 Connections/linkages/coherence Recommendations should follow on from the main conclusions and reflect consultation with key stakeholders.

13 Applicability of lessons or recommendations Practical applicability of the presented recommendations and the degree to which these are within influence of the those in charge, especially the commissioners of the evaluation

14 Accessibility of the research results Clarity and completeness with which the evaluation report and its summary reflect the core of the research, especially the main findings

This adjusted list has one sub-questions fewer (Quality assurance through internal or external supervision), merged 2 questions on Validity and 2 questions on Reliability as well

as shift of one question (the use of and adherence to international standards) from Validity criteria to Reliability.

16

3 Assessment of the results of SHO's use of the funds

3.1 Assessment of the results: process and limitations

A. On criteria: As was mentioned in the introductory chapter, the evaluation reports of the individual participating organisations were "likely to have different objectives, methodologies, focus, lay-outs and depth". In most reports there were distinctive sections dealing with one of the DAC criteria. However, not all reports covered all criteria. In NLRC and UNICEF reports (who used internal formats), DAC criteria were partly covered, or implicitly and therefore, the information about criteria had to be searched throughout the document, which made the exercise rather time-consuming. In some of (all) the reports conclusions and/or recommendations were missing, in many reports findings and conclusions were the same; in some reports effectiveness and efficiency were used interchangeably. In few reports (e.g. in several Real Time Evaluations) the majority of standard criteria such as impact, sustainability, coverage, coherence, appropriateness and efficiency were not relevant as reports mainly dealt with organisational / institutional aspects of the response rather than with project activities.

The limitation introduced to the meta-evaluation by such unsystematic use of standard criteria is that the issues not addressed in the reports (such as impact, sustainability, strengthening of responses, gender or cooperation) could not be assessed. Another limitation is related to the fact that the projects were diverse (single sector focus or multi-sector focus, varying outputs in same sector-projects, different degrees of participation in clusters, different project-approaches, more or less people-focused or sector focused etc.), and that the ToR of their evaluation were not standardized. This made it hard to compare projects on the basis of these evaluations

10, and thus difficult to summarize since one risks either to regroup features which are not

exactly of the same nature, or not to give due respect to the subtleties or specific added value of the projects.

B. On consultations: In addition to the desk study, one written interview and three focus group interviews were organised with SHO key stakeholders from the participating organisations in the Haiti Earthquake Response. They provided clarifications on different issues such as working with local (partner) organisations, the cluster-organisation (UN), adherence to Standards, internal agency evaluation and involvement and opinion on project-evaluations. In addition, these key stakeholders were consulted on the third evaluation question with respect to the presentation of the Haiti Earthquake Response results by SHO. The latter has also been the major subject of the interview with two staff-members involved in the accountability aspect of the Haiti Earthquake Response of the Ministry of Foreign Affairs.

C. On other documentation: Little use has been made of the other documents produced by stakeholders in

the Haiti Earthquake Response process, in particular those of SHO, Algemene Rekenkamer (National Court of Audit), various comments on each other's reports, the policy comments by the Minister of Foreign Affairs

11, and the most recent IOB

12 desk study on Humanitarian Assistance to earthquake

victims in Haiti (in statu nascendi). They are neither summarized nor used in any other way in this meta-evaluation, which is explicitly concerned with the evaluation reports produced by independent consultants. The main reason for this approach is to avoid repetition and overlap. In addition, as financial accountability issues are covered by the ARK (Rekenkamer, Dutch Court of Audit), they are merely referred to their reports. Finally, the scope of this meta-evaluation (i.e. time and resources available) as described in the ToR narrowed down the focus of the evaluation from the beginning. It is assumed that this evaluation produces results that are complementary to those presented in the reports cited.

10

One representative of one of the participating organisations in the Haiti response called comparison of projects the comparison of "apples and pears". 11 Most of the cited documents can be found on Internet. 12 Policy and Operations Evaluation Department of the Dutch Ministry of Foreign Affairs.

17

D. On value judgement: since this is outside of the scope of this evaluation, no value judgement is passed on the quality of individual organisations or projects; nor on whether all the organisations funded by SHO did sufficient work, nor what their efforts have meant for Haiti after the response had been accomplished.

3.2 Main conclusions of the evaluations of responses to the earthquake: synthesis

The interventions funded by SHO were very variable in terms of sector involvement, scope, amount of funding and duration. The regular SHO partners have received the bulk of the funding, with Cordaid more than 25%, ICCO, NLRC, Oxfam Novib Unicef each between 10 and 20 %, Save the Children, Terre des Hommes, Tear Fund and World Vision each 1,5 or 3 %. The so-called guest organizations: Care Netherlands, Dorcas, Leger des Heils, Plan Nederland, VNG International and Habitat for Humanity received each around 1%. Two organisations: Tear Fund and Dorcas, only provided emergency relief for a relatively short period till end 2011; three organizations: Cordaid, Oxfam Novib and Terre des Hommes till the end of 2012; five organisations: NLRC, UNICEF, ICCO/KiA, Leger des Heils and VNG International till end of 2014. Most organisations were active in more than one sector / cluster, except Plan Netherlands, Habitat for Humanity and VNG who were active in only one sector. ICCO/KiA, UNICEF, Tear, Terre des Hommes were involved in 5 to 8 sectors, the rest in 2 - 4 sectors

13.

Table 3-1. Overview of participation in main clusters by SHO-partners

Clusters SHO partner participation

Shelter and NFI Cordaid; ICCO KiA; NLRC; World Vision Tear Fund; Habitat for Humanity, Christian Aid, UNICEF; Oxfam Novib; Act Alliance

WASH Care; ICCO KiA; Oxfam Novib; NLRC; UNICEF; Tear Fund; Terre des Hommes; Dorcas; VNG Int.; World Vision; Christian Aid; Save the children, Leger des Heils; Act Alliance

Food security ICCO KiA; Oxfam Novib; Tear Fund; World Vision; Christian Aid; Act Alliance Healthcare Cordaid; ICCO KiA; UNICEF; NLRC; CARE NL; World Vision; Christian Aid; Act Alliance Education ICCO KiA; Save the Children; Tear Fund; Leger des Heils; UNICEF; Plan NL Protection Care; ICCO KiA; UNICEF; Oxfam Novib; Act Alliance; Save the Children; Christian Aid;

NLRC; World Vision

In SHO's 6

th joint report (13 January 2010 - 31 December 2013) results per cluster were presented (at that time

92,6% of the funds were spent). In Annex 2 of this SHO Joint Report (pp. 42-66) an overview of the results per 31 December 2013 is given, grouped per sector and SHO participant and compared with the planning for reconstruction for the period 2010 - 2014. SHO members, before receiving funds, handed in an individual plan and proceeded to implement the projects. SHO Members coordinated their activities through the clusters. This relatively autonomous space for manoeuvring for the members is a part of the specific character of the SHO Haiti Earthquake Response. Participant organisations didn’t see value in fixing joint targets for the programme as a whole. In that sense, SHO did not intend to have a corporate identity in Haiti. As for attempts to accumulate results in reports by SHO in the Netherlands, according to some stakeholders from the participant organisations, they felt like "adding apples to oranges" as each of the projects had its own character and particularities. The latter remained another part of the character of the SHO Haiti Earthquake Response.

3.3 Overview of performance by criteria

The following sections will give an overview of the performance of the response interventions by the SHO participants. This is largely based on the evaluation reports commissioned by the SHO members themselves. The sections focus on strong and weak points of the disaster response and what can be learned from it.

13 The sectors are Camp Coordination / Management, Early Recovery, Education, Emergency Shelter and NFI, Emergency Telecommunications, Food Security, Health, Logistics, Nutrition, Protection, Water Sanitation Hygiene, Child Protection, Gender Based Violence

18

The sub-chapter-classification follows the evaluation criteria prescribed by the SHO commissioner of this meta-evaluation. Statements are repeated in different sections of the chapter as the criteria are mutually linked rather than mutually exclusive, e.g. efficiency might be related or influence effectiveness; relevance might have the same effect on sustainability and effectiveness. This chapter ends with a summary of findings. The findings repeatedly mentioned in the evaluation reports, either as good practices, challenges or recommendations, will be elaborated upon in the concluding chapter of this report.

Relevance/appropriateness

Was the response tailored to local needs, enhancing ownership and accountability? With regard to the SHO response as a whole, positive experiences are captured in the short run combined with critical feedback in the long run. On the whole, people were satisfied, especially in the relief phase, when essential measures were taken and support given to quickly reduce suffering by earthquake victims and satisfy their immediate needs. Factors contributing to successful interventions meeting the needs of the local beneficiaries (which led to expressions of gratitude) were mentioned, just like good practices which proved appropriateness of interventions, positively contributing to ownership. However, as the response continued, reports captured factors that constrained or decreased relevance in the longer run. Let us review the various statements, remarks and suggestions contained in the reports. A convincing example of an intervention responding to primary human priorities was an education project by UNICEF

i. The significance of the reopening of schools 3 months after the disaster was immense for both the

affected communities and the government. It created the feeling of momentum towards recovery. The quick and effective employment of global tools in the first couple of months

ii also contributed to the relevance of the

interventions. Whether the response was tailored to local needs and enhanced ownership appeared first and foremost dependent on the degree of participation of the Haitian people, communities and government. The readiness to have Haitians themselves lead the search and rescue and care for the vulnerable in the early phase of the disaster response created a sense of belonging to a community and reconciliation among sections of Haitian society. The second main factor playing a role in tailoring response to local needs was a good knowledge of the context. ACT Alliance is reputed to have this good knowledge

iii. In every phase of the response it contributed to

success when needs assessments in affected areas were carried out by or with help of local communities, such as with the support by local youth clubs

iv. This approach not only increased the appropriateness of the use of

materials and technical solutionsv, but also encouraged voluntary participation and commitment by young

people. The use of local resources, in terms of labour and material also led to pride and ownership as well as to short-term stimulus of the local economy. The employment of committees of beneficiaries helped to manage power dynamics

vi, functioning as a feedback mechanism at the same time.

A final element contributing to relevance of the response was related to the interaction and communication (e.g. sharing of project information) between the projects and the beneficiaries so that the latter knew what to expect. Most of the factors weakening the relevance of the response were reportedly related to insufficient knowledge of the Haitian society and culture by the NGOs, insufficient transparency and comprehensiveness which damaged trust, poor participative management and feedback mechanisms, and lack of usage of local resources including expertise, materials, water and transport systems. All the above-mentioned factors restricted the degree of accountability by beneficiaries, which had also consequences for the sustainability of realized constructions such as schools and water systems. Accountability depended on the degree of ownership of the interventions. SEED-SCALE project

vii is an example of the bottom-up build up of community-NGO-state

organisational alliances that resolve community problems and empower inhabitants, especially women.

19

Effectiveness and impact

Were objectives and targets realized? Were outputs used and maintained by beneficiaries? Do the realizations have a wider effect on the Haitian society? Reporting on the response to humanitarian crisis is prone to focus on outputs: donors and public demand accountability with quick results and visible, understandable, easy to communicate information. Therefore, agencies tend to focus on quantitative data on deliverables rather than how beneficiaries used realizations (like shelters and schools, and WASH facilities etc.), appropriateness of design, control of quality and maintenance and degree of ownership, which concerns the use of outputs or outcome

viii. SHO has published an infographic

(see below), aimed at the Dutch public that illustrated the main common SHO-realizations of the programme.

Figure 3-1 Infographic published by SHO on Haiti Earthquake Response

Having said that, the evaluation reports including the SHO final report do pay attention to the use of outputs and, as a result, changes in people’s lives. Most reports mention that projects were effective, especially in the relief phase, meaning that the most urgent needs for the victims of the earthquake in terms of shelter, drinking water, food, cash, NFI

ix, health and

sanitation were satisfied. International organizations such as UNICEF were able to mobilize their response quickly, with a strong support from their HQ, flexible funding, disaster preparedness, the presence of experienced staff, local capacities and leadership and pre-stocking of emergency items, which enhanced effectiveness. The availability of large contingency funds (instead of finite budget structures) allowed adapting programmes more easily

x. On the contrary, delays in funding or insufficient funding prevented timely

realizations and satisfaction of complementary needsxi. Generally, strict bureaucratic approaches, too strict

procedures and protocols have reduced flexibility and effectivenessxii

.

“The best way to guarantee relevance and appropriateness of interventions is to be a learning organisation, to be analytical, reflective and willing to learn through unconventional approaches, and to be responsive to new priorities; to advance a culture of learning” (World Vision)

20

A general observation in many reports concerned the inexperience of all organisations with the urban character of the disaster, which made them face many challenges, especially those related with space, land use and title deeds. The shelter programmes, for instance, have had to deal with threats of evictions. As the government failed to issue a moratorium on evictions, beneficiaries and NGOs were helpless against powerful landlords. Timely availability of housing solved looming eviction from displacement camps

xiii .

Because of inexperience with the urban nature of the disaster much had to be learned in a short period. Where constant monitoring and evaluation took place this supported individual and collective learning. Standard solutions, design and approaches appeared to need flexible adaptations. In shelter programmes the learning concerns the need to use modular structures

xiv so that beneficiaries can more easily adapt them to their needs

on the basis of their family size and cultural norms. Shelter programmes cannot be planned in detail and must have a built-in flexibility to allow adaptations. This becomes difficult for organisations that had opted for single technical designs and large upfront purchase orders. Because of adaptations, and of rising market prices the cost of shelters increased so that the number of realizations was less than planned. However, the evaluation

xv

report argued that in general the shelter programmes14

were effective. In health, working with packages of psychological, social and material assistance, and training of community-level health workers in psychology and mental health have had a positive outcome

xvi. Unfortunately, the health

workers later could not find employment in the Healthcare system and went to look for other jobs. Effective health promotion has led to positive change in health-seeking behaviour

xvii. The long experience that ACT

Alliance had in the country, and consequently the large networks of organisations and individual resource persons they could dispose of considerably contributed to the success of their WASH and shelter programmes. Income generating projects learned that the majority of beneficiaries had no interest in working on a cooperative basis but preferred business on individual basis (for which grants were insufficient

xviii . In some

WASH projects effectiveness was high due to the use of excellent detailing and material as well as the appropriate choice of technical solutions, but in other cases design problems compromised good operation and maintenance

xix. Safe water messaging by multiple organisations following the cholera epidemic was generally

effective. Impact of the response (in this stage) was difficult to measure

15. Some examples were given of probable

positive impact on the Haitian society: increased school attendancexx

and social solidarityxxi

; a behaviour change in sanitation and hygiene due to mass messaging after the cholera outbreak by well cooperating NGOs; restart of the local economy

xxii and steady employment availability for trained construction workers

xxiii. Among

negative impact were mentioned increased indebtedness caused by micro-finance, sexual harassment of young girls in camps, on-going psychosocial difficulties and INGO activities weakening the position of government.

Efficiencyxxiv

Was the use of funds cost-effective? Were resources employed in an optimally efficient way? It must be noted from the very beginning that all expenditures were accounted for by recognised accountant firms in Haiti and by the ARK (Rekenkamer). Cost -effectiveness or value-for-money was not studied with hard calculations, rather it was judged positively or neutrally, as for instance in statements that costs and expenditures (such as schools and WASH facilities) were comparable with those of other organisations. Having said that, the various statements in the reports allow for conclusions about efficiency indirectly. A number of reports state that efficiency in the emergency response operations was particularly achieved with flexible approaches and use of local resources. Indeed, easy disbursement of funds by HQ

xxv, avoiding

14

The Cordaid programme represented about 25% of the SHO funding 15 A World Bank study (... /country/Haiti/overview) mentions that the Haitian economy has recovered since the earthquake - although it remains the poorest country of the American continent.

21

bureaucratic procedures facilitated the fast and efficient functioning of implementation chains16

. Working with local partners, using existing structures, resources, know-how and Haitian staff and the use of volunteers and cash-for work and community work gave good value for money

xxvi . A large network and excellent relations

xxvii

enabled to make maximum use of local resources. However, generally, local mechanisms that worked well in assisting disaster-affected populations including private one's (e.g. private transport systems, cash transfers from diaspora and mobile phone networks and food-selling by street vendors) were ignored. One of the evaluation reports

xxviii mentions that some of the imported relief items (such as bottled water, hardware

materials and pharmaceuticals) and services were also available locally and could have been utilized if there had been a better contextual understanding. Several reports argued that expensive activities

xxix or overspending of budgets (caused by unforeseen

community needsxxx

, were justified and contributed to efficiency in the first phase of the emergency. But, in the course of time, the lack of a clear strategy, "scattered" approaches (of fundraising), insufficient and ineffective monitoring and an irregular cash flow to partners

xxxi, lack of an inter-sectoral planning and strategy created

duplication and inefficiencyxxxii

. An example is continued expensive water trucking while other water provision solutions could have been found

xxxiii.

Major constraints to efficiency were spiralling prices, lack of streamlining in administrative procedures

xxxiv,

bureaucratic, slow and inflexible recruitment proceduresxxxv

, long and/or inadequate procurement processes and limited delivery designs

xxxvi. Two evaluation reports

xxxvii recommended to be realistic about budgets and

timeframes, and to include risk assessments in programme planning. A remarkable finding

xxxviii talks about a strong culture of risk-avoidance rather than risk management in large

organisations such as UNICEF. This is combined with a predominant culture of procedures-over-results, which works against the need for a timely and efficient response. This evaluation report states that "staff by and large loath to take risks, fearing more sanction for disregarding procedures than for taking well-intended risks that might yield good results". Other issues that restricted efficiency in operations are lack of knowledge of the local language and high turnover of staff, a persistent problem in Haiti. The latter has resulted in loss of knowledge, among others, about local partners and led to frustrations.

Sustainability and Connectedness of the SHO earthquake response interventions

To what degree is it likely that the results of the response have a lasting effect? Have long-term and interconnected problems been taken into account? It has to be noted from the beginning that typical emergency processes and logistics are hardly appropriate for projects aimed at sustainable results

xxxix. A number of reports stress that the best guarantee for long-lasting

results is to combine, right from the beginning, emergency and development support. For this, the adoption of a holistic approach is needed, that does not only address the most immediate needs, but also considers long-term solutions, e.g. for housing, WASH, livelihoods, psychosocial support, education and other dimensions of family life

xl. Several organisations have acted with such a perspective or vision in mind

xli and have taken

initiatives that seem to have been favourable starting points for sustainable achievements. Terre des Hommes

xlii proposes to spend 20% of the funds on immediate emergency response, 70% on reconstruction and

10% on development projects. Three sets of concepts keep coming back in the reports: 1) ownership, 2) embedding and capacity development, and 3) advocacy and phasing-out/handing-over/follow-up. Ownership. More sustainable projects are anchored in the institutional context in order to ensure legitimacy and ownership, requiring a fruitful exchange with the government and a good knowledge of legal, social and cultural contexts. Inclusive monitoring & evaluation mechanisms are crucial for sustainability

xliii. Empowering

beneficiaries to take over is crucial as wellxliv

.

16 The implementation chains as such were sometimes very long, with four or five links, and were in itself no examples of efficiency.

22

Capacity building. Close coordination and capacity building in all counterpart institutions, especially national and local government and partner organisations

xlv is another pre-requisite for sustainability. Reports

xlvi mention

examples of capacity building throughout school projects (such as the training staff in maintaining upgraded schools according to operating guidelines) as well as embedding schools in the community. This enabled to develop accountability mechanisms and monitoring systems in schools and thus benefitted the school organisation. A lesson learned here is that capacity building does not stand-alone, it requires proper embedding as illustrated by the case of trained community health workers who could not be integrated in the government health system

xlvii.

Advocacy and phasing out. Still, the challenge is that neither beneficiaries nor government are able to fulfil financial obligations for maintenance

xlviii, e.g. WASH facilities due to lack of income by female-headed

householdsxlix

. Further, the provision of free medical services enabled by humanitarian aid seemed to frustrate the private sector and the government who felt that this undermined the long-term health provision in Haiti

l.

These challenges are linked to the third pre-requisite for sustainable response – employment of adequate advocacy strategies that seek long-term government-led solutions, especially in the education and health sectors. Moreover, advocacy is to be coupled with the correct phasing-out, exit strategy, handing over and follow-up

li .

Absence of these elements e.g. the lack of accompanying measures and follow-up in the livestock programme is regretted

lii. Where agencies

liii did provide for follow-up funding to continue raising public awareness,

accountability for maintenance and assuring the quality of drinking water in WASH projects, it has been successful. A proper phasing out and follow up is naturally linked to ownership and capacity building: close cooperation with local government and local organisations enables strong community ownership after the halt of humanitarian aid, and hence continuation of activities. Obviously, "connectedness" is strongest when organisations have (directly or through partners) extended networks and relations and good knowledge about the society and culture in which they operate.

Adherence to standards/Coherence

Did the response take into account humanitarian and human rights considerations? Did the response adhere to Code of Conduct or Sphere standards? The subject of adherence to international standards is neither systematically treated in all reports nor was a clear difference always made between kinds of humanitarian standards. However, its importance is illustrated by a recommendation from one of the evaluation reports

liv, that all agencies' staff should be introduced to and

understand the Sphere Humanitarian Charter and Code of Conduct in Disaster Relief, to familiarize themselves with the role and responsibilities of INGOs, the UN and the government. The way in which realisations are presented, valued and commented upon in the reports, including in sections of lessons learned and recommendations, it can be concluded that the implementing organisations, in general, paid attention to, and took into account humanitarian standards. In one of the reports

lv adherence and

promotion of Sphere standards (with emphasis on protection and psychosocial aspects), to HAP17

standards and national standards is explicitly treated. In other reports

lvi much stress is laid on behaviour, especially the

application of Christian values such as dignity and respect without mentioning Code of Conduct or Sphere. However, some reports mention the non-adherence to Sphere standards (transitional shelters

lvii, schools

lviii)

and how the pressure under which organisations had to work sometimes prevented strict adherence to rights-based approaches

lix. One report

lx mentioned that Sphere standards were known, but not strictly applied,

adhering instead to own quality standards. Despite that, the evaluation suggested that strict application of Sphere standards could have increased ownership. All UNICEF projects adhered to own standards that were (as

17 Humanitarian Accountability Partnership Standard in Accountability and Quality Management

23

much as possible) based on national standards. In one of the evaluation reportslxi

it is argued that Sphere standards should be used as indicative and not as absolute standards, and that indicators need to be contextualized in consultation with beneficiaries. Further, one of the reports

lxii states that newly hired Haitian staff has little or no experience with the NGO

sector or with the policies, norms and practices within their agency. In fact, many have little or no familiarity with international humanitarian standards. Possibly to respond to this, one of the SHO-supported projects had a specific objective to promote the application of Sphere standards

lxiii. Notably, NGOs were more interested to

participate in such training than international organisations.. Taking a closer look at the contents of the Code of Conduct, it can be concluded that most of the principles were adhered to. Particularly discussed in the preceding section (i.e. on sustainability and connectedness) are principles such as building response on local capacities, participation of affected communities, reducing future vulnerabilities, the respect of the local social and cultural context, etc. The relevant aspects in this context may be recapitulated as follows:

the use of local human and material resources, standards and procedures,

working with local (partner) organizations or local offices with extended local networks and contacts,

recruitment of local staff,

involvement of local communities in planning (needs assessment, selection of beneficiaries), implementation (volunteers, cash for work) and monitoring;

communication with beneficiaries and feed-back mechanisms (accountability towards beneficiaries);

strengthening capacities of beneficiaries, communities, local government and higher institutions;

respect for local culture (especially in shelter, housing and WASH projects);

adapting the response to the context (inexperience with urban nature of the disaster).

Adherence to cross-cutting themes

Gender: was gender considered in the agencies' emergency assessment and implementation? Vulnerable groups: were the special needs of vulnerable groups considered in assessment and implementation? Although not to a large extent but reports do assess gender aspects of interventions. On the positive side, one of the reports

lxiv mentions the adaptation of design of latrines and showers ("to provide more privacy, dignity

and security to women"), anotherlxv

notes that "every effort has been made in the field to avoid gender inequalities" including taking into account children (unaccompanied, with disabilities). On the critical side, few reports (LdH, VNG) note insufficient evidence of integrating gender-related issues. The subject of gender violence became a hot issue with the presumed "rape epidemic" in 2011. Care researched the subject and produced a comprehensive report on gender issues. Its main conclusion with respect to rape was: "rape occurs far less frequently than we have been led to believe by a series of high profile reports from specific grassroots gender based activist organizations and activist-scholars and journalists working in Port-au-Prince"

lxvi.

Because of allegations and rumours about the extent of gender violence, organisations were pushed into a defensive position. While one report

lxvii mentions sexual harassment of young girls in camps, another

lxviii states

that the extent of gender violence is not known, while Care lxix

stressed gender violence but talks of misinformation about the "rape epidemic". In its evaluation report, Leger des Heils was recommended

lxx to

include and integrate gender-sensitive issues in the latrine design by adding locks from inside, and separating toilets for girls and for boys. Only in one report the issue of "empowerment" of women was mentioned, as a recommendation

lxxi: focus on mobilizing women to get more involved in the one area where they are

conspicuously absent: local level politics and community development. This recommendation is, without detailed elaboration, revealing with regard to the gender-based approach of certain disaster response actions. It may be concluded that in disaster response such as was taking place in Haiti, gender violence took place (the scale being less obvious) and should have been an important point of attention, especially in shelter and WASH programmes. Probably this was indeed the case but the subject was insufficiently elaborated upon in the

24

evaluation reports, which may be due to the report formats, templates or questions raised in the Terms of References. Vulnerability is more frequently mentioned in the reports. In some cases vulnerable people were explicitly selected, as in the case of ICCO

lxxii where 22 most urgent cases were selected in a participative way

18 or, more

implicitly, in the Tear Fund reportlxxiii

which states that vulnerable groups seemed to be targeted by partners with varying degrees of transparency. In some projects

lxxiv vulnerability was systematically taken care of in all

actions. Oxfam's activities also focused on actors who worked with the most vulnerable peoplelxxv

. Several reports mentioned the difficulties in dealing explicitly with the most vulnerable. In the Joint ICCO Christian Aid Real Time Evaluation

lxxvi it is mentioned that the overwhelming number of people in need of

immediate help, combined with limited manpower and time resources, meant that it was next to impossible to systematically define, let alone prioritize who were the most vulnerable victims. The Cordaid Shelter Programme Evaluation Report mentions legal frameworks and policies (in an environment of huge inequities and extreme poverty) to which the shelter program was bound, which contributed to the exclusion of many vulnerable people. Between the lines sometimes the issue of stigmatization of disabled victims, children and those with HIV

lxxvii is mentioned.

It may be concluded that the crosscutting themes of gender and vulnerability were taken into consideration in the response projects, with variable explicit attention. However, in a number of cases the themes were not systematically dealt with by the evaluators.

Coverage

Did the response reach major population groups facing life-threatening risk wherever they were? Was there appropriate geographical coverage? To start with, it has to be noted that there is (was) no uniform or consistent criteria for site or beneficiary selection

lxxviii. There is, however, a recommendation to do more research on mapping the extent of the

problem henceforward. The coverage assessment varies. Oxfam’s coverage is considered to be very good

lxxix. The Tear Fund

evaluationlxxx

mentions that their partners worked in the geographical areas where it was most needed. Two reports

lxxxi claim that a predominantly urban humanitarian response overlooked displaced people in rural areas.

Another report mentions that there was a lack of information on the rural situation and Internally Displaced People

lxxxii. One of the projects

lxxxiii, differently from other organisations, focused on hidden rural communities.

N.B. even urban response requires a lot of "rural knowledge" as urban dwellers originate from rural settings. In few cases it was mentioned that a combination of pursuit of larger coverage and limited resources led to insufficient achievements: 300 transitional shelters for a community of over 10,000 people

lxxxiv and standard-

amount of distributed cash diminished in order to let more (newly arrived IDP) families also profitlxxxv

. One of the reports

lxxxvi mentions that most of the humanitarian organisations involved in health services left

Haiti by 2011. A few organisations stayed longer like Médecins Sans Frontières, World Vision, Handicap International and a few others, like the French Red Cross. These organizations all covered specific geographical areas without overlap with Cordaid. Another type of coverage concerns the orientation of schools (public/private; Adventist/Catholic). One evaluation report

lxxxvii mentions that different schools, irrespective of their type were covered, thus whole

communities benefitted since state exams were to be held in buildings to which students from other schools and areas also had access.

18 Choice of beneficiaries has obviously not always happened in a participative way. In the TDH evaluation it is stated: "the question in how far it is legitimate that external actors chose beneficiaries remains a delicate question, with the risk of unfairness".

25

Strengthening of responses

Did the response of the SHO agencies strengthen and complement the response of local organisations and coping mechanisms, or hinder them? The Haitian government, as a victim of the disaster, was ill-prepared to deal with the 2010 earthquake, and this affected the response by international organisations. The Care evaluation

lxxxviii speaks of the 'blurred' role of the

government and the humanitarian communities. According to this, many people found that the Haitian government has failed to take the lead and to provide an operational framework for humanitarian action (Cordaid). Various organisations involved government agencies in their response rather than the reverse. On the other hand, in an evaluation report

lxxxix, the idea that there was little local capacity is called a biased

assumption, and an agency is criticized for the limited engagement with the people and cooperation with local authorities. According to another report

xc, Haitian leaders strongly felt that there were existing networks that

could have been better built upon to inform and shape the initial response. Top-down, non-inclusive humanitarian activities have been mentioned before (see section on Sustainability), as well as the suggestion to reduce INGO activity to prevent further weakening of the government, and a need to reinforce government and civil society

xci. Cordaid report

xcii mentioned that it was a challenge to find the balance between the role of

civil society and the state in building the health and welfare systems. On the other hand, there is evidence that most SHO agencies did involve both government and local NGOs/CBOs, especially at local level. The majority worked with local partner organisations with whom they already had previous working experience. These local partners were involved in all stages of the response, from needs assessment to implementation and monitoring, and certainly increased knowledge, skills and capacity with regard to disaster risk reduction and management. Disaster prevention, preparedness and disaster coping mechanisms and risk reduction activities were especially undertaken by Oxfam Novib and NLRC and Tear Fund. Since a number of local partners had prior relations with different formal institutions, this helped the implementation of projects and got fast approvals. Most organisations were conscious of the necessity to increase participation and collaboration in decision making at community and sectoral levels, including with local NGOs, government and private sector

xciii and worked to increase the capacity of local partners and the

local authoritiesxciv

even if coordination between these parties was not always optimal. Almost all reports mention the close collaboration between SHO-partners and communities, community organisations, beneficiary families, existing community structures and local authorities. Communication with these beneficiaries had improved and feedback mechanisms created. During the response the importance to build enduring bottom-up community-NGO-State organisational alliances became clear. Interviews with SHO member representatives showed that nearly all local partner organisations worked together with local government and NGOs. Collaboration on a national level as well as the representation of the Haitian government in UN clusters produces a mixed picture, understandably, as it is more difficult to achieve than on local level.

Cooperation

What was the level of co-operation in the field between the SHO agencies? Evaluations do not report much on inter-SHO collaboration in the field in Haiti. Focus group interviews with SHO member representatives revealed that SHO had hardly an identity in Haiti, and very little cooperation could be organised between them, in spite of an attempt from the Dutch embassy. As SHO didn’t put efforts in profiling itself in Haiti, hardly any local partner of SHO member organisations had an idea what SHO was; if partners of SHO member organisations worked together it was because they met each other in communities or in sector-meetings. This is not unusual since SHO is a coordinated, specifically Dutch funding mechanism. In contrast, much has been said about another cooperation/coordination institution, the UN cluster system, criticised in some cases and found useful in some others. This cluster system started to operate reasonably after three months, but because of the enormous amount of actors and the multiple roles the clusters assumed (such as strategy development, coordination, fund management, implementation), it was rather inefficient,

26

except in the sectors Health and WASH. Notably, in these two sectors (Health and WASH) the government played a bigger role. Most of the clusters were criticised because of their weak overall leadership, the frequent turnover of staff, the overwhelming number of cluster members, the limited participation of Haitian organisations, the non-separation of roles, the lack of knowledge about needs of earthquake victims and language barriers. In the course of time it appeared more efficient to work with "baby clusters", on a more local level, and in cooperation with the local authorities

xcv.

The UNICEF evaluation report

xcvi mentions that the Education cluster's leadership was weak owing to

inadequate staff deployment as well as the inability to clarify roles and responsibilities between UNICEF and the other co-lead, Save the Children. The report recommends to embed clusters within technical ministries, and further, to develop a better coordination between the clusters. Apart from the clusters, other coordination structures were mentioned such as, among others, the inter-agency working group on camp transitions

xcvii , the DR Advocacy Network, launched by Tear Fund, Partner National

Societies already present in Haiti (NLRC), contacts with the UN Humanitarian coordinator (ICCO).

27

3.4 Summary of findings

Relevance From the evaluation reports it appears that, especially in the emergency and relief phase SHO interventions convincingly met the immediate needs of the local beneficiaries. In the reconstruction phase most projects were considered to improve the living conditions of the earthquake victims. Whether the response was tailored to local needs and enhanced ownership was dependent: 1) on the degree of involvement and participation of the Haitian people, communities, government; 2) on good knowledge of the context and of the needs of victims, and 3) on the quality of communication between the responsible organisations and the beneficiaries. Where Haitians themselves led the search and rescue and care, this created feelings of belonging, pride and community (ownership). Needs assessments appeared most relevant when local communities were closely associated; relevant societal and cultural factors were then also more naturally taken into account. Community involvement contributed to appropriate solutions. Where the response was not in line with the three aspects mentioned, this reduced commitment, responsibility, initiative and accountability by beneficiaries - with risks for effectiveness and sustainability of the activities. In implementation, willingness to learn, and flexibility on the part of the intervening parties appeared the best guarantee to enhance ownership among the affected people and thus to relevance and connectedness of the interventions. Effectiveness Based on SHO’s 6

th common reporting and evaluation reports it can be concluded that the projects were

effective, again more so in the relief phase, as the most urgent needs were satisfied. Flexibility in funding, in following procedures and protocols and preventing red tape as much as possible, are among factors that have contributed to enhanced effectiveness. The main difficulty faced by many organisations was the urban character of the earthquake, with which only few organisations had experience, and for which most were ill-prepared. The challenges faced were mainly concerned with space, land use and title deeds. Much had to be learned; the learning was most successful in projects which performed well in monitoring and evaluation. A general conclusion, finding or recommendation found in the evaluation reports concerned the importance to initiate a flexible, need-based, multi-sectoral approach through local organisations, and to plan from emergency relief to development in interventions which increases the chances of sustainability and lasting impact. Efficiency The degree of efficiency of the SHO response had to be found out mainly indirectly. The evaluations judge efficiency as positive or neutral. Obviously, working with local resources (personnel and materials) appeared most efficient and cheapest. Flexible approaches and avoiding superfluous bureaucracy in implementation also contributed to efficiency. In the relief phase, overspending occurred and funding of expensive activities took place that was considered justified because of the necessity to undertake quick action, and to cope with unforeseen needs. Efficiency was hampered by external factors (such as rising prices, slow procurement) and systemic internal factors. Several reports recommend to be realistic in planning expenses, and to include risks-assessments in planning. Sustainability and connectedness As observed before, the best chances for sustainability of results is to adopt a holistic approach, i.e., plan a development support in the wake of emergency relief. The SHO organisations that remained operational for longer periods in Haiti (end of 2014 or even longer) managed to apply such an approach, although of a different scope and in different degrees. Ownership of emergency response projects, anchoring of the response activities in government structures, communities and culture as much as possible, capacity development of all actors on all possible levels (technical, organizational, managerial, etc.), and advocacy, were mentioned as critical elements for sustainability. In general, connectedness was strong when organisations disposed of good networks and relations in the Haitian society. Standards From the reports it can be concluded that all implementing organizations adhered or tried to adhere to humanitarian standards, be it Sphere and Code of Conduct or other humanitarian standards (UN/national standards, Christian standards, etc.). When (all) standards could not be respected, this was most often due to

28

the pressure under which the organisations had to work, lack of space (namely, dimensions of houses and/or positioning of latrines) or lack of resources to optimally respect dimensions and quality. In one instance it was argued that standards should be used as indicative and not as absolute, and that indicators need to be contextualized. Looking to the list of relevant elements in this context, it may be concluded that all SHO responses responded to the majority of recommended standards. Cross-cutting themes Although gender analysis was lacking in many of the projects, vulnerability was taken into consideration in the response projects. Gender is explicitly treated in the evaluations about shelter and WASH projects. Children, disabled persons and HIV/Aids patients were (scarcely) mentioned. Rumours about a "rape epidemic" were probably exaggerated but gender violence was a factor to be seriously reckoned with. The degree in which this was indeed the case cannot be clearly understood from the reports. Vulnerability seems to have been taken care of in most responses. During the emergency phase it was not always possible to determine and prioritize who were the most vulnerable victims. Coverage Weak coordination and collaboration characterized the emergency phase of the response. The humanitarian response was predominantly urban and may have overlooked rural areas, on which good information lacked. The geographical coverage seemed eventually to be satisfactory, and no reports about duplication were found. Considering the limited financial resources, some organisations covered too large areas to be able to satisfy urgent needs of their target groups. Strengthening of responses The SHO partners have done serious and most often successful efforts to cooperate with the authorities, and to intervene via local community organisations in order to strengthen responses, from the perspective of sustainability and follow-up. Generally, the contact with local government agencies was easier than those with the (much affected) national government. However, the collaboration with (especially) the Ministry of Health was fruitful. The Ministry of Education was also responsive. There appeared to have been more local capacity available than assumed by many organisations of which, generally, insufficient use has been made (e.g. private enterprises), thus preventing or limiting ownership and sustainability. Cooperation As it was not designed to be, the intentional collaboration between (partners of) SHO member organisations in Haiti was scarcely recorded. Most SHO member's partners worked in UN-clusters charged with sector-coordination. The cluster-system has been difficult to operate and was much criticized, also in the consulted evaluation reports. In the course of time it appeared more efficient to work in geographically located "baby clusters" which also had better contacts with the local government. There were also other inter-agency structures or networks with which SHO partners connected.

3.5 Evaluation and Learning

The information, arranged and summarized above, is almost totally derived from evaluation reports of emergency response projects by SHO members or their partner organisations. The SHO (partners) have themselves commissioned these evaluations and have written or contributed to write the Terms of References and chosen the evaluators. They have also approved and commented on the outcomes of the evaluation reports. What is SHO members opinion about these evaluations, how do they judge their quality, did they agree

with the conclusions? One of the key Dutch SHO-member informant, interviewed for this evaluation, estimated that one third of the evaluations was qualitatively well designed and implemented, one third was reasonable and the quality of one third was moderate to bad

19. The biggest problem was to find good evaluators who were knowledgeable about

19 This judgement more or less correspond with the quality assessment of this meta-evaluation.

29

context of Haiti and spoke Creole, and who wanted to execute the evaluation for a reasonable price. Others found the time that was spent on the evaluations too short and the reports superficial, the parts on efficiency of insufficient quality. Others wondered how the evaluators looked at sustainability. While most informants (more or less) agreed with the conclusions (and one report was highly praised), one person was shocked to see in the conclusions that beneficiaries had had the feeling of not being sufficiently involved in the project. In a few cases there have been discussions between the local evaluators and the commissioning party since the latter did not agree with the evaluator's conclusions. The conclusions of the evaluation reports were said to have served enhancing capacities of partner organisations, but how this had happened was not known. Have the SHO member organisations learned anything from the evaluations they commissioned in Haiti,

or from the experience in Haiti in general? How have they internally (or mutually) evaluated the Haiti response? Have lessons been followed-up in other disasters?

Focus group interviews

20 with SHO member representatives have revealed that indeed the organisations

dispose internally of evaluation and/or learning mechanisms, either formal or informal. There is little evidence of exchange of experiences or synergy between the organisations other than collaboration in the field through clusters. Some organisations have discussed various evaluation reports, but have hardly capitalized knowledge, formalised lessons or applied these. Other organisations such as Plan, World Vision and Tear Fund have formally evaluated and integrated learned lessons in manuals or policies: - Cordaid: the Shelter Group work to improve shelter standards, and a Multi-stakeholder approach (which was further developed in Haiti and applied afterwards), and - World Vision: an Emergency management system (from Australia), and an Early Childhood Development project, and (long-term) trauma counselling approach. Certain tangible internal organisational changes have been introduced (HR issues, staff turnover issues, internal cooperation in alliances, communication). World Vision organised 'learning events' within the sectors/clusters (Education, Protection, WASH, NFI, Telecommunications), facilitated by consultants from Geneva. ICCO evaluated its Haiti-response with all concerned internal compartments and external parties (partners, UN, ACT). The (organisational efficiency) learned lessons were applied in disaster responses afterwards: Horn of Africa, Syria, Philippines, Nepal and Ebola epidemics. The Haiti response's most important learning concerned the complex urban character of the emergency. The experiences have been reflected upon, and they seem to have been taken at heart during later emergency responses (see above). The Meta-evaluation of SHO in Pakistan report

21, in its Concluding Remarks states (p.30): "We hope that the

information presented in this report will be a proper and inspiring input to the learning process of the SHO and that the SHO members will draw on recommendations to improve efforts in the emergency sector". It is not impossible that this has happened but the question how it has happened could not be answered by the available SHO member organisation's stakeholders consulted. The last section in the mentioned Pakistan report about learned lessons concludes (p.28): "Based on the reports considered, our overall conclusion is that lessons learnt are poorly drawn which complicates a proper learning process on emergency responses". In this report, the conclusion is slightly more positive as we have understood from (focus group) interviews with stakeholders from several SHO member organisations that they dispose and make use of learning mechanisms, which are formalised in different degrees, and through which experiences and lessons learned pass from actors involved in previous emergency responses to those involved in the next one's, also in varying degrees.

20

See list of participants, in annex 2. 21 Meta-evaluation and Synthesis of the 2010 Pakistan Floods Response by SHO participants. A synthesis of Conclusions. Verona Groverman, Joke Hartmans, 5 December 2012.

30

3.6 Final remarks

To conclude this chapter we cite from one evaluation report that starts with the following very negative assessment about the humanitarian response in Haiti by donors and the Haitian government in general

22:

" The donors and the Haitian State got caught up in humanitarian urgency projects which were badly designed and coordinated, without ever developing real rehabilitation programmes to allow the affected population to resume a normal life. Today, in 2014, the need for public policies to support the most vulnerable households, in particular in the rural areas and the mega-slums, remains an urgent priority. It is in this context of humiliating hopeless programmes and consequent public policies that the initiative and the development of the rehabilitation project of Croix des Bouquets is situated" (p.7). The same report concludes by saying: "that the evaluated project was a model promoting a rupture with the classical aid approach, a model in which capacities are transferred to men and women so that they can take their future in their own hands. This vision is perfectly integrated in the mission of the executing agency RNDDH

23 and the ICCO development cooperation

24". Nevertheless, the report recommends embarking on a

third response phase. This shows that, apart from short-term emergency relief in which SHO excelled, only a long-term response may hope to achieve "real rehabilitation to allow the affected people to resume a normal life".

22 Many of the substantial criticisms of the global international humanitarian earthquake response point to a lack of a people centred response - contextual understanding, connectedness and integration with Haitian capacity. Global humanitarian response was severely criticised, e.g. for exclusion of Haitian participation at all levels, poor coordination, weak needs assessment and, risk averse humanitarian leadership leading to lack of integration and connectedness with local realities (substantial literature on this, cf Lord Ashdown, DFID Humanitarian Emergency Review, 2011). 23

Réseau National de Défence des Droits Humains - RNDDH. 24 Rapport de l'évaluation externe finale du Projet de relèvement de la commune de Croix des Bouquets, Réalisé par le GRAMIR, Port-au-Prince, Décembre 2014 (translated from French).

31

4 Concluding remarks, lessons and recommendations

In the preceding chapter we have summarized the main findings with regard to the SHO involvement in the Haiti earthquake response based on data presented in local evaluation studies. Thereby, conclusions were drawn about 10 performance criteria

25 subjects. This is in accordance with the ToR of this evaluation that

mentions: "The second section of the evaluation should focus on drawing some general conclusions regarding the total results of SHO’s use of funds. This section of the report will draw on those [i.e. local] evaluations that have been positively evaluated during the quality check. It will analyse the use of the funds according to OECD DAC criteria for disaster response evaluations". The Pakistan Floods Response report was mentioned as a good example of what the Steering Committee expected. As mentioned, the information in the evaluation reports was not always complete, but by assessing as many as 30 of them, we believe to have arrived at a representative picture about the SHO response as a whole, a picture constructed on the basis of local assessments. This is not the complete picture about the SHO Haiti response since other aspects were covered in other reports: the SHO reports and summaries of achievements, the Dutch Court of Account's (Rekenkamer's) reports on accountability issues, various SHO member internal Real Time Evaluations, an inspection report on humanitarian assistance in Haiti by the Ministry of Foreign Affairs' Policy and Operations Department (to be published shortly). This meta-evaluation report must be considered complementary with the mentioned documents. The conclusions will not be surprising for those involved in emergency aid responses, as many similar remarks (conclusions, lessons learned, recommendations) appear in successive reports about this sector. We believe that the SHO has been able to enact a "rupture with the classical aid approach", in a positive sense. In this chapter we will reflect on several findings arrived at in the preceding chapter. We will also reflect on the third issue mentioned in the ToR (p. 2), namely to "… consider whether SHO could have done more during implementation and closure of the projects to present the results achieved in Haiti in a more uniform/harmonised manner or whether this is not desirable given the nature of SHO funding, differences between organizations, and context of the emergency…”.

4.1 Lessons

From the studied material, its conclusions and recommendations, we think that we can extract and synthesize a (limited) number of lessons which have been learned by the SHO partners (NGO sector) in three fields: vision, approach and organisation. Vision:

1. First and foremost SHO members learned that a disaster such as the earthquake in Haiti touches people in all aspects of their lives, (so to say) in an "integrated" way. The vision of NGO emergency aid organisations should not be (sectorally) top-down, but should be bottom-up, based on simultaneously existing multi-sectoral needs. The organisation of coverage of aid should be such that in the affected areas these needs can simultaneously be satisfied. Capacity and resources should be adequate for this purpose. Local people should be the point of departure, not sectors or clusters. The approach matching such a vision is a holistic approach.

2. SHO members see the importance of having a clear vision in emergency response, based on a coordinated and strategic partner-centred approach. The core of the vision is that capacity building among partners should be an on-going process, especially in the fields of planning, monitoring and evaluation, financial accountability, reporting, coordination with other agencies and advocacy. The studies have remarked that these capacities are instrumental for effectiveness and efficiency and sustainability of the actions. Training and coaching are important to embed in the organisations' consciousness about adhering to standards, inclusion of cross-cutting themes (especially gender and vulnerability) and strengthening of responses. It also prepares them for a possible next calamity.

25

Relevance/Appropriateness, Effectiveness, Efficiency, Sustainability/Connectedness, Impact, Coverage, Coherence/Standards, Cross-cutting themes,

Strengthening of responses, Cooperation

32

Approach:

1. SHO members realize the importance of a need-based approach taking into account the survival strategies of the affected people and based on participation in decision-making at community level, including NGOs, government and the private sector. This approach should lead to projects that are appropriate, participative, responsive and support self-reliance. It creates social cohesion, morale, commitment, solidarity, ownership, increases effectiveness and prevents social tension. Challenges of a needs-based approach are the identification of needs (whether to do that centrally or individually; globally or area/community/geographically), the dissemination of this knowledge and its translation into appropriate action.

2. SHO members see the advantage of a multi-sectoral approach (either or not in collaboration with others) as compared to a single-sectoral approach. The approach should take into consideration the involvement of all relevant parties concerned (local and national government, communities etc.) in order to embed the results with direct and indirect beneficiaries.

3. SHO members realise the importance to plan, right from the beginning, an evolution from emergency aid in the relief phase to development interventions, in order to increase sustainability and lasting impact of the response. Such an approach starts with joint needs-identification and planning. At the end of the intervention an appropriate exit- or handing-over procedure should be conceived. This point has been repeatedly mentioned in the reports.

4. Practice has shown the importance to have an effective communication with beneficiaries, with use of modern communication tools (meetings, SMS, radio, car speakers, posters). Good communication and feedback mechanisms steer expectations and thus enhance participation.

Organisation:

1. SHO members recognize the importance of an effective cluster-approach (cf cooperation and coordination) or baby cluster approach. The lessons learned from the Haiti response is that the cluster should be under the (co-) leadership of a local Ministry or local government agency, that Haitian organisations should have easy access to the cluster-meetings, and that the local language should be used. Strong leadership and flexible coordination are essential. Clusters should not play multiple roles.

2. SHO members realize the importance of flexibility in funding, procedures, technical designs, organisation, adaptability to local needs, accompanying measures, cultural preferences or constraints. The importance of "un-earmarked" funds is stressed here to be able to be flexible. A balance is to be found (in advance) with the (donor) accountability demands and procedures in force. The danger that threatens such an approach is too much standardization in procedures and in solutions, and too detailed and strict planning. Monitoring should especially serve learning. Local variety and pragmatism should be acceptable, just like variability and diversity in approach. The implementing organisations should have relative autonomy and room for manoeuvre. Organisation and staff should be energetic and not prone to exaggerated risk-avoidance.

3. Evaluation studies have pointed at the importance of an efficient organisation which manages to keep speed and prevents delays; which works carefully with contractors on planning, materials, quality-criteria and other parameters, which assesses risks, monitors progress constantly, and envisages follow-up. Ideally, staff turnover is low and knowledgeable resource persons remain operational to guarantee continuity.

4. SHO members see the importance of having a risk-mitigation strategy right from the beginning with regard to all interventions and budgets.

4.2 Result-presentation by SHO

The third question in the ToR raised the issue whether SHO should have presented the results achieved in Haiti in a more uniform/harmonised manner. It was suggested to discuss this question with SHO and Ministry of Foreign Affairs stakeholders. We believe that this question is related to remarks by the Dutch Court of Audit (Rekenkamer), from 2012-2013, about the SHO reporting. Although the Rekenkamer in 2013 concluded that

33

consistency and accessibility had considerably improved over the years, comparability between the results achieved by organisations, has not been achieved. The issue can not be objectively studied and analysed, it rather concerns the nature, intentions and operational character of the SHO with which one may agree or disagree. Our opinion, shared with, and agreed by all consulted stakeholders mentioned above, is the following.

1. Since the SHO Haiti earthquake response 'programme' was not formulated in a uniform or harmonized manner, one can not expect a uniform and/or harmonized reporting. We mean by a 'uniform' and 'harmonized' programme a programme which is formulated according to the art of "result-based management" or "management of development results" or "logical-framework analysis". In such formulation the programme has interlinked objectives, results, activities, and assumptions/risks, defined in a coherent way, whereby targets have been quantified in indicators. The whole of such a programme may be called coherent or uniform or harmonized. But the SHO 'programme' was not like that. The SHO members were given the freedom to do what they thought necessary, and implement according to their ideas, competence and capacities and those of their local partners or offices. They performed in the context of clusters, but each organisation did this in its own way. This resulted in 'incomparable' outputs, or what a SHO resource person called 'apples and oranges'.

2. We remind the reader of the remark under Effectiveness and impact (section 3.3) where it is said that flexibility in funding (contingency funds instead of a finite budget structure) leads to a better effectiveness, as it facilitates adaptation to the context and realization of unexpected, complementary needs. This is another reason to support diversity in planning, execution and thus reporting.

3. Diversity in the SHO approach should not be seen as a weakness but as a strength. The diversity in approach and results is a reflection of the identity of the joint Dutch humanitarian organisations. Those who counted on uniform, harmonized results, and likewise reporting, had wrong expectations.

According to the SHO member representatives consulted, there is nothing wrong with the actual reporting format, with which we agree. The information provided in the SHO reports, on the websites and the information in reports and on the websites of individual SHO member organisations is easily accessible, comprehensive and sufficient for interested readers and the media. The general public which has donated money ("my uncle", "my grandmother", "my daughter", according to the resource persons) is not interested and does not expect detailed information packed in a certain jargon which it understands only partly. A picture (info-sheet), nicely printed, with photos and a number of quantified outputs (like the SHO picture presented in chapter 3) is sufficient for them. SHO should not feel the obligation to present more, as this would mean to create a false security. The representatives mentioned as a Good Practice the TV-production made by Jeroen Pauw, which presented a broad picture of the context, and has played a positive role in visibility. We recommend that SHO, in the future, will follow this example and present a short film to be broadcasted on TV, being the principal instrument that was used in the Giro555-action.

34

ANNEXES

35

Annex 1. Evaluation Report Quality Assessment Grid

Assessment criteria Explanation Source: IOB*

Source: ALNAP**

A Validity (Does the research measure what it indents to measure?)

1 Positioning of the subject of the evaluation in the policy/instutional context

The report refers to relevant political context, assumptions and institutional settings of the subject of evaluation. It provide analysis of the affected area and population (including relevant historical, social, economic, political and cultural factors)

1.2.2 3.1

2 Research questions and evaluation criteria (EHA)

The report outlines research questions and defines which standard criteria of humanitarian action evaluation (defined by OECD/DAC) is applied to assess the evaluation subject

1.1.2 4.3

3 The use of and adherence to international standards

The report assesses the intervention against appropriate international standards such as the Red Cross/ NGO Code of Conduct

- 2.5

4 Appropriateness of the overall evaluation methods

The report justifies selection of used research methods and techniques, their appropriateness, relative to the evaluation's primary purpose, focus and users, pointing out the strengths and weaknesses of the methods

1.2.1 and 2.1.1

2.3

5 Evaluation constraints The report outlines key constraints to carrying out the evaluation (e.g., lack of time, difficult travelling conditions, lack of baseline data, poor agency monitoring systems, lack of access to key information sources, difficulties setting up control groups, use of translators), and the effect of these constraints.

2.2.2 2.6

6 The needs and livelihoods assessments that informed the intervention / Intervention objectives

The report provides an analysis of the needs and livelihoods assessment practices and their effect on the intervention as well as a clear analysis of the crisis, including key events (and a chronology where appropriate). The report assesses the relevance of the intervention objectives to the contextual analysis and needs/livelihoods assessments

1.3.1 3.1, 4.2.i and 4.2.ii

7 Cross-cutting issues The report analyses consideration given to a) gender equality, b) protection, c) capacity building, d) advocacy and e) vulnerable and marginalised groups (elderly, disabled, children, HIV/AIDS sufferers, other) throughout the intervention and the effect on the intervention.

- 4.4

36

B Reliability (How rigorous are the evaluation results?)

8 Explanation of research process The report describes and explains the evaluation process, including potential adjustments to the initial set-up and arrangements

2.4.1 2.2

9 Representativeness of a sample and/or case studies

The report clearly explains (with the use of known sampling rules and techniques) how conclusions from the researched sample and/or case studies are valid for the entire researched population / generalised for the entire researched population

2.2.1 -

10 Verification of findings/triangulation

The report outlines how data have been checked and cross-checked with the use of different methods and (secondary) sources

2.1.2 5.1.i

11 Quality assurance through internal or external supervision

Checking of the design and / or implementation of the research by the reference or steering group internal or external to MFS-funded organisation

2.4.2 -

12 Substantiating of conclusions with findings

Degree to which the conclusions were substantiated with findings 1.4.2 5.1.iii

C Usefulness (how applicable are the evaluation results?)

13 Clarity of the research goals Clarity of purpose of the evaluation, for which the research results will be or are used 3.1.1 -

14 Research questions answered by the conclusions

Comprehensiveness to which all evaluation questions were addressed 3.2.1 5.2.i

15 Connections/linkages/coherence Recommendations should follow on from the main conclusions and reflect consultation with key stakeholders. 3.2 5.1.iii

16 Applicability of lessons or recommendations

Practical applicability of the presented recommendations and the degree to which these are within influence of the those in charge, especially the commissioners of the evaluation

3.2.2 -

17 Accessibility of the research results

Clarity and completeness with which the evaluation report and its summary reflect the core of the research, especially the main findings

3.1.2 5.2.iii

* Beoordelingscriteria, indicatoren en componenten, "Methodische kwaliteit van Programma-evaluaties in het Medefinancieringsstelsel-I 2007 – 2010", http://www.rijksoverheid.nl/bestanden/documenten-en-publicaties/rapporten/2012/01/25/iob-rapport-methodische-kwaliteit-van-programma-evaluaties-in-het-medefinancieringsstelsel-i-2007-2010/iob-rapport-methodische-kwaliteit-van-programma-evaluaties-in-het-medefinancieringsstelsel-i-2007-2010.pdf ** The ALNAP quality proforma 2005 (v. 02/03/05) for assessment of the quality of humanitarian evaluations, http://www.alnap.org/material/79.aspx

37

Annex 2. Overview of overall scores for all assessed SHO evaluation reports

Name of the evaluation report (MDF code) Overall score Budget SHO, Euros Total budget SHO % of total Main cluster

Category A

1 09 ICCO 2013 HR Communication Project Evaluation 62 - - - -

2 01 Cordaid 2012 Shelter Programme Evaluation 61 17.319.316 21.142.518 82% Shelter and NFI

3 13 ICCO 2014 de Croix des Bouquets Rehabilitation Project Evaluation 60 75.000 102.000 74% Multiple

4 Plan 2014 SHO Evaluation 60 563.537 598.206 94% Education

5 03 UNICEF 2010 Inter-Agency Real Time Evaluation 60 - - - Multiple

6 03 World Vision HERO Evaluation 59 1.716.252 200.000.000 1% Multiple

7 07 ICCO 2014 ACT Alliance Evaluation 58 3.026.445 23.985.511 13% Multiple

8 2015 DORCAS Evaluation Housing Project 57 905.434 - -

9 03 Rode Kruis 2012 WatSan Evaluation 56 - 21.375.000 0% WASH

10 02 World Vision 2013 ECD-LS Evaluation 56 455.872 1.248.542 37% Multiple

11 14 ICCO 2014 Renafanm Centre Reconstruction Project Evaluation 56 275.000 350.000 79% Multiple

12 01 UNICEF 2013 WASH Schools Evaluation 56 - 3.052.632 - WASH

Category B

13 01 Oxfam 2012 Earthquake Recovery Phase Evaluation 55 15.000.000 83.600.000 18% Multiple

14 01 LdH 2015 TSA School Infrastructure Final Evaluation 54 928.582 1.688.330 55% Education

15 02 LdH 2015 School Water System Programme Evaluation 54 163.734 240.786 68% WASH

16 03 2013 LdH School Repairs Project Evaluation 54 - - - -

17 VNG 2013 Palmes Region Municipal Reconstruction Program Evaluation 53 810.000 4.623.557,50 18% WASH

18 02 Oxfam 2010 Earthquake Real Time Evaluation 53 15.000.000 83.600.000 18% Multiple

19 01 tearFund Earthquake Appeal Evaluation 52 1.214.848 18.594.973 7% Multiple

20 16 ICCO 2012 CROSE / AVSF Post Earthquake Program Evaluation 52 314.002,40 527.475,40 60% Multiple

21 06 ICCO 2014 Young Girls Tabarre Mentoring Project Evaluation 52 54.698 54.698 100% Multiple

22 01 Rode Kruis 2010 Real Time Evaluation of Earthquake Operation 52 - - - Multiple

23 01 TDH 2012 Income Generating Activities Approach Evaluation 52 - - - -

24 05 UNICEF 2011 Operational Response Evaluation 52 - - - Multiple

38

25 02 ICCO 2014 Post-Earthquake Project Evaluation 51 162.610 325.219 50% Multiple

26 15 ICCO 2010 Christian Aid 2010 Real Time Evaluation 49 3.026.445 23.985.511 13% Multiple

27 01 World Vision 2012 CLEP Evaluation 49 670.762 3.500.000 19% Multiple

28 03 ICCO 2014 Houses and Radios Project Evaluation 49 200.000 200.000 100% Multiple

29 08 ICCO 2014 Life Hygiene Improvement Project Evaluation 49 155.529 155.529 100% Multiple

30 02 UNICEF 2012 Semi permanent reconstruction of schools evaluation 49 - 32.812.500 - Education

31 04 UNICEF 2013 Partnership Sanitary Departments Evaluation 44 - 2.406.015 - Health

Category C

32 02 Cordaid 2011 MHPS Final Evaluation 43 400.368 1.365.649 29% Health

33 01 Care 2010 Save the Children Joint Evaluation 43 - - - Multiple

34 02 Care 2013 Merged HEALTH SECTOR Project Evaluation 43 1.684.068 2.482.388 68% Multiple

35 02 Save the Children 2013 Children Education Project Evaluation 43 1.813.732 1.813.732 100% Education

36 2013 H4H Apprendre en construisant pour batir un meilleur Haïti 41 738.788 2.212.646 33% Shelter and NFI

37 12 ICCO 2014 Des Palmes Region Reconstruction MTE 41 1.593.610 4.623.426 34% -

38 03 TDH 2011 Earhtquake Victims Assistance Project Evaluation 36 - - - WASH

39 02 tearFund 2012 Disaster Management Team Programme Evaluation 35 1.127.363 10.356.485 11% Multiple

40 10 ICCO 2011 World Vision Sphere Project Evaluation 31 90.000 159.576 56% -

39

Annex 3. Persons consulted

Meta Evaluation Steering Committee

Caroline Hodges Oxfam Novib

Marijn Both NLRC Rens Rutten Cordaid Participants Focus interviews 8th of September 2015 (The Hague and Amersfoort)

Jip Bekkers Terre des Hommes

Laura Smits Terre des Hommes

Birgit van Delft Save the Children

Marieke Koelfsema UNICEF

Piet Spaarman Cordaid

Eef Krol NLRC

Fred Rietkerk World Vision

Peter Kamphof TearFund

Melle Brinkman Plan Nederland Written Interviews

Dick Loendersloot ICCO Kia Interview Ministry of Foreign Affairs 9th of September 2015, The Hague

Pieter Blussé MFA, Coordinator for Socio-Economic Reconstruction, Department of Stabilisation and Humanitarian Aid

Joyce Imambaks Controlling Unit DSH, Finance Advisor Briefing meeting 19th of September 2015, The Hague

Paul Wolters Terre des Hommes

Florine Bos UNICEF

Melle Brinkma Plan Nederland

Gart van Leersum Care

Martijn Marijnis ICCO

Caroline Hodges Oxfam Novib

Fred Rietkerk World Vision

i UNICEF 2011 Operational Response Evaluation ii Rode Kruis 2010 Real Time Evaluation of Earthquake Operation

iii Source: interview with key SHO informants

iv Plan 2014 SHO Evaluation

v ibid

40

vi

Tear Fund Earthquake Appeal Evaluation; VNG 2013 Palmes Region Municipal Reconstruction Program Evaluation vii

Care 2013 Merged HEALTH SECTOR Project Evaluation viii

Rode Kruis 2013 Shelter Lessons Learned ix ICCO 2010 Christian Aid 2010 Real Time Evaluation; ICCO 2014 ACT Alliance Evaluation

x Rode Kruis 2013 Shelter Lessons Learned

xi ICCO 2014 Renafanm Centre Reconstruction Project Evaluation

xii ICCO 2014 ACT Alliance Evaluation; Plan 2014 SHO Evaluation

xiii DORCAS 2015 Evaluation Housing Project

xiv UNICEF 2012 Semi permanent reconstruction of schools evaluation; Cordaid 2012 Shelter Programme Evaluation

xv Cordaid 2012 Shelter Programme Evaluation

xvi Cordaid 2011 MHPS Final Evaluation; ICCO 2014 ACT Alliance Evaluation

xvii World Vision 2012 CLEP Evaluation

xviii ICCO 2014 ACT Alliance Evaluation; Tear Fund 2012 Disaster Management Team Programme Evaluation

xix LdH 2015 TSA School Infrastructure Final Evaluation; LdH 2015 School Water System Programme Evaluation

xx Tear Fund 2015 Earthquake Appeal Evaluation

xxi ICCO 2014 ACT Alliance Evaluation

xxii ICCO Christian Aid 2010 Real Time Evaluation

xxiii Tear Fund 2012 Disaster Management Team Programme Evaluation

xxiv ICCO 2014 ACT Alliance Evaluation

xxv ICCO 2010 Christian Aid 2010 Real Time Evaluation

xxvi ICCO 2014 ACT Alliance Evaluation; TdH 2014 Haiti- von der Nothilfe zu nachhaltigen Projekten; Care 2010 Save the

Children Joint Evaluation xxvii

ICCO 2014 ACT Alliance Evaluation; World Vision 2012 CLEP Evaluation xxviii

Care 2010 Save the Children Joint Evaluation xxix

Rode Kruis 2012 WatSan Evaluation; xxx

Plan 2014 SHO Evaluation xxxi

ICCO 2010 Christian Aid 2010 Real Time Evaluation xxxii

UNICEF 2011 Operational Response Evaluation xxxiii

Rode Kruis 2012 WatSan Evaluation xxxiv

Plan 2014 SHO Evaluation xxxv

Cordaid 2011 MHPS Final Evaluation; UNICEF 2011 Operational Response Evaluation xxxvi

Rode Kruis 2012 WatSan Evaluation xxxvii

Tear Fund 2015 Earthquake Appeal Evaluation; Plan 2014 SHO Evaluation xxxviii

UNICEF 2011 Operational Response Evaluation xxxix

TdH 2014 Haiti- von der Nothilfe zu nachhaltigen Projekten xl ICCO 2014 ACT Alliance Evaluation; DORCAS 2015 Evaluation Housing Project; TdH 2014 Haiti- von der Nothilfe zu

nachhaltigen Projekten; Tear Fund 2015 Earthquake Appeal Evaluation xli

ICCO 2014 ACT Alliance Evaluation xlii

TDH 2012 Income Generating Activities Approach Evaluation xliii

ibid xliv

Plan 2014 SHO Evaluation xlv

LdH 2015 TSA School Infrastructure Final Evaluation; 02 Cordaid 2011 MHPS Final Evaluation; Tear Fund 2015 Earthquake Appeal Evaluation xlvi

LdH 2013 School Repairs Project Evaluation; Plan 2014 SHO Evaluation xlvii

Cordaid 2011 MHPS Final Evaluation xlviii

Plan 2014 SHO Evaluation xlix

DORCAS 2015 Evaluation Housing Project l Care 2010 Save the Children Joint Evaluation; ICCO 2014 ACT Alliance Evaluation; Rode Kruis 2010 Real Time Evaluation of Earthquake Operation li Cordaid 2012 Shelter Programme Evaluation; ICCO 2013 HR Communication Project Evaluation; Oxfam 2010 Earthquake

Real Time Evaluation; Rode Kruis 2010 Real Time Evaluation of Earthquake Operation; Care 2010 Save the Children Joint Evaluation; Plan 2014 SHO Evaluation; H4H 2013 Apprendre en construisant pour batir un meilleur Haïti lii Tear Fund 2012 Disaster Management Team Programme Evaluation

liii Cordaid 2011 MHPS Final Evaluation

liv Care 2010 Save the Children Joint Evaluation

lv ICCO 2014 ACT Alliance Evaluation

lvi World Vision 2012 HERO Evaluation; World Vision 2012 CLEP Evaluation

lvii Tear Fund 2012 Disaster Management Team Programme Evaluation

lviii LdH 2013 School Repairs Project Evaluation

lix Cordaid 2012 Shelter Programme Evaluation

lx Tear Fund 2012 Disaster Management Team Programme Evaluation

41

lxi

Cordaid 2012 Shelter Programme Evaluation lxii

Care 2010 Save the Children Joint Evaluation lxiii

ICCO 2011 World Vision Sphere Project Evaluation lxiv

World Vision 2012 CLEP Evaluation lxv

Plan 2014 SHO Evaluation lxvi

Care Haiti: Gender Survey, Life Saving Interventions for Women and Girl in Haiti; conducted in Communes of Logane and Carrefour, Haiti; August 2013 lxvii

ICCO 2014 ACT Alliance Evaluation lxviii

Rode Kruis 2012 WatSan Evaluation lxix

Care 2013 Merged HEALTH SECTOR Gender Survey lxx

LdH 2013 School Repairs Project Evaluation lxxi

Care 2013 Merged HEALTH SECTOR Gender Survey lxxii

ICCO 2014 Young Girls Tabarre Mentoring Project Evaluation lxxiii

Tear Fund 2012 Disaster Management Team Programme Evaluation lxxiv

Care 2010 Save the Children Joint Evaluation; World Vision 2013 ECD-LS Evaluation; World Vision 2012 HERO Evaluation lxxv

ICCO 2011 World Vision Sphere Project Evaluation lxxvi

ICCO 2010 Christian Aid 2010 Real Time Evaluation lxxvii

Tear Fund 2012 Disaster Management Team Programme Evaluation lxxviii

DORCAS 2015 Evaluation Housing Project; ICCO 2010 Christian Aid 2010 Real Time Evaluation lxxix

Oxfam 2010 Earthquake Real Time Evaluation lxxx

Tear Fund 2012 Disaster Management Team Programme Evaluation lxxxi

Care 2010 Save the Children Joint Evaluation; UNICEF 2010 Inter-Agency Real Time Evaluation lxxxii

ICCO 2014 ACT Alliance Evaluation lxxxiii

DORCAS 2015 Evaluation Housing Project lxxxiv

Tear Fund 2012 Disaster Management Team Programme Evaluation lxxxv

ICCO 2010 Christian Aid 2010 Real Time Evaluation lxxxvi

Cordaid 2011 MHPS Final Evaluation lxxxvii

Plan 2014 SHO Evaluation lxxxviii

Care 2010 Save the Children Joint Evaluation lxxxix

UNICEF 2010 Inter-Agency Real Time Evaluation; UNICEF 2011 Operational Response Evaluation xc

Care 2010 Save the Children Joint Evaluation xci

ICCO 2014 ACT Alliance Evaluation xcii

Cordaid 2011 MHPS Final Evaluation xciii

Care 2010 Save the Children Joint Evaluation xciv

Rode Kruis 2013 Shelter Lessons Learned; Rode Kruis 2010 Real Time Evaluation of Earthquake Operation xcv

Interview with key informant from SHO member organisation xcvi

UNICEF 2011 Operational Response Evaluation xcvii

World Vision 2012 HERO Evaluation