Embed Size (px)
Citation preview
Terms of Reference
Health and Social impact - arcenciel in Lebanon
Intervention summary
As part of its evaluation system, the French Development Agency evaluates, after completion, the projects and programs it finances.
The project “Phase 2: Support for the establishment of a local social protection system in vulnerable municipalities” is implemented by arcenciel. This action is funded by the French Development Agency (AFD).
The project aims at strengthening the resilience of health and protection services and facilities, accessed by the most vulnerable populations, affected by the Syrian crisis, including refugees and host communities.
The project’s activities include:
Strengthening Municipalities capacities; Medical & social services; Provision of technical aids, medications, hygiene, supply and food kits; Short professional trainings (life support, sewing, cooking); Psycho-social activities (circus, swimming and sailing activities, carpentry trainings
etc.); Awareness Raising (sessions, events, etc.) with community members on health and
social; Networking with health and protection public and non-governmental agencies; Capacity building on employment, coaching; Protection support (home visits, referrals etc.)
The project is implemented through community centres and mobile units in the regions of Beirut, Mount Lebanon, Akkar and Bekaa.
Therefore, arcenciel invites you to submit an offer for the following service: impact evaluation of the “Phase 2: Support for the establishment of a local social protection system in vulnerable municipalities” project, started in 2017 for a period of almost 3 years, until its closure in 2020.
1. Purpose and overview of objectives
This external evaluation is an assessment of outcomes achieved through the implementation of the project activities that started in 2017 and ended in June 2020. This evaluation is the second one on this project. The first one, done in 2019, was evaluating relevance, efficiency, effectiveness, impact and viability of the funded project with regard to the context, policy and intervention procedures of the AFD group and will be provided to the consultant/team of consultants.
The purpose of this impact evaluation is to collect evidences and data in order to allow arcenciel and other partners working in related sectors to learn from the impact of program and present results, conclusions, lessons learnt and further recommendations with regard to the program and the implementation of its approach. It aims to assess the level of achievements in the activities of the project and to verify to which extent the project’s activities contributed to improving the health and social wellbeing of the targeted population in the project’s catchment areas and how it impacted the Municipalities capacities. It aims at measuring the pace and quality of implementation, variances, and the reached outcomes. The evaluation will be used as part of the organisational learning process aiming to replicate best practices and increase programme effectiveness.
2. Audience
The audience who will be directly benefiting from this evaluation are arcenciel and the French Development Agency (AFD). A summary of the evaluation will be shared with external stakeholders and partners.
3. Methods
An external consultant or team of consultants will be responsible for conducting the impact evaluation. The consultant/team is expected to conduct a desk review of relevant project documents and monitoring reports/data, meet with senior management, program staff, centre staff, and relevant stakeholder, and participate in consultation and debriefing meetings. The firm/team is then expected to develop a detailed methodology, appropriate data collection tools, and a detailed data analysis plan. Any quantitative survey should include a sample size calculation that will render the results statistically valid. Discussions on the sample size and proposed tools are to be shared with the M&E team before the start of the actual fieldwork.
The consultant will be expected to include a mix of quantitative and qualitative data collection methods such as surveys, focus group discussions, key informant interviews, and field observation visits. Triangulation of data through different tools and sources is considered fundamental.
Overall, the evaluation will be done according to a list of criteria which includes relevance, effectiveness, efficiency and sustainability, and through desk research, meetings, and interviews with relevant groups and individuals.
4. Implementation information
a. Contact
The consultant/team will work closely with arcenciel’s project coordinator, M&E officer and Program Managers. Centre staff will be also working closely with the consultant/team.
b. Location
Health centres operated by arcenciel in the regions of Bekaa, Akkar, Beirut and Mount Lebanon.
c. Deliverables (in French or English)
- Methodology tools
- Qualitative and quantitative data collection tools (interview guide, questionnaire, etc.), templates should be provided in English or French and Arabic.
- Framework Note
On the basis of the methodological research, the consultant shall provide a document (which should not exceed 30 pages, including the narrative of the project which must not exceed 10 pages) summarizing the framework of the evaluation after the beginning of the research. This note i) will present the intervention logic of the project; ii) will specify the main questions that will be used to focus the evaluation work on a limited number of key points in order to allow for more targeted information gathering during the field phase, finer analysis and a more useful report; iii) will establish the steps of reasoning that will allow the consultant to answer the questions (judgement criteria); iv) specify the indicators to be used to answer the questions and the corresponding sources of information; v) accurately confirm the steps of the method (including the number of man-days allocated to each of these steps defined in its proposal), as well as the methodological principles that will guide the evaluation and how these principles will be implemented in practice (e.g. if the evaluation is intended to be participatory, then the note should explain how the participatory dimension will be implemented).
This note will include a project narrative in order to reconstruct its progress at its various stages. It will include a general presentation of the context and its evolution during the implementation period, a description of the project (objectives, content, amount, financial product, stakeholders, operating mode) and a summary of the project's progress since its inception. Within this framework, the consultant will make a distinction between the project appraisal phase (from identification to the granting of AFD financing), its implementation
phase (from the signing of the financing agreement to the completion of the project), and finally the significant events that occurred between the project completion date and the date of the evaluation. This synthesis will focus on the facts and factors that feed into the evaluative analysis.
This framework document must be validated by arcenciel and by AFD before the work can continue.
- Transcript of interviews and focus group discussions (FGD)
- A progress report as an intermediate deliverable, for example after field visits. This deliverable can be a simple Power point presentation (PPT). It ensures that the consultant does not start on the wrong path.
- Draft report
A draft of the final report, which should not exceed 40 pages outside the appendices, will be produced at the end of the consultant's work. arcenciel will provide its comments and observations to the consultant within three weeks of receiving this report.
- PPT reflecting initial findings for arcenciel
- A clear and concise, well-written final impact evaluation report containing lists and tables, facts and figures demonstrating the state of the end-line situation and other findings at the end of the project, as well as a detailed comparative analysis of other health models implemented in similar contexts. The final report, incorporating observations from the first draft, should be available within 15 days after receiving comments. If the observations express differences of assessment not shared by the consultants, they may be annexed to the final report and commented on by the consultants.
- Meeting with the donors to present the findings of the report
- Summary of the final report for dissemination to external related stakeholders. This summary will include a presentation of the project (context, stakeholders, modalities, objectives, content) and will include the main conclusions, lessons and recommendations of the report. This summary note aims to promote the dissemination of evaluation lessons. It should assume that readers of this note will not necessarily read the full report. It must be both very operational and understandable by everyone.
The interim and final reports should be submitted in electronic format to the following e-mail addresses: [email protected] _ [email protected]
Required Format for the impact evaluation report (see more details in Annex 1):
The final report will be submitted to arcenciel and should be around 40 pages (excluding the annexes). A good report must be synthetic. All additional useful material may be annexed. arcenciel reserves the right to refuse a report if it proves indigestible. The final report should be produced in English or in French and must include but not be limited to the following:
Cover page
Table of contents List of abbreviation Executive Summary that should include major findings of the impact evaluation,
including indicator tables, conclusions and recommendations; Context analysis The objectives of the evaluation A description and justification of the methodology used, timing of evaluation, and
challenges/limitations of the analysis A scientific presentation of results Challenges, recommendations, and conclusions Annexes
d. Required qualifications and expertise
The consultancy is open for individual consultants or teams and companies.
The candidate:
· Preferably holds a postgraduate degree in social sciences, development studies, medical studies or a related field and/or has at least three years of experience with familiarity in Health and Social field;
· Preferably has at least 5 years’ experience in conducting similar evaluation in Middle East;
· Excellent skills in research and data collection;· Ability to work both independently and as part of a team;· Speaks good Arabic. English and French is also required;· Ability to conduct interviews in Arabic;· Preferably has knowledgeable about refugees’ context;· Is willing and available to travel in all Lebanon.
e. Timeline
Consultant bidding: Beginning of May 2020
Consultant selection: End of May 2020
Contract signature: End of May, beginning of June 2020
Draft report: August 2020
Final report: September 2020
Summary/presentation of the final report: September 2020
Presentation open to partners and co-construction of recommendations: September 2020
f. Length
The research is expected to take up to 12 weeks including: document review; methodology reviewed and validated internally; field visits and centres visits; finalisation of first draft; internal review and validation process by the partner (AFD); final draft approved.
g. Logistic support
The consultant/team will be responsible for the logistical planning for the meetings, interviews and field visits with the support of the project coordinator and M&E officer.
5. The consultant/team of consultants
The consultant/team will work closely with arcenciel staff in Beirut. The consultant’s support shall ensure an effective data collection and data analysis using a dynamic and a participative approach with stakeholders and beneficiaries for a clearer reflection on the different levels of the project’s results. He/She will be required to travel to Mount Lebanon, Akkar and the Bekaa regions for observation and data collection purposes.
6. Roles and responsibilities of the consultant/team
The consultant will be expected to implement the various stages of preparation, planning, implementation and analysis of the impact evaluation. The evaluation will be expected to include, but not be limited to, the following groups:
Targeted beneficiaries (Refugees and Host communities) (women and men); to be included in quantitative and qualitative assessments
Partners and stakeholders involved in the project at different levels:o Targeted Municipalities and staff members: Service providers (case
managers, social workers, medical and non-medical staff)o Community actors: community health workers, informal settlement focal
pointso Government agencies: Ministry of Public Health (MPH), Ministry of Social
Affairs (MoSA), municipalities
1. Develop an overall impact evaluation methodology: He/she is excepted to share and discuss the methodology with arcenciel.
2. Develop qualitative and quantitative data collection tools: The consultant/team will develop, test and finalize the tools. He/she is expected to discuss the draft tools with arcenciel and finalize the tools after the testing.
3. Develop a framing note: He/she is excepted to share and discuss the framing note with arcenciel.
4. Lead on data collection: conduct desk review but also, through questionnaires, conduct key informant interviews and FGD and fill out surveys with relevant stakeholders and target groups as per the evaluation methodology
5. Lead analysis of data: The consultant/team will share the raw data and basic analysis with arcenciel, then he/she will analyse this data and present a summary of
the initial findings. The consultant/team will carry out the literature review and submit a written summary in the report. He/she will be responsible for the safe and secure storage and treatment of this data.
6. Develop a draft report: He/she is excepted to share and discuss the draft of a report with arcenciel.
7. Present and discuss initial findings: The consultant/team will present and discuss the findings with arcenciel in consultative meetings in order to refine analysis and ideas for recommendations.
8. Develop a final report: Bringing together the findings from the literature, quantitative data and qualitative analysis to finalise the report. The report should highlight a set of operational recommendations and lessons learned. Also, the report might address all arcenciel comments.
9. Present and discuss findings: The consultant/team will present and discuss the findings with arcenciel and AFD in a meeting.
10. Develop a summary and present the key findings of the report: In English for partners, external stakeholders and donor. The summary is a maximum of 2 pages long. It will be published on AFD's website.
Application Submissions Guideline:
arcenciel invites all interested and qualified candidates to submit the 5 following documents
(in separate files):
· CV (resume and list of previous consultancies);
· Brief narrative proposal (tasks, timeline, methodology, deliverables, outcome of
those deliverables etc.) including an understanding of the present terms of
reference; that deliverables are dynamic where arcenciel might request certain tasks
and deliverables not anticipated by the consultant.
· Brief financial proposal (tentative budget); Full amount will be paid by arcenciel by
check in USD.
· Letter of interest, clearly explaining how their experience meets desired
qualifications
· Signed statement of integrity
The 5 files should be sent, over email, by May 25th, 2020, to: [email protected]
Any incomplete application will not be considered eligible.
Annex 1 – Model format for the evaluation report
The evaluation report must not exceed 40 pages, excluding annexes, (font size 12). Detailed information on the context, the project or the general aspects relating to the methodology and analytical approach must be included as annexes.
The report’s cover page will include the following mention:
“The purpose of this ex-post evaluation is to formulate a reasoned opinion on the relevance, effectiveness, efficiency, impact and sustainability of the funded project with respect to the context, policy and procedures of AFD Group’s intervention. The evaluator has examined the outcomes of the project in the light of the objectives fixed. He/she has also reviewed the execution and functioning of the project in its different phases of implementation and monitoring.
This evaluation has been financed by AFD. The observations, assessments and recommendations expressed in this report are the sole responsibility of the authors.”
The report can be based on the following model format:
1.Table of contents and list of acronyms
2.Executive summary (see template in Annex 2)
The summary is a maximum of 2 pages long. It will be published on AFD's website.
On the first page: it recalls the key data (financial product and amount, date of signature of the agreement, completion date, project duration) and describes the project: context, stakeholders and operating mode, objectives, content and expected achievements. The consultant will also indicate in 3 to 4 lines the main points of successes or failures to be retained (These lines will be used for an internal arcenciel presentation of the evaluation).
On the second page: the consultant will present the main conclusions on the project's performance, based on the evaluation questions or, to the extent possible, on the DAC evaluation criteria based on the reclassification of the evaluation questions The consultant will present the overall conclusions and lessons learned from the evaluation on the right hand side of the second page.
3.Evaluation methodology (3 to 4 pages)This section succinctly (if necessary, details may be annexed) describes the methodology used, the general implementation modalities of the evaluation assignment and any difficulties encountered.
The consultant shall ensure that the methods of investigation and analysis used and the procedures followed for collecting information and ensuring its quality and validity are clearly presented.
4.Detailed project narrative (5 to 10 pages)
If the narrative is too long (i.e. more than 5-10 pages), then it will be appended, and only a summary of the narrative will be included in the body of the report. This synthesis will emphasize the facts and factors that inform the evaluative analysis.
This section must include:
a general presentation of the context and its developments occurring during project implementation;
a description of the project (objectives, contents, amount, financial product, intervening parties, procedures);
a summary of the project’s progress since its inception (if necessary, include detailed figures and collected data in the annexes), highlighting the main events that affected the project, presenting allocation and volume of mobilised funds, pointing out the main difficulties encountered and mentioning, where appropriate, any realignments carried out.
In this framework, the consultant will ensure a distinction is made between the project appraisal phase (from identification to the allocation of AFD’s financing), its implementation phase (from the signing of the financing agreement to project completion) and, finally, the main events that occurred between the project completion date and the evaluation date.
5.Project performance (12 to 17 pages)
It is recommended that the evaluative analysis be organized by evaluative questions.
The consultant must clearly explain the progressive progression of the steps between observations (raw data), findings (developed indicators, ratios) and judgements made. [The consultant should ensure that the judgements on gender ( if the project's gender DAC rating is 1 or 2, see Annex 4) is taken into account.]
6.Conclusions (4 pages)
The evaluation should lead to a reasoned judgement and conclusions on the performance of the AFD-funded project. This section must summarise the consultant’s overall assessment of the project’s performance in the light of the evaluative analysis. The strengths and weaknesses of the project must be clearly described. This summary does not follow the order of the questions or that of the evaluation criteria.
The consultant will formulate a limited number of conclusions in order to guarantee their high standard of quality. He will clarify or omit any value judgement that is not sufficiently
supported by facts. He will also make sure that the evaluation criteria are used in a balanced manner. The conclusions and any lessons learned are listed, categorised and prioritised in a few pages. The methodological limits will be mentioned, as will any differing opinions.
7. Lessons and/or recommendations: 2 pagesThe consultant will present in this section the lessons learnt and his/her recommendations. He/she will seek to maintain a clear distinction between the conclusions that will not lead to any specific action and the lessons/recommendations. The latter must, however, stem from the conclusions. The lessons/recommendations must be grouped together and classified by order of priority.
Annexes to the reportThe annexes can include:
Terms of Reference list of people met assignment schedule field interviews report list of documents consulted list of activities specifically examined and the situation map of interventions eventual details on the chosen evaluation method logical framework of the intervention reconstituted ex-post tables on the funds mobilised, the results or the impacts
Annex 2 – Summary template
Annex 3 – Information sources
- Document review: indicate the list of documents that have actually consulted and specify to which they are referred when they provide important elements in the argument.
- Semi-directive interviews: specify how the interlocutors were selected, how the interviews were conducted and whether any could not be met. Refer as much as possible to the original comments by quoting the interlocutor who mentioned them. It is possible not to mention the names of the interlocutors if they raise sensitive issues, but it is then necessary to provide indications on their status and function to allow the reader to reposition the subject.
- Focus groups: encouraged from a learning perspective, but to be used with discernment, in addition to other methods. Ensure that the manner in which the work was carried out (who was invited and how, how the exchanges were conducted) is well documented
- Existing data: many sources can be mobilized: household surveys, satellite data, etc.
- Ad hoc surveys: Use only if other data sources are not available. Specify the sampling method, pay attention to the design of the questionnaire, ensure quality control during agent training, supervision during collection and at the time of entry.
- Field observations: The way in which the sites visited are selected, the duration and conditions of observation must be briefly described. It is recommended to take the GPS points of the locations and represent them on a map at the end of the mission. Taking pictures to illustrate the purpose is also recommended.
All the methods below must preserve the principle of informed consent (the interlocutor must be informed of the reason for which he or she is being questioned and explicitly express his or her agreement). Anonymity must also be preserved when dealing with sensitive subjects or when the environment raises security risks for partners and beneficiaries.
Annex 4 – DAC criteria
The DAC criteria definitions can be used by the coordinator to define the evaluation questions. These criteria are illustrated in a figure at the end of the annex.
Relevance
The consultant shall first examine the consistency of the intervention with respect to the objectives and issues determined at the outset (relevance of the project at its origin). When appropriate, it will also assess the founding hypotheses that led to the construction of the intervention logic. He shall also assess how this consistency has evolved over time (evolution of the project over time).
In this framework, the consultant will systematically examine to what extent the project is consistent with:- national guidelines (define which as appropriate);- the definition, needs and expectations of the final beneficiaries (beneficiaries to be defined depending on the project). On this specific point, and particularly in order to judge the relevance at the origin of the project, the consultant may, for example, examine the efforts that were made at the time of the appraisal in order to define the final beneficiaries and take into account their points of view);- the strategies and interventions of the other stakeholders (to be defined as appropriate: NGOs, donors, local authorities…).- AFD’s strategies (specify which ones according to the case: geographical, sectoral, cross-cutting.
Coherence
The consultant will assess both the internal and external consistency of the project: - the internal coherence of the project. This will involve reconstructing and judging the consistency of the various resources and tools mobilized to help achieve the objectives on the basis of the logic of the intervention;- the capacity to integrate the situation and the respective interests of men and women at the time of the project design via preliminary analyses;- the external coherence of the project (consistency with the interventions of the other actors : to be defined as appropriate—bilateral and/or multinational donors, public administrations, NGOs…).
Effectiveness
The consultant shall first assess the achievement rate of the project (comparison between the expected outcomes and actual outcomes). The consultant shall then assess the extent to which the project’s objectives were fulfilled (particularly the specific objectives).
The consultant shall analyze the differences observed for both of these assessments, and shall seek to analyze the project’s contribution to the results achieved.
If necessary, the consultant may give an assessment of the project's ability to adapt to changing circumstances; this may lead a project to change the initial results and objectives (articulation with the relevance criterion).
If the sustainable development opinion made recommendations [Within this framework, the consultant will analyze the implementation of the recommendations contained in the sustainable development opinion].
For projects that include gender as the main or secondary objective (notes 1 and 2 of the OECD-DAC marker)[the results shall by systematically presented for men and women.].
If aggregately indicators have been defined ex ante. [The consultant will pay particular attention to the analysis of the aggregately indicators]
Efficiency
Efficiency examines the relationship between the resources implemented and their costs, on the one hand, and the funded outcomes achieved, on the other hand. This will involve assessing whether the resources of the project (funds, expertise, time, etc.) have been converted into results in an optimal manner.
The evaluation carried out by the consultant must make it possible to:- assess whether the required resources were actually implemented, in a timely fashion and provide an optimal cost/efficiency ratio;- analyze any observed delays or cost overruns;-In cases where this is feasible [measure the unit costs of the projects financed, as well as the per capita investment amounts for the project (i.e. the total amount of investments, or of a category of investment, compared to the number of beneficiaries concerned) and, if possible, to situate these costs and these amounts per capita compared to those observed in other projects.]
Depending on the case, it may be useful to specify in the ToR the levels of analysis that could be specifically examined, particularly concerning the procurement procedures, the financial management procedures, the institutional arrangement, the conditions precedent, etc. Certain infrastructure projects can warrant a cost-benefit analysis.
Impact
Impact measures the benefits of the intervention on the relevant stakeholders and, particularly, the final beneficiaries. Here, the consultant will analyze the long-term effects (or the perspective of effects), both positive and negative, primary and secondary, that can be reasonably attributed either partly or entirely to the intervention under evaluation, whether directly or not (direct and indirect effects), or intentionally or not (expected and unexpected effects).
This criterion is only relevant in cases where we have some hindsight after the completion of the activities. It is therefore not always relevant to include it in the evaluation process. The project’s contribution to the sustainable development objectives (economic development, poverty reduction, reduction of inequalities, including gender inequalities, biodiversity preservation and fight against climate change) as set out in the sustainable development opinion, can serve as a framework to evaluate the impact of the project.
For projects that include gender as a main or secondary objective (notes 1 and 2 of the OECD DAC marker) or if gender-related questions arose during project implementation [The consultant shall systematically evaluate the project’s impacts on gender equality and relationships. This will specifically involve evaluating the impacts of the distribution of responsibilities, access to and the control of resources, the workload of women…].
Sustainability
Sustainability is defined as being the continuation of benefits resulting from a development action after the intervention has reached completion. It is consequently assimilated to the likelihood of obtaining sustainable, long-term benefits.
The consultant will therefore examine here whether the fulfilment of the objectives that were set and the results already obtained, or in the process of being obtained, are of a sustainable nature, or even likely to be amplified, over the long term, and, if so, under what conditions. In this framework, he will ensure he assesses the sustainability of the structures/institutions initiated or supported under the project, as well as the sustainability of the effects created by the AFD-financed project.
The main aspects to be analyzed may be defined by the coordinator of the evaluation according to the project evaluated. It could particularly be useful to differentiate between the different levels of sustainability (institutional, technical, financial, environmental) that the analysis should cover.
Added value of AFD’s action
In this section, the consultant will assess AFD’s specific contribution and added value with respect to other interventions or ways of proceeding available to the contracting authority. It will be necessary here to make a critical judgment that shall highlight both the strengths and weaknesses of AFD’s action. This analysis may focus on the different stages in the project cycle, the characteristics of the financial resources provided by AFD, the cross-cutting issues (partnership and dialogue, capacity building, etc.) or on any other issue identified by the consultant.
This discussion on the “added value” brought by AFD shall be clarified by the evaluation coordinator, particularly with respect to the specificities of the project, the sector concerned, the financial product and the nature of the support provided by AFD. Depending on the case in point, this may involve:
- the nature of AFD’s influence on the strategies of local actors;- the appraisal process- the involvement of actors (and final beneficiaries)- the partnership between the local actors and AFD;- the operating method and the implementation methods;- the supervision mechanism;- AFD’s eventual contribution to capacity-building;- possible input from AFD in the area of environmental and social risk control;- AFD’s potential contribution to gender equality;- the characteristics of the financial resources provided by AFD with respect to other available comparable resources (grants, loans);- the leverage or catalyzing effect of AFD funding on the behavior of other actors;- …
This list of criteria is not restrictive: partnership, participation, gender equity, capacity-building, conservation of natural resources, etc. may also be specific complementary themes for study depending on the nature of the interventions being evaluated.
