View
219
Download
0
Category
Preview:
Citation preview
External Review of the Office of Independent Evaluation
Caribbean Development Bank
April, 2016
Principal ReviewerMarlène Läubli Loud,
Review Panel MembersJohn Mayne
Bastiaan de Laat
1
Primary audiencesThe main target audiences of this Review are
Members of the Oversight Assurance Committee Members of the Board of Directors Independent Evaluation Office Caribbean Development Bank Management and Staff
It is assumed that audiences are familiar with the OIE and evaluation function. Background information and description of evaluation processes and procedures are therefore minimal.
Review Panel Members
Marlène Läubli Loud (DPhil) is currently an independent consultant and trainer in public sector evaluation. She has worked with a range of organizations, small and big including the European Commission, the World Health Organisation, the United Nations Evaluation Group, the UK Employment Department, UK Health Promotion Agency (now merged and become NICE), and the English Nursing Board. She was head of the Research and Evaluation Unit at the Swiss Federal Office of Public Health for nearly twenty years where she gained much experience in evaluation management, and especially in the ways and means for improving the use and utility of evaluation in organisations. She continues to have a keen theoretical and practical interest in this area. Prior to this, she was an independent evaluator in the UK, specializing in the evaluation of developmental programmes in health and general education. She was also a research fellow at the Department of Education, University of Surrey and in the Social Science Faculty, University of Oxford, UK.
John Mayne (PhD) is an independent advisor on public sector performance. He has been working with a number of organizations and jurisdictions, including several agencies of the UN, the Challenge Program on Water and Food, the European Union, the Scottish Government, the United Nations Secretariat, the International Development Research Centre, the Asian Development Bank and several Canadian federal departments on results management, evaluation and accountability issues. Until 2004, he was at the Office of the Auditor General where he led efforts at developing practices for effective managing for results and performance reporting in the government of Canada, as well as leading the Office’s audit efforts in accountability and governance. Prior to 1995, John was with the Canadian Treasury Board Secretariat and Office of the Comptroller General. He has authored numerous articles and reports, and edited five books in the areas of program evaluation, public administration and performance monitoring. In 1989 and in 1995, he was awarded the Canadian Evaluation Society Award for Contribution to Evaluation in Canada. In
2
2006, he became a Canadian Evaluation Society Fellow.
Bastiaan de Laat (PhD) is Evaluation Expert and Team Leader at the European Investment Bank (EIB) where over the past two of years he has been in charge of major evaluations in important areas such as Climate Action, SME support and Technical Assistance. He has a longstanding experience in evaluation as well as in foresight. Founder-director of the French subsidiary of the Technopolis Group (1998-2006) he led many evaluations for and provided policy advice to a great variety of local, national and international public bodies. He trained several hundreds of European Commission staff and national government officials in evaluation and designed monitoring and evaluation systems for various public organisations. Before joining the EIB he worked as Evaluator at the Council of Europe Development Bank. He has developed tools and performed programme, policy and regulatory evaluations, both ex ante and ex post, in a variety of fields. He has also made several academic contributions, most recently with articles on evaluation use and on the "Tricky Triangle", on the relationships between evaluator, evaluation commissioner and evaluand. In his private capacity, Bastiaan served as Secretary General of the European Evaluation Society and was recently elected Vice-President.
Acknowledgements The Review exercise could not have been possible without the support and commitment of the OAC, the OIE and the CDB. The exploratory discussions with the OIE staff, members of the Board of Directors as well as with the CDB management and staff provided great insight and were a valuable contribution to the Review.
We are indebted to the Head of OIE, Michael Schroll, and his team for their cooperation, insight, and readiness to provide us with any information requests. We are also grateful to them for their useful comments regarding the first draft of our report, and for their suggested improvements.
We are especially appreciative of OIE’s administrative assist, Denise Padmore, for her help in coordinating the interviews during our 10-day field study in Barbados and in providing us with all the documents we requested.
3
AcronymsLIST OF ABBREVIATIONS
AMT Advisory Management TeamAPECAPAR
Audit and Post Evaluation CommitteeApproach PaperAppraisal Report
BOD Board of DirectorsBMCs Borrowing Member CountriesBNTF Basic Needs Trust FundCDB Caribbean Development BankCSPDAC
Country Strategy PaperDevelopment Assistance Committee
DFI Development Financial InstitutionED Economics DepartmentEOV Evaluation and Oversight DivisionFI Financial InstitutionIRLMDB
Immediate Response LoanMultilateral Development Bank
mn millionM&E Monitoring and EvaluationMfDR Managing for Development ResultsOAC Oversight and Assurance CommitteeOIE Office of Independent EvaluationPAS Performance Assessment SystemPBG Policy-Based GrantPBL Policy-Based Loan PCR Project Completion Report PCVR Project Completion Validation ReportPPES Project Performance Evaluation SystemPPMS Portfolio Performance Management SystemSDF Special Development FundTA Technical Assistance WB World Bank
4
Preface Evaluation work at the Caribbean Development Bank (CDB) has been ongoing since the early 1990s, although initially it was mainly focused on the ex-post evaluation of projects. However, in 2011, the CDB reviewed its evaluation system to bring it up to date with the good practices of international development organisations. In December that year, it produced its comprehensive Evaluation Policy (December 2011) setting out the aim and objectives and guiding principles for CDB’s evaluation system.
The Policy provides for the establishment of the Office of Independent Evaluation (OIE). Its main objective is to provide “CDB’s Board of Directors, President, Advisory Management Team, CDB staff and other stakeholders and partners with timely, credible and evidence-based information on the relevance and performance of CDB’s projects, programs, policies and other development activities.” (Evaluation Policy, 2011, p. 1).
To oversee and assess good practice, the Evaluation Cooperation Group (ECG) for Multilateral Development Banks (MDBs) recommends that the MDBs’ evaluation system and independent evaluation units be the subject of a review on a regular basis. The aim here is to help the institutions adopt recognised evaluation standards and practices so that its policies may benefit from evidence-based assessments.
In mid-2014, a new Head of the OIE was appointed and, following an initial learning period, he called for a peer review of the evaluation system. Even though the OIE had only been in existence since 2012, it was considered timely to take stock of what had been done so far in order to tease out the priorities for the next 3-4 years.
It was originally anticipated that such an assessment could be done by the ECG as part of the OIE’s application for ECG membership. This did not prove possible, since the CDB’s operation is considered too small for such membership. A review was therefore commissioned to independent experts in evaluation who are knowledgeable and experienced in the management of internal evaluation units.
Main Aim of the ReviewThe Review’s main aim is to provide the CDB’s Board of Directors with an independent assessment of the OIE and CDB’s evaluation system. The intention is to highlight the factors that help or hinder the OIE’s independence and performance in order to identify where improvements could be made. This report will be presented together with a Management Response to the CDB’s Oversight Assurance Committee and its Board of Directors at its meeting in May 2016. It is anticipated that an action plan will be drawn up on the basis of the Board’s decision on how to address the recommendations put forward.
Report StructureThe Review starts with some general background information about the CDB and the setting up of an independent evaluation function. It also sets out the reasons for an external review and why this was requested at this particular point in time. Part One also outlines the Review methodology, which, is presented in more detail in Appendix II. Part Two reports on the Review’s findings and conclusions with regard to a certain number of criteria. The Panels conclusions and recommendations for future work are the subject of Part Three.
The Panel is grateful for the complete freedom it was given to form its own opinions and to reach conclusions based on its analysis. The findings, conclusions and recommendations
presented in this paper are those of the Peer Review Panel members. The views of the CDB, are provided separately in the Management Response that accompanies this Report.
5
Table of Contents
6
Executive Summary
7
Part One: Introduction and BackgroundIn order to understand the development of the Office of Independent Evaluation’s (OIE) work, a brief description of the CDB’s current reforms is needed. First, there has been a change over the last decade in the nature of the programmes the bank supports; for example it has become increasingly engaged in funding policy-based operations and social development issues. Similarly, there have been changes in the whole of the development field, which is grappling to deal with complex issues such as gender and climate change. To meet today’s challenges and ensure that its work practices reflect the international standards of Multilateral Development Banks (MDBs), the CDB has introduced a number of measures aimed at improving its effectiveness and efficiency. For example, in line with international standards for Management for Development Results (MfDR), it has introduced a Results Based Management Framework for organising and assessing its performance.
In 2011, the CDB commissioned an external consultancy to undertake an assessment of its evaluation function in order to develop a policy that took account of good practices within the international development community.1 The CDB’s Evaluation Policy (referred to hereafter as the Policy) is a direct response to that review; it reflects the standards and good practices of the Evaluation Cooperation Group (ECG) of the Multilateral Development Banks (MDB) as well as the evaluation principles and standards of many professional associations.
Similarly, the Bank showed its commitment to having evaluation as a core function by establishing an independent evaluation unit that reports to the Board of Directors. It is responsible for assessing the Bank’s activities and interventions, but especially for drawing out the key lessons and recommendations for improving the Bank’s performance. As such, the monitoring tasks formally under the responsibility of the Evaluation and Oversight Division (EOV) were handed over to the Bank’s Operations and Economic Divisions. The OIE then validates the credibility and rigour of the self-evaluations.
In addition to the OIE, the Bank also set up other independent functions: internal audit, risk assessment and management, integrity, compliance and accountability. The mainstreaming of three cross-cutting themes (gender, energy and climate issues) into CDB’s work has also been initiated. At the same time, there are limited funds available as the CDB is working within a Board-sanctioned policy based on the principle of a zero real growth, which is in line with the budget policy of other MDBs.
In short, the bank has taken many important steps towards updating CDB’s management practices in line with other MDBs. However, the introduction of many innovations in parallel requires coordination and a shift in working practices and thinking. There is also the need to engage in different types of evaluation; evaluations that take into account cross-cutting themes and different levels of complexity. As such, whilst this review is particularly focused on the CDB’s Office of Independent Evaluation (OIE), its work and utility depend to a large degree on the development of other management practices and the degree to which evaluation is able to link into their work.
The Review in BriefA full description of the Review’s mandate, approach, process and methods are provided in Appendices I and II. It was designed to address the following four key questions as set out in the appended Terms of Reference and Approach Paper:
1 Osvaldo Feinstein & Patrick G. Grasso, Consultants, May 2011 Consultancy to Review the Independence of the Evaluation and Oversight Division of the Caribbean Development Bank
8
To what degree is the Office of Independent Evaluation independent at the strategic, functional and operational levels? Which measures help or hinder such independence?
To what extent is the OIE achieving its 2 strategic objectives? (which are (1) the timely delivery of good quality evaluations and PCR Reviews and (2) strengthening capacity building, networking and communication) How useful are the OIE’s procedures and products towards this end?
How adequate are the financial and human resources of the OIE for carrying out its tasks and achieving its objectives?
How effective is OIE in relating with its internal partners to develop evaluation capacity?
Our assessment of the OIE is largely based on the recommended criteria of the Evaluation Cooperation Group for Multilateral Development Banks; governance and independence, credibility, use and transparency.
The data used for analysing and interpreting the findings relied on exploratory, semi structured interviews with OIE staff as well as with CDB senior and middle managers and members of its Board of Directors. Whilst much of the interview data was collected during a 10-day intensive, on-site visit to the Bank, the majority of the Board members were interviewed through Skype. The interview data was complemented by a review of a range of key documents including the Bank’s Evaluation Policy, various kinds of reports on, or about evaluation, the complete set of minutes of meetings between the OIE and the Oversight Assurance Commission2 and the subsequent chairman’s report to the Board for the study period 2012 to 2015, OIE staff biographies as well as a number of other organisations’ evaluation principles, good practices and standards. A full list can be found in the Appendices (Appendix V). Not least, the Reviewers have also drawn on their own knowledge and experience of evaluation management to complement data analysis and interpretation.
Scope and Limitations
The Review was asked to concentrate on the 4-year period since the establishment of the OIE, January 2012 to December 2015, but more particularly on the changes introduced since the new Head of the OIE was appointed (June 2014 to December 2015).
It has mainly focussed on the strategic role of the OIE within the CDB as well as its functional and operational roles and responsibilities.
It was planned as a Review and not a fully-fledged evaluation; this was due to the limited time and resources available for the exercise as well as the fact that a ”light” review is in keeping with the spirit of the OIE’s Terms of Reference. The Review could not undertake any in-depth analysis of documents or consult with country level stakeholders or other external sources of expertise. Moreover, of the 29 people identified for interview, despite several reminders (by email or telephone) , the Panel were unable to either contact or secure the agreement of 5 of the 14 Board members, and 1 CDB senior manager. In light of this experience, as well as the time invested in securing the interviews “at a distance”, the planned on-line survey to follow-up on face-to-face interview data was abandoned.
We regret that in the time available, full justice could not be done to all the material provided to the Panel by the OIE. Nevertheless, the documentary review and interviews focussed on addressing the key questions, and we are therefore confident that the main issues raised in the Terms of Reference have been addressed in this report.2 The Audit and Post Evaluation Committee, now the Oversight Assurance Committee, is a Board Committee responsible for the oversight of evaluation and other key management functions.
9
10
Part Two: What the Review FoundIn the first place, the Panel should like to commend the CDB for its efforts in supporting the establishment and independent functioning of the OIE. Similarly, in spite of some of the challenges raised in this Review, the current Head of OIE and his team are to be commended for their efforts in advancing the evaluation function in the right direction of the UNEG Norms and Standards and ECG guidelines on Good Practices.
In this part of the report, we shall present our findings and conclusions in relation to each of the criteria below used to assess and respond to the four ToR questions:
the evaluation policy governance independence the OIE strategy, practices and work programme usefulness of evaluation, evaluation use communicating evaluation results adequacy of resources, and finally the working relationship between self and independent evaluation .
The Evaluation PolicyThe CDB Board agreed an Evaluation Policy (the Policy) in December 2011. It sets out the guiding principles and provisions for the OIE. It also aims at guaranteeing the independent functioning of the Office of Independent Evaluation (OIE) by having it report to the Board of Directors through the Oversight Assurance Committee, OAC. However, the President retains oversight on administrative matters such as travel, procurement of consultants.
Generally speaking, the Policy reflects many of the ECG’s recommendations on evaluation independence and good practices. Similarly, the evaluation criteria for judging outcomes are the five developed by the DAC, that is relevance, effectiveness, efficiency, impact and sustainability. In general, the Policy is intended to maximize the strategic value, timeliness and the learning aspect of evaluation.
Yet in reality, the Policy provides a framework for what could be achieved under optimal conditions. It is overambitious in terms of what could be done with the current level of resources. The Policy is considered useful as a reference mainly for the OIE, senior CDB staff and the OAC.
Many important tasks outlined in the evaluation policy, however, have not been done so far by either the OAC or the OIE. For instance, the OAC has yet to produce an annual report on OIE’s performance and the OIE has yet to establish a database of evaluation lessons, recommendations, actions and management responses.
To conclude: The Evaluation Policy is a mission statement of what could be achieved in time with sufficient financial and human resourcing. It reflects the internationally recognized evaluation principles and standards, but is probably somewhat ambitious for the OIE to fully put into practice for a number of years.
Governance IssuesOversight of the OIE is entrusted to a Committee of the Board of Directors (originally called the Audit and Post-Evaluation Committee, APEC, and now the Oversight Assurance Committee, OAC, to reflect its broadened mandate since October 2015). The OIE reports to the Board through the OAC. There are 5 members, of which only 2 are located in Barbados.
11
The OAC meets 5 times per year, the day before Board meetings. It has oversight responsibility for external and internal audit, independent evaluation, risk management and integrity, compliance and accountability in relation to CDB’s work.
The OAC Chairperson prepares a very brief resume of the day before’s meeting to present to the Board for its approval. The report generally covers progress, shortcomings and risks but is only a small part of the Board meeting so that generally there is little discussion; evaluation is only one of many items on the agenda. (We were told that the report to the Board averages approximately 10 minutes). Some of our interviewees could not recall any discussion about evaluation during Board meetings or remember reference being made to any evaluation report.
Yet there was keen interest expressed for having rigorous evaluation, especially in having it draw out important lessons about CDB interventions. Members interviewed perceive the OIE as a credible entity and are satisfied with its methodological approach. The Panel was therefore surprised to find that despite OAC’s awareness of the data problems in the BMCs (e.g. lack of rigorous monitoring and statistical data and the consequent effect on the rigour of OIE’s evaluations) as well as the delays in the submission of self-evaluations and their validations, the OAC’s reaction is negligible. The Panel could find no evidence to show that the OAC has exerted any pressure on the CDB or on the BMCs through their representatives on the Board to redress these problems.
A major problem for the OAC is the volume of paperwork and length of individual documents it receives in parallel from the CDB and its independent offices, generally very shortly ahead of its meetings. Both Board and OAC members expressed their deep concern about the need for the more timely delivery of reports and background papers for their meetings. The OAC members fear they are unable to do justice to their oversight responsibilities. Hence, based on the Panel’s review of the minutes and comments from the OIE, the meetings appear to be more formalistic, with OIE’s presentation of highlights from evaluation reports and management’s response, but little evidence to suggest that there is much discussion or systematic follow up on the recommendations, agreed actions or the lessons drawn, The “follow up on actions agreed” does not appear to be a systematic item on each OAC meeting’s agenda..3 Similarly any attempt to identify key messages for various stakeholders other than the CDB is not mentioned in the minutes or reports to the Board.
In response, the OIE has greatly improved the presentation of its technical reports by summarising the main points in its “Brief Reports” (e.g. the Tax Administration and Tax Reform and Technical and Vocational Education and Training evaluation). This is commendable and certainly a step in the right direction although the Panel considers that they should have a sharper focus on the strategic issues (which are the end of the brief rather than the beginning), be condensed and be made more “reader friendly”.
To conclude: The OAC firmly supports having an independent evaluation function which produces rigorous evaluations. It attaches much importance to evaluation’s ability to highlight key lessons. However, the OAC it is not performing its oversight function with sufficient firmness to bring about any change regarding the challenges evaluation raises or has to deal with. This is not helped by the lack of any systematic report on “follow up of actions agreed” which could be particularly useful for tracking changes as a consequence of an evaluation and management’s response.
The OAC could do better justice to its oversight responsibly if it were to receive all background documents systematically at least two weeks before its meetings. Moreover, the volume and length of documents received at any one time is considered to be overwhelming. The number
3 At the APEC meeting in May 2012, it was agreed that the OIE would prepare a Management Action Record to highlight the follow up actions taken to the recommendations of all evaluation reports, every two years, with the first report presented to APEC at the March 2013 Board Meeting. There is no record of this having ever been done or of the APEC / OAC’s following up on such request.
12
and/or importance of agenda items competing for attention at any one session is an additional handicap.
Independence of the Office of Independent Evaluation (OIEIndependence is central to the integrity and trustworthiness of evaluation. It is an agreed requirement within the development agencies and in the evaluation community as a whole. In examining the issue of independence and good practice, reviewers are guided by the Evaluation Cooperation Group’s recommendations on good practices, the CDB’s Evaluation Policy and by the 2011 consultancy review of independence relative to the CDB’s evaluation and oversight division4. The appraisal is based on a comparison of the ECG’s recommendations on independence5 and the current OIE status.
OIE and Independence: Recommendations from the OECD Evaluation Cooperation Group (ECG)
The ECG’s considers the issue of independence according to three specific areas: organisational, or structural independence, behavioural, or functional independence and protection from outside interference, or operational independence.
Organizational independence, ensures that the evaluation unit and staff are protected against any influence or control by senior or line management, and have unrestricted access to all documents and information sources needed for conducting their evaluations. Also, that the scope of evaluations selected can cover all relevant aspects of their institution.
Behavioural independence, generally refers to the evaluation unit’s autonomy in selecting, conducting and setting its work programme and in producing quality reports which can be delivered without management interference.
Protection from outside interference refers to the extent to which the evaluation function is autonomous in setting its priorities, and conducting its studies and processes and in reaching its judgments, and in managing its human and budget resources without management interference.
Conflict of interest safeguards refers to protection against staff conflict of interests be they current, immediate, future or prior professional and personal relationships and considerations or financial interests for which there should be provision in the institution’s human resource policies.
The OIE’s Independence in Practice
A comparison of ECG recommendations with OIE practice is presented in Appendix IV, (Tables 1, organisational independence, Table 2, behavioural independence, and Table 3, protection from external influence or interference.) This section summarises the Panel’s assessment of each of these aspects. 6
Organisational / structural independence
The CDB has succeeded in establishing an independent, stand-alone office that has direct dialogue with the OAC, the Board and senior management. However, there appears to be a detachment between the OIE and CDB that is of concern to the Panel; on the one hand, between
4 Osvaldo Feinstein & Patrick G. Grasso, Consultants, May 2011 Consultancy to Review the Independence of the Evaluation and Oversight Division of the Caribbean Development Bank5 ECG 2014 Evaluation Good Practice Standards, Template for Assessing the Independence of Evaluation Organizations, Annexe II.1 6 Based on ECG (2014) Template for Assessing the Independence of Evaluation Organizations, Evaluation Good
Practice Standards, Annexe II.1
13
the OIE and operations staff, and (2) on the other, in terms of the structural arrangements between the OIE and senior management.
1) In agreeing for the OIE to concentrate on strategic and thematic, in-depth evaluations, responsibility for project monitoring and evaluation were given over to the operational departments. The division is clear and respected. However, it has its drawbacks. With the OIE no longer systematically involved at the front-end of project design, the monitoring data needs are likely to be poorly defined. Weak monitoring data will contribute to weaker evaluations.
In the reviewers’ opinion, it is a common misunderstanding to assume that providing evaluator advice on monitoring and evaluation data will comprise evaluator independence. On the contrary, evaluation input into project design is essential to assure that the logic, indicators and data needs are addressed so that at some future point in time an evaluation of the achievements can be empirically grounded.
This is not to say that the OIE no longer has any influence at the front-end design stage; it has merely shifted the point of focus. The OIE is now systematically providing such input more generally to the corporate planning teams for the tools and systems they are developing to support the MfDR framework. The monitoring data for projects and their implementation should be improved once the Project Performance Evaluation System (PPES) and the Portfolio Performance Management System (PPMS) are updated and operational.
2) In the second place, the OIE has limited formal access to the Advisory Management Team (AMT) weekly meetings where the President and senior management gather to exchange up-to-date information on the dynamics of CDB policy and practice. The OIE is not regularly invited in any capacity to these meetings or given a copy of the agenda or minutes; the OIE is occasionally invited to attend in order to discuss an evaluation report or management feedback. For the OIE, this means that it is unlikely to pick up on the ‘when’ and ‘what’ of key decisional issues or provide input into the discussion based on evaluative information. Its observers status at Loans Committee meetings, or as a participant informer at the OAC and BoD meetings and discussions do not necessarily provide the same insight as to the dynamics of management actions and/or decisions. .
To respond to this situation, the President has agreed to meet regularly with the Head of the OIE in order to keep him up to date with CDB strategic thinking. This is a welcomed change.
OIE Independence and Behavioural Issues
The Panel has some concerns about behavioural issues. For example, there is a lack of quality data available and considerable delays are incurred in processing both the independent evaluation reports as well as OIE’s validation of the CDB’s self-evaluations. Delays are generally due to receiving feedback on the independent reports from first, the relevant operations department, then from the AMT, and then on providing the OIE with a management response that is initially drafted by operations staff before being reviewed by the AMT. (OIE reports cannot be submitted to the OAC without the relevant management response). This two-layer process for preparing submissions to the Board is inefficient and could potentially be a threat to evaluation’s independence in the future by delaying OIE’s timely reporting to the OAC.
OIE validations of the CDB self-evaluations are also submitted to the OAC, but it is in both sides’ interest to clear up any misunderstandings beforehand. Despite attempts to improve the timeframe for completing these validations, delays are more the norm than the exception.
14
OIE and Protection from External influence or interference
The OIE’s independence in the design, conduct and content of its evaluations does not appear to be subjected to any external interference. But securing funding from any sources outside the OIE’s administrative budget, i.e. from the Social Development Fund, is an unduly complex and long process. As such we consider that the current funding process can affect the OIE’s choice with regard to the type of evaluations it can undertake as well as the methodologies of its studies. (See Figures 1 and 2 later in this Review)
Avoidance of Financial, Personal or Professional conflicts of interest
This particular aspect refers to the organisation’s Human Resources Policy; there must be provisions in place to protect against staff conflicts of interest, past, present or potential. We requested via the OIE, to have evidence from human resources on any such provisions but did not receive an answer. We can only assume that this aspect of independence, past or present, does indeed form part of normal CDB Human Resource Policies
To conclude: We are impressed with the measures CDB has taken to assure the organisational independence of the OIE. Senior and line management alike accepts its independent status. But there is some indication of independence having possibly gone too far. The OIE is operating at arm’s length from operations and has limited access to senior management strategic discussions. Regarding behavioural independence, the quality of OIE’s reports is affected by the lack of rigorous data and many types of delays; difficulty in easily accessing documentation, and delays in the exchange of reports between the OIE and operations area. Both have affected the timely delivery of reports to the Board. Few evaluation reports are publically available. The OIE’s resources are limited; this is a hindrance to the OIE’s independence since it cannot cover the full range of MDB type evaluations or have complete autonomy over the selection of themes and methodologies. As for protection from outside interference, our concerns are largely to do with OIE’s independence over staffing issue; there are potential loopholes in current arrangements that could undermine OIE’s autonomy over its staff.
OIE’s Strategy, Work Practices and Work ProgrammeFollowing the approval of the CDB’s Evaluation Policy, the OIE attempted to develop a plan for its implementation. It considered such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but the programme proved to be over ambitious.
Much of the period 2012 to 2015 has been taken up with preparing for OIE’s shift in focus from project-based evaluations to the high-level thematic and in-depth strategic studies. From 2014 the OIE has therefore adopted a three-way approach; (1) for self-evaluations, reducing its time input to support the process and (2) for independent evaluations, taking stock of the gaps in coverage and expertise, and increasing the involvement of its own staff in conducting evaluations (3) networking to share experiences with centres of expertise and align OIE with international practices. The OIE plans to conduct 2-4 high-level studies per year from 2016. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed
But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities (e.g. knowledge management and brokerage in particular) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set delivering utility-focused and timely evaluations as a key objective. But it lacks clarity on how the OIE proposes to surmount the time and data issues, which are far from new. The challenges that have to be dealt with to enable the
15
OIE to move up the MDB evaluation pyramid7 are brought out in the remaining sections of this Review, not least given the limited resources available. (See Appendix III for more on OIE’s work practices)
To conclude: The OIE has made a first step in proposing a strategy for establishing itself as an independent evaluation resource and implementing the Evaluation Policy. The strategy as it stands, however, does not sufficiently take account of the full range of activities and time commitment needed to support evaluation management. Furthermore, there is no road map to indicate how the OIE intends to realise the full range of tasks and responsibilities set out in the Evaluation Policy, and/or which should be prioritised.
The Value / Usefulness of OIE’s Independent EvaluationsEvaluation is a powerful tool that can provide useful, evidence-based information to help inform and influence policy and practice. But useful evaluations depend not only on the evaluators’ skills, but on several other important factors as well; 1) on planning evaluations to be relevant to the priorities of the organisation’s work and for their results to be delivered in time to be useful; on the degree of 2) consultation and ultimately ownership by those who seek evaluative information; on the 3) tools used to support the evaluation process per se; and on the 4) credibility and quality of the evaluation products8.
1. Planning relevant and timely evaluations
The OIE is now working on a 3 year rolling work plan that sets out the broad areas for enquiry. So far, there are no agreed criteria for making the selection of the specific topics for independent evaluation, although the priorities tend to reflect those of the CDB’s strategic plan. Nevertheless decision-making is rather arbitrary based on a process of dialogue between the OIE and the CDB and the OIE and the Board.
One of the OIE’s two objectives for 2015 therefore, was to define a work plan and agree priorities based on an approach that is “utilisation-focused”. This means that the studies are selected and planned to be relevant and useful to the organisation’s needs.
The OIE has achieved this objective with respect to its latest studies, which concerns the Social Development Fund (SDF) Multicycle 6&7 Evaluation, the Haiti Country Strategy evaluation and the evaluation of the CDB’s Policy Based Operations. Each of these three have been planned to deliver their results in time to provide the CDB Board of Directors with relevant information for negotiating the next round of funding. In spite of some delays due to a myriad of reasons, not least to the extra effort needed to secure essential data, the studies will deliver on time.
The processes for agreeing OIE’s work plan and specific evaluations on the one hand, and, in securing alternative funding on the other, are shown in Figure 1 below. The Panel was surprised at learning how overly bureaucratic (the internal approval process), and inefficient (in view of the time it takes) the process seems to be. Our concern here is that such a process could possibly pose a threat to assuring the Board of “timely studies.”
Figure 1: Selection of Evaluation Topics and Funding Source
7 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).8 These aspects reflect the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
Consultation with CDB Operations and OAC/Board for selection of
evaluation topic
3-year Work Programme and Budget (approved by Board)
Annual OIE report and work plan
submission to OAC
16
Internal review of Approach Paper
Specific Evaluation Study Design and Budgeting
OIE Draft Terms of Reference / Approach
Paper
Detailed ToR or Final Approach Paper if sufficiently detailed.
Finalise Approach Paper and submit to OAC/Board
Final Approach Paper
OAC ApprovalOAC minutes
Paper
Funding Track
Final Approach Paper/ToR
Board approval necessary If above USD
150,000
Board notification only if USD 150,000 or
below
Board Approval
Board Paper
OIE – Selection of consultants (if any) contracting
OIE Admin Budget or …
… SDF
Prepare TA Paper (content similar to Approach Paper but different
format.
TA Paper
Approval – Internal Loans Committee
OIE – Selection of consultants (if any)
contracting
17
To summarise, with the latest three independent evaluations, the OIE has achieved its 2015 objective of planning and delivering useful and timely studies. But the process in securing approval and funding, particularly for studies requiring additional resources, is inefficient.
2. Consultation and ownership
“The credibility of evaluations depends to some degree on whether and how the organization’s approach to evaluation fosters partnership and helps build ownership and capacity in developing countries.”
(ECG good practices)
As previously mentioned, the OIE engages with the OAC, CDB senior management and operational departments for agreeing its 3-year work plan and then for selecting the specific topics and themes. It also discusses the evaluation approach paper (design and implementation plan) with the CDB and OAC before completing the final version. However, preliminary and final drafts of the report are only submitted to the CDB line and senior managers for comment and factual errors; the final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the results and their implications. Discussions with the OAC are more limited due to the overburdened agenda of OAC and Board meetings, as previously discussed.
Figure 2 below provides an overview of the evaluation implementation and stakeholder engagement processes.
Figure 2: Evaluation Study Implementation and Feedback Loops
Arrangement AFully outsourced / external
consultants; oversight by OIE
Preparations:Detailed evaluation plan (incl tools,
timeline, etc.) and logistics
Production of Inception Report / Approach Paper
Arrangement BConducted by OIE
staff
Arrangement CJointly: external
consultants and OIE
Terms of Reference
Prepares Inception Report /
Approach Paper OIE
Board notification only if USD 150,000 or
below
Data Collection and AnalysisBoard notification only
if USD 150,000 or below
18
1.2.
Notes to Figure 2
1. The OIE informed the Panel that this is an abbreviated version as there are e.g. additional steps (secondary processes) when evaluations are procured (tendering or single source), when there are additional review loops and updates to OAC etc.
2. OAC may also decide to return the report to OIE, the Panel were informed, or demand from Management specific actions based on the report.
This process is engaging and appears to have secured senior management and OAC interest and buy-in as witnessed in the latest studies. But there is the downside too! The process takes much time and, in our view, is partly unnecessary. The Panel appreciates that staff from operations as well as the AMT may both want to confer on an appropriate management response, but this should not be the case for reviewing an independent report for factual errors. The two-phase approach seems somewhat inefficient and unnecessary in our opinion.
Prepare for disclosure and dissemination
Presentation/workshop:Interim findings and conclusions for immediate feedback and validation
Summary and ppt for workshop presentation
and discussion with CDBSubmission of Draft Final
Report to OIE
Final OIE approved report to CDB Senior Management for Management Response
Draft Final Report
Review loops – OIE and CDB (potentially also BMC)
Feedback to evaluation lead
Submission of Final Report to
OIE
Final Report
Final Report and Management Response submitted to
OAC/BoardFinal Report and
Mgt. Resp.
Management Response
OIE ApprovalFinal Report and Management Response considered by CDB
AMT
OAC/Board endorsed
19
Contact between the OIE, the CDB and/or the OAC during the actual study implementation is most often restricted to the occasional progress report, particularly when studies run behind time. Occasionally, however, the OIE arranges discussions with operations for reflecting on emerging findings, but we are not sure of how systematically this feedback loop is.
There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of other contexts for improving buy-in and providing strategic input as well.
More generally speaking, outside of an evaluation study, the OIE has limited dealings with the operations. The OIE has an advisory role in providing them with help, particularly with providing training, guidelines and tools to support self-evaluations. We are nevertheless concerned about the seeming distance between these two and how this has affected the perceived value of evaluation. (For further on this point, please see the section below on “Self- and Independent Evaluations”)
But the Panel wishes to stress that this is not the case for newly appointed senior management. We found a much more open attitude to evaluation and appreciation of its potential value. For example, we learned that the OIE was recently invited by a senior manager to share evaluative knowledge and experience with his staff regarding policy based operations.
Certainly, we can say that overall, the key stakeholders within the CDB are adequately integrated into the evaluation process as to foster their buy-in and ownership. However, the Panel considers the process of reviewing draft reports is inefficient. The operations staff is less convinced than senior management about the value of evaluation. There are no “advisory groups” in place to accompany individual studies, which can add value to ownership as well as the perceived value of evaluation. Such a group could help improve evaluation’s image.
3. Tools to support the evaluation process
So far, the OIE has mainly focussed on improving the tools to support operations and its self-evaluations. This has left the OIE with little time to produce the checklists or tools to support its own studies. There are plans to develop an OIE Manual for guiding its own processes. Such plans should be encouraged, as these documents will form a very important part of training, particularly for newcomers to the OIE team.
In the meantime, the OIE refers to the Performance Assessment System (PAS) Manuals for its independent evaluations; the operations area also use the same for completing their reports on public sector investment, lending and technical assistance, policy based loans, and country strategy programmes. The manuals are based on DAC criteria and ECG principles. Much emphasis is given to the rating system and how and what should be rated. However we find them lengthy, unwieldy and overcomplicated. Moreover, such manuals should be used for reference, but cannot and should not replace first-hand training in how to plan, conduct and manage the evaluation process.
Quality Assessment (QA) and Quality at Entry (QaE)
There was a transition period between 2012 and 2014 to establish the OIE. Work on the PAS, QaE, PCRs, ARPP, which had started earlier, was therefore completed after OIE came into existence, but it effectively had no formal ‘home’ in Operations. The Panel was told that there had been some discussions about creating a Quality Assurance unit within CDB (OPS) but the current status is unclear.
The QaE Guidance Questionnaire was developed before and completed by the OIE. It was used to assess the documents that came across to the OIE for comments at the Review Stage. The results were then sent to the Portfolio Manager/Project Coordinator indicating any gaps/issues that needed to be addressed or clarified. QaE Guidance Questionnaires were developed for all the Bank’s lending products, CSP and to assess the quality of supervision.
20
After the QaE was launched bank wide, several operations officers saw the merit in using the QaE Guidance Questionnaire in the field and adopted it as a tool for their use during the appraisal mission in order to cross check and test their data collection and analysis.
The use of the QaE by OIE was discontinued in 2014 due to limited resources and a stronger focus on evaluations. It still sometimes comments on specific appraisals, but very selectively.
Both QaE and QaS (quality at supervision) are also addressed in the PAS Manuals. In addition the QaE and PAS have been incorporated in Volume 2 of the Operations Manual OPPM.
The Review Panel assessed the QaE forms. They are relatively standard, adapted to the specificities of the CDB. They contribute to judging a project’s expected quality in a relatively objective way. As such, they are are helpful, as a benchmark, in the ex-post assessment of projects.
The Panel considers that the lack of an established Quality Unit in the CDB (and independent from OIE) is a weakness that should be addressed in the near future.
4. Credibility and Quality of Evaluation Products
As with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
Independent evaluations are processed as follows; the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis).
During this period of transition, much of OIE’s work has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. With the change of Head in June 2014, it has secured the OAC’s agreement to reduce the number of validations to a maximum of 6 per year out of an annual average of approximately 15 in theory. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
With regard to the independent evaluations, since 2012, the OIE has produced a range of studies and approach papers. Our review is based on those listed below as provided by the OIE, and cover the period from May 2012 to December 2015. The list includes 3 evaluations (in blue), 4 Assessment studies (in brown) 14 validations of self-evaluations (in green) and 3 Approach Papers (in purple) for upcoming evaluations. These are listed below in Table 4.
Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
Board Meeting
Date Type / Topic
251 May 2012 Ex-Post Evaluation Report on Road Improvement and Maintenance Project, Nevis -St. Kitts and Nevis.
Validation of Project Completion Report on Sites and Services – Grenada.
21
Assessment of Effectiveness of Implementation of Poverty Reduction Strategy 2004-09.
253 Oct. 2012 Assessment of Extent and Effectiveness of Mainstreaming Environment, Climate Change, Disaster Management at CDB.
254 Dec. 2012 Assessment of the Implementation Effectiveness of the Gender Equality Policy and Operational Strategy of the Caribbean Development Bank.
Validation of Project Completion Report on Enhancement of Technical and Vocational Education and Training – Belize.
Assessment of the Effectiveness of the Policy-based Lending Instrument.256 May 2013 Validation of Project Completion Report on Expansion of Grantley Adams
International Airport – Barbados. Validation of Project Completion Report on Fifth Water Supply Project –
Saint Lucia. 261 May 2014 Validation of Project Completion Report on Immediate Response Loan,
Tropical Storm Gustav, Jamaica. Validation of Project Completion Report on Social Investment Fund,
Jamaica. Validation of Project Completion Report on Disaster Mitigation and
Restoration – Rockfall and Landslip, Grenada.263 Oct. 2014 Validation of Project Completion Report on Basic Education Project –
Antigua and Barbuda263 Oct. 2014 Approach Paper for SDF 6 & 7 Multicycle Evaluation
264 Dec. 2014 Validation of Project Completion Report on Policy-Based Loan – Anguilla Validation of Project Completion Report on Immediate Response Loan -
Tropical Storm Arthur – Belize. Evaluation of Technical Assistance Interventions of the Caribbean
Development Bank Related To Tax Administration and Tax Reform in The Borrowing Member Countries 2005-2012.
265 March
2015
Approach Paper for the Evaluation of Policy Based Operations
266 May 2015 Validation of Project Completion Report on Upgrading of Ecotourism Sites – Dominica
The Evaluation of the Caribbean Development Bank’s Intervention in Technical and Vocational Education and Training (1990-2012)
267 July 2015 Validation of Project Completion Report on The Belize Social Investment Fund I Project − Belize
268 Oct.2015 Approach Paper Country Strategy and Programme Evaluation, Haiti
Our review and analysis is based on the UNEG Quality Checklist for Evaluation Reports (http://www.uneval.org/document/detail/607) as well as on ECG guidance (Big Book on Good Practice Standards).
Approach Papers
Three Approach Papers (APs) were available to the panel for review (see Table [ref] above). An AP describes the rationale for the evaluation, the background to the topic evaluated, the evaluation framework (criteria and questions) and approach. It also describes the team and provides an initial planning. Being the first main deliverable of OIE’s evaluation process, APs are the starting point and therefore a major determining element in the roll-out of each evaluation. Therefore APs “have to get it right”.
22
The APs examined are clearly written, well-structured and of reasonable length.9 We were surprised to find, however, that they do not make explicit the objectives of the evaluated intervention(s), e.g., through a clear objective tree, or through an explicit theory of change, intervention logic or logframe. Whilst one of the APs contains, in an appendix, a results framework for the evaluation, the results framework for the intervention (PBO) itself is lacking.
Inception reports
Only one Inception Report was available for review (SDF 6&7). This gives an in-depth description of the evaluated programme and provides a clear Theory of Change. It is good practice that this is established after a pilot field mission, which helps to amend the initial AP on the basis of field observations and sharpen the evaluation questions if needed.
However, it is still considered to be good practice to have the Theory of Change elaborated in the initial design documents . This would facilitate OIE evaluations after project completion. Establishing the Theory of Change of any intervention would be included in the QaE form more explicitly, to be developed between the Quality unit referred to above, and OIE.
Evaluations and Assessments
Three evaluations and four assessment reports completed during our review period were considered for this review. Assessments are similar to evaluations but have a narrower scope; they focus on a limited set of aspects or judgment criteria, mainly effectiveness, i.e. achievement of objectives. Evaluations generally base their judgment on the internationally recognised DAC criteria as well as aspects of the CDB and BMC’s management of the intervention.
In general, these reports are of reasonable quality. In the main, they explain the evaluated object10 and provide evaluation objectives. The findings are organised around the evaluation criteria or questions detailed in the scope and objectives section of the report. They are based on evidence derived from data collection and analysis methods as described in the methodology section. The reports tend to dwell on the limitations that the evaluation encountered, but without becoming defensive. In one case (PBL Assessment) the report starts with a summary of the reviews on the topic done by other MDBs. This was a pleasant surprise and indeed a good practice that could well be adopted in future evaluations too.
However, the reports also show several significant weaknesses:
- Reports do not always provide clear (reconstructed) intervention logics or theories of change for the intervention(s) evaluated.11 Evaluation criteria and questions are defined at a fairly general level. They are translated into more precise “research questions” (in an “Evaluation Design Matrix”, for each project for each criterion). However, it is unclear how these questions relate to the intervention logic (as this is not made explicit). This may be done in inception reports (of which, as noted above, only one was available for review), but should be done also in the final evaluation reports.
- The reports do not describe the link from the evaluation questions to the answers, how the evaluation judgments are made and how these ultimately transform into ratings for each criterion and each project. In other words, the explanation provided in the evaluation frameworks is inadequate. The “evaluation design matrix” currently used does not provide
9 Opportunities remain of course to be more concise and to move parts to appendices, e.g., detailed descriptions of the evaluation team or part of the description of the evaluated intervention.10 Sometimes in great length: for instance with the SDF 6&7 multicycle evaluation report it is only at page 30 that we find the beginning of the report on findings…11 Again with the SDF 6&7 evaluation, it is said to be guided by a “Logic Model” which is not explained.
23
sufficient insight into how ultimately an intervention’s performance is judged.12 Links between findings, conclusions and recommendations could be improved by making this more explicit. In other words, reports should include the story on how the evaluand is credibly linked to any observed outcomes and impacts, and should be clear on how causal claims are made.
- With the exception of the PBL Assessment, reports are lengthy and detailed. One reason for this is an over-emphasis on ratings. Their detailed discussion, project by project, criterion by criterion, occupies a very prominent position in the evaluation reports’ main body of text. Although ratings are traditionally an important element in evaluations of MDBs, too strong an emphasis can be tedious and may distract the reader from the real lessons to be drawn. The detailed discussion of ratings, and their evidence base, would be better placed in an Appendix, with a brief summary in the main report. This would help give the lessons and recommendations a more prominent position than is now the case. This would also help make the evaluation reports not only shorter but also more interesting to read; this could help add value to evaluation’s image within the organisation.
- The reviewers feel that the OIE evaluations tend to over-emphasise objective-based evaluation13 and the DAC criteria to the exclusions of considering other evaluation approaches such as Developmental Evaluation (Patton, 201014); evaluation should be case specific and answer the actual information needs of managers and other decisions makers rather than always concentrating on final performance.
- Related to the previous point (and again with the exception of the PBL Assessment) executive summaries (approximately 8 pages) are too long. For the evaluation report to increase potential impact, they would need to be reduced to 2 to 3 pages and be more focused; again this could be done by dwelling less on the individual ratings of projects and more on key findings, lessons and conclusions. More generally, reports could be better adapted to the needs of the different audiences. Although not strictly limited to evaluations, The Health Evidence Network Reports15 are a model that could be adapted for evaluation reporting purposes; they are specifically geared towards addressing policy and decision-making.
- The “Recommendations to BMCs” are an interesting feature of the reports, (although we are unsure to what degree such recommendations could be effectively followed up by OIE or the Bank, but certainly could taken up with BMC Board members.
- Reports (e.g. the evaluation report on Technical Assistance) focus much on technical problems that were encountered during the evaluation. Although these are important issues, again to improve the report’s flow and “readability” this section would be better placed in the Appendix. What counts is the story of the intervention, not the story of the evaluation (see “Limitations” section in the TA report for instance)
12 Marlène: I moreover have the idea that the methodology (often described as “visits”) is based on interviews and little hard evidence. Any view on this?.JM: My “interview-based evalutions”!!13 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.14 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press15 See the reports available at the WHO’s Health Evidence Netowkr at http://www.euro.who.int/en/data-and-evidence/evidence-informed-policy-making/health-evidence-network-hen
24
- OIE Validations of Project and Country Strategy Programme Completion Reports (referred to globally as PCRs hereafter)
As with other MDBs, the OIE has the mandate to validate the operational departments PCRs. The validations tend to repeat the different items reported in the PCRs and then provide extensive comment on each. The PCVRs go into great depth and detail, which makes the documents rich and complete. This is their strength – but also their weakness. The depth and level of detail, as well as the repetitions from the original PCRs, makes PCVRs (overly) lengthy (20-40 pages) and difficult to read. The OIE reported spending approximately 27.2% of its time on validating PCRs in 2015 compared with 44.4% on its core work, i.e. doing or managing the higher level evaluations. That is more than half of its evaluation work is being spent on the validation process. Finally, the PCVRs now seem to be, to a great extent, a standalone output of OIE. It is not always clear to us how they are being used as the “building blocks” for the OIE’s independent evaluations. Making this clearer in the independent evaluations would help show the link and therefore the value of the time being spent on the self-evaluation validations.
To conclude, the review finds that the OIE has taken steps to improve the perceived utility of evaluation in several ways. In the first instance, by planning its work to provide relevant and timely evidence geared towards helping the Board with its oversight and decision making tasks. The topics are selected through dialogue between the OIE and key CDB stakeholders and reflect priorities of the CDBs strategic plan. Secondly, by securing the interest and consequently the buy-in of the OAC and CDB senior management through engaging their input throughout the evaluation process. This is evidenced by the reported interest in the latest three studies, the Country strategy programme in Haiti, the evaluation of policy-based operations and the SDF 6& 7 multicycle assessment.
The OIE products are of an acceptable quality and could be even better if some of the shortcomings were addressed. However, the products themselves do not impair the utility of OIE’s work; this is undermined in several ways: (1) by the time delays in commenting on PCRs (OIE) and providing feedback to the independent evaluations (operations and management) (2) by the inefficient processes for agreeing topics and funding sources as well as providing OIE with management responses to its reports.
Putting Evaluation to Use: transparency, feedback and follow-upThere are several ways that evaluation can be, and is being used. As John Mayne has pointed out in his many publications on the issue,16 when we talk of evaluation use, we are mainly thinking about its Instrumental use—use made to directly improve programming and performance. But there is also conceptual use - use which often goes unnoticed or more precisely, unmeasured. This refers to the kind of use made to enhance knowledge about the type of intervention under study in a more general way. Or even Reflective use— this refers to using discussions or workshops to encourage and support reflection on the evaluation findings to see how they might contribute to future strategies.
In the case of the CDB there is some evidence to suggest that “use” is not only instrumental, but other types are also developing. For example, in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessons. (Reflective use)
Another important use of evaluation, as recommended by the ECG, is that from time to time a synthesis of lessons is drawn from a number of evaluations and made available publically. In fact the Panel was impressed to hear that in the past, the evaluation unit had drawn together a synthesis of lessons from evaluations of the power sector. (Conceptual use) But none have been
16 See for example, his opening chapter to Enhancing Evaluation use: Insights from internal Evaluation Units, Läubli Loud, M. and Mayne, J. 2014, Sage Publications
25
done since setting up the OIE although it is now on the “to do list” for 2016 (OIE’s 2016 Work Plan).
As for instrumental use, responsibility for using the knowledge generated through evaluation and for possibly drawing up an action plan of what should be done is up to CDB senior management and the relevant CDB department and division. Oversight on applying recommendations and picking up on the lessons drawn is the responsibility of the OAC.
Evidence on how evaluations have actually contributed to decisions or negotiations is lacking, Certainly the OIE is unaware of the extent to which its evaluations are put to use. On the one hand we read in the OAC minutes that lessons learned are integrated into the next phase. On the other, we were told that often in the past, the evaluation results were “too old” to be of use as the lessons had already been drawn and used way before the report was completed. Similarly, people’s gaps in memory on how well the evaluative information from previous studies may have been used may also account for the scarcity of evidence.
In response, the Panel questioned CDB staff and the OIE about a particular study, the Technical and Vocational Education and Training Assessment. The feedback was somewhat contradictory. On the one hand, the study was criticised as “confirming” news rather than bringing “new news”. However, on the other, we learned that In October 2015, the Board of Directors approved a proposal for the revision of CDB’s Education and Training Policy and Strategy. Work on this has already begun and an external consultant has been engaged to lead the process.
Although it is one of the OIE’s tasks to set up a database on results and lessons learned from evaluations, so far this has not been a priority. There is also currently no systematic tracking of lessons or recommendations arising from the evaluations, or on any progress in their uptake. The OAC has on occasion, raised questions about specific evaluations and asked that they be kept informed on progress as an “action information” on subsequent agenda. However, in our review of the minutes from 2012 to December 2015 such requests do not appear to have been followed-up. The OAC oversight of evaluation use appears to be insufficient.
The OIE’s role in supporting CDB’s organisational learning is clearly specified in the Evaluation Policy, with many good suggestions for knowledge sharing activities such as “brown-bag lunches, workshops, pamphlets and short issues papers” (p. 19). So far, however, the OIE’s lead role on the knowledge sharing side appears to be quite limited. It has provided advisory input in Loan Committee discussions, and organises workshops together with the relative operational department for discussing the implications of evaluation studies. Ultimately, of course, the uptake of evaluation results and knowledge is in the hands of management. But the evaluation unit has an important role to play in terms of knowledge broker and knowledge manager. Both have tended to be underplayed in OIE’s work plan so far.
Transparency: The Communication StrategyIn recent times and with the approval of its new Disclosure Policy, the CDB has started to post its independent evaluation reports on its website. (There is nothing on the self-evaluations). The website also presents a good overview of the role and function of the OIE and evaluation within the CDB. This is a step in the right direction for sharing information. However, in our view, the CDB’s communication strategy is the weakest part of the evaluation system to date.
We have already commended the OIE in its efforts to engage the CDB and the OAC in deciding on which evaluations to pursue, on the draft reports, their conclusions and recommendations, as well as on the lessons drawn. But reporting and communicating the lessons seem to be entirely targeted at the Board and the CDB.
We feel that actively engaging with the more indirect stakeholders, for example project implementers in the BMCs, NGOs or project beneficiaries is relatively weak17. There appears to
17 A broader communication strategy is one of the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
26
be little reflection on drawing out significant messages for the broader group of stakeholders, or on how then to transmit them to the “right” people in the “right” way (knowledge brokerage). Moreover, the 2015 budget provides only US$2’000 for communication – nothing of which is intended for outreach.
To conclude, evidence on the uptake of evaluation is sparse. It is unfortunate that so far no systematic record keeping system has been put into place to track lessons learned or the uptake of recommendations (or actions agreed from management responses). Furthermore although the Evaluation Policy specifies the need for “distilling evaluation findings and lessons learned in appropriate formats for targeted audiences both within and outside the CDB” (p.19) such a targeted communication strategy has yet to be developed.
Strengthening Evaluation Capacities and Networking From the onset in 2012, the OIE has stressed the importance of developing and strengthening evaluation capacities within the OIE, the CDB and, subject to available resources, in borrowing member countries. Building evaluation capacity in BMCs and the CDB is one of the OIE’s mandated tasks. It has been a priority that figures on the work plan from the beginning (Work Programme and Budget 2012-2104) The idea of developing an internship programme for graduates from the Caribbean region was one idea that was advanced to help build local evaluation resources. However, the capacity-building has primarily been focused on OIE and CDB staff to date. One of the OIE’s two objectives for 2015 therefore was to take up the challenge and “strengthen evaluation capacities and networking” to include reaching out to the BMCs.
Developing OIE staff capacities
The change from project level to strategic and thematic evaluations does require different evaluative skills and competencies. The MDB Evaluation Pyramid presented below in Figure 3 shows the different types of evaluation and changing resource needs as one ascends the pyramid. Implicit here also is the change in the type of expertise and competencies needed as evaluation aspires to the higher levels.
In line with the OIE’s shift from project-based to in-depth thematic and strategic evaluations, for 2015, the OIE set itself the objective of networking and developing working partnerships with regional and international evaluation entities and academic institutions. The rationale was twofold: (1) secure further support and guidance as well as (2) increase its outreach and coverage through joint work and international exposure. Another implicit aim was to benefit from partners’ contacts in the BMCs wherever possible so as to improve data collection and quality.
Figure 3: The MDB Evaluation Pyramid18
18 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
27
The OIE has therefore linked up with Carleton University in Canada and the University of the West Indies, Barbados campus. The OIE was also approached by the Development Bank of South Africa to exchange experiences about setting up an evaluation entity in a “small” development bank. However, its attempt to become a member of the Evaluation Cooperation Group was not successful for reasons beyond its control.
The OIE is to be commended in addressing the issue of staff competencies and professional development more generally. New developments in evaluation as well as new developments in the scope of OIE’s work may necessitate new competencies. For this reason, organisations such as the International Developmental Evaluation Association have recommended that the competencies of evaluators and evaluation managers should be periodically reviewed. Several publications now exist on competency requirements and suggestions for the periodic review of staff competencies.19
It is not within this remit to compare and contrast OIE’s competencies with those recommended by international and national agencies. However, what we can say is that the OIE demonstrates great forethought in taking this on board.
Capacity building within CDBThe OIE’s objective also consists of continuing to develop measures for improving the monitoring and self-evaluation side of CDB’s work. OIE’s strategy here is to use the windows of opportunities on offer through some of the training sessions that are being organised by CDB as part of its shift towards MfDR e.g. by Corporate Planning Services and Technical Assistance. For 2016 it is planned to have the OIE present at the annual staff meeting and Learning Forum.
The OIE also organises some ad hoc training with operations, for example to help understand new tools e.g. for drawing out lessons from self-evaluation reports and, more generally, in
19 E.g. IDEAS, (2012) Competencies for Development Evaluation Evaluators, Managers and Commissioners, the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice (2010) and the Swiss Evaluation Society’s Evaluation Managers Competencies Framework (2014)
28
helping staff appreciate how evaluation can add value to the organisation’s work. Measures include providing advisory services on demand, and providing training alongside the introduction of new or revised tools.
Capacity building in the BMCs
This is an ambitious task and would require additional investment; from the bi-annual work plans, it would seem that it therefore tends to be put to the bottom of the to-do list. However, from what we understand, the OIE’s strategy is to join together with the Carleton University and the University of the West Indies, using their networks in some of the BMCs, to try to develop this aspect. But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.
To conclude, we cannot comment on the quality or reaction to such training, but can commend the OIE for making capacity building one of its priority objectives. From both the Policy and the documents we reviewed, we note that capacity building was always an issue to be tackled, but one which tended to be put to the bottom of the “to do” list. Working with the BMCs is a priority but will require focus and additional human and financial resources.
Adequacy of the OIE’s financial and human resources
Human Resources
OIE staffing consists of 5; the Head and 3 professional staff plus one administrator. Three of the five were recruited from within the CDB. There is some indication that the Board will be expecting the OIE to conduct impact evaluations at some time in the near future. The OIE itself is also keen to engage its staff in the independent studies and to focus on high-level evaluations. However, actual staff capacity contrasts with such increasing demand and expectations. The Panel is of the view that in trying to meet such expectations, the OIE will be overwhelmed and less likely to deliver credible and useful evaluations. of evaluation Moreover there are many other designated OIE activities that should be recognised as valuable work; the validations, building CDB and BMC evaluation capacity, providing supervision, advice, knowledge management and brokerage as well as managing evaluation contracts, The time needs of dealing with all of these may be underestimated in OIE’s work plans and budgets; all are important for assuring best value from evaluation. The Panel is concerned that a demand for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level evaluations may undermine the importance and time needs of other essential tasks.
Limited financial resources for OIE’s work programme
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses. (more details on budgets and OIE practices are provided in Appendix
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound
29
estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.
Self- and independent evaluationIn line with international standards, CDB’s evaluation system covers both self and independent evaluation. Self-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes. Both are important as the self-evaluations are at the very heart of the evaluation function; they are said to be the building blocks for the more strategic evaluations that the OIE is now undertaking.
The Evaluation Coordination Group recommends that the self-evaluations be carried out by the relevant operations department and in turn, reviewed and validated by the organisation’s independent evaluation office. The CDB’s Evaluation Policy therefore talks of “validating all self-evaluations” as being one of OIE’s essential oversight tasks.
Within CDB, the self-evaluations should provide management with performance assessments and thereby serve an accountability function to the CDB and Board. To support the process, the OIE provides operations with manuals and checklists for guidance. Once a self-evaluation report is to hand, it is given over to the OIE for the validation of its technical quality and credibility.20
However, in the CDB case, there are well-documented issues that have affected the quality and timeliness of the self-evaluations on the one hand, and therefore the quality of the foundation on which to build the independent evaluations. Paucity of documentation within CDB, paucity of data collected and available in the Borrowing Member Countries (BMCs), time delays in producing completion reports and in turn, having them validated by the OIE - all such issues were systematically raised during interviews and in some of the independent evaluation reports.
Generally speaking, many of the monitoring data problems appear to be due to a lack of management oversight. For example, with the introduction of results-based management, the logic frame and monitoring and data needs are systematically being built into intervention design. However, the BMCs are not delivering the data as contractually agreed at the outset.
20 According to the Evaluation Policy, OIE should validate all PCRs and CCRs but due to the backlog of reports and the delay in completing them (sometimes years later) since October 2015, the OIE has secured OAC agreement to validate a maximum of 6 per year, which are selected in consultation with the OAC.
30
Incentives to support any significant change towards building a results-based culture seem to be weak and sanctions seem to be rarely enforced when the supply of data is lacking or lengthy delays to the projects occur. Although we can appreciate the complexities of trying to enforce monitoring compliance, this means that often, project deadlines have had to be extended, data gaps are not being satisfactorily dealt with and in turn, there has been a void in the quality and quantity of available evidence for the CDB’s self-assessment of project performance.
For some time, this lack of oversight has been tolerated. Part of the problem is the very real problem of time; staff has more pressing priorities and there is little incentive to complete the self-evaluation reports in a timelier manner. There is also the absence of any focal point within senior management to drive the process and deal with the problems. According to the Evaluation Policy (p.15)
“The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”.
But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.
No record is kept of how the self-evaluation results are actually used. They do not appear on the CDB website, but we were told that the findings are integrated into the following project designs. Hence we are somewhat unclear as to the utility of these reports at present. The situation is exacerbated by a rather confused image of evaluation: some operations staff consider OIE’s input (through validations or independent evaluations) to be over-critical, regulatory and adding little value; it is seen as a threat rather than an opportunity for learning. Yet at the same time, evaluation is understood to be an integral part of result-based management.
Some managers appear keen to change the status quo. For example a revised and simplified template for producing project completion reports is being considered, and mid-term project reviews are expected to be more stringent in looking at monitoring plans and practices and tying disbursements to performance. In some cases we also learned of incentives being introduced to encourage project managers to complete their reports in a timelier manner. But much remains to be done and, since the OIE is no longer responsible for monitoring and project evaluations, there is a void that needs to be filled. It is up to line managers to drive this work forward.
To conclude, it is fair to say that in view of the above “frustrations” between the OIE and operations, the value that evaluation might offer to the operations area is ill recognized. the development of a learning and evaluation culture within CDB has been somewhat hindered. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advance a learning environment in which evaluation can play a major part. Moreover, with the transition to the OIE and change in focus, there is a void in the operations area that has not been satisfactorily dealt with.
31
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”21
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
21 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
32
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”22 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 23 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
22 CDB (2011) Evaluation Policy (p.2)23
33
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
34
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
35
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
36
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
Independence of the Office of Independent Evaluation (OIEIndependence is absolutely central to the integrity and trustworthiness of evaluation. It is an agreed requirement within the development agencies and in the evaluation community as a whole. In examining the issue of independence and good practice, reviewers are guided by the Evaluation Cooperation Group’s recommendations on good practices, the CDB’s Evaluation Policy and by the 2011 consultancy review of independence relative to the CDB’s evaluation and oversight division24. The appraisal is based on a comparison of the ECG’s recommendations on independence25 and the current OIE status.
OIE and Independence: Recommendations from the OECD Evaluation Cooperation Group (ECG)
The ECG’s considers the issue of independence according to three specific areas: organisational, or structural independence, behavioural, or functional independence and protection from outside interference, or operational independence.
Organizational independence, ensures that the evaluation unit and staff are protected against any influence or control by senior or line management, and have unrestricted access to all documents and information sources needed for conducting their evaluations. Also, that the scope of evaluations selected can cover all relevant aspects of their institution.
Behavioural independence, generally refers to the evaluation unit’s autonomy in selecting and conducting setting its work programme and in producing quality reports which can be delivered without management interference.
Protection from outside interference refers to the extent to which the evaluation function is autonomous in setting its priorities, and conducting its studies and processes and in reaching its judgments, and in managing its human and budget resources without management interference.
Conflict of interest safeguards refers to protection against staff conflict of interests be they current, immediate, future or prior professional and personal relationships and considerations or financial interests for which there should be provision in the institution’s human resource policies.
The OIE’s Independence in Practice
Organisational / structural independenceOn the whole, the Panel acknowledges and commends the efforts being made by the CDB to assure OIE’s organisational independence. The CDB’s Evaluation Policy provides for the OIE’s organisational independence from line management and the interview data suggests that there is also wide acceptance and acknowledgement of why the OIE should have such independent status. Table 1 below provides our overall assessment of this aspect of OIE’s independence when compared with ECG recommendations. 26
24 Osvaldo Feinstein & Patrick G. Grasso, Consultants, May 2011 Consultancy to Review the Independence of the Evaluation and Oversight Division of the Caribbean Development Bank25 ECG 2014 Evaluation Good Practice Standards, Template for Assessing the Independence of Evaluation Organizations, Annexe II.1 26 Based on ECG (2014) Template for Assessing the Independence of Evaluation Organizations, Evaluation Good
Practice Standards, Annexe II.1
37
Table 1: OIE organisational independence compared with ECG recommendations
Aspects Indicators CDB Evaluation Policy (EP) and Practice
The structure and role of evaluation unit
Whether the evaluation unit has a mandate statement that makes clear its scope of responsibility extends to all operations of the organization, and that its reporting line, staff, budget and functions are organizationally independent from the organization’s operational, policy, and strategy departments and related decision-making
Partially Complies The Policy is broad enough to cover the full range of MDB type of evaluations. However in practice this would not be possible without additional human and budget resources
The unit is accountable to, and reports evaluation results to, the head or deputy head of the organization or its governing Board
Whether there is a direct reporting relationship between the unit, and
a) the Management, and/or
b) Board or
c) relevant Board Committee, of the institution
Complies - OIE reports to the Board of Directors (BoD) through its Oversight Assurance Committee (OAC)
The unit is located organizationally outside the staff or line management function of the program, activity or entity being evaluated
The unit’s position in the organization relative to the program, activity or entity being evaluated
Complies - The OIE is located outside, and is therefore independent of CDB line management
The unit reports regularly to the larger organization’s audit committee or other oversight body
Reporting relationship and frequency of reporting to the oversight body
Complies - The OIE reports x 5 per year to the OAC . Board approval for an additional executive meeting between the Head of the OIE and the OAC at least once per year was given in October 2015
The unit is sufficiently removed from political pressures to be able to report findings without fear of repercussions
Extent to which the evaluation unit and its staff are not accountable to political authorities, and are insulated from participation in political activities
Complies
Unit staffers are protected by a personnel system in which compensation, training, tenure and advancement are based on merit
Extent to which a merit system covering compensation, training, tenure and advancement is in place and enforced
Partially Complies - with CDB human resource policy. However the skill needs of OIE staff ought to be regularly reviewed in light of its move towards higher-level evaluations. Appraisal of skill needs and hiring of relevant staff should be completely under the authority of the Head of Evaluation. This is not sufficiently clear in the Policy or other documents we reviewed.
Unit has access to all needed information and information sources
Extent to which the evaluation unit has access to the organization’s
a) staff, records, and project sites;
b) co-financiers and other partners, clients; and
Complies –The available evidence suggests that there is no reason to doubt such access. But systematic and easily accessible documentation is lacking in the CDB; it is one of its weak points.. Delays in getting hold of the relevant documents can have consequences on the timeliness of
38
Aspects Indicators CDB Evaluation Policy (EP) and Practice
c) programs, activities, or entities it funds or sponsors
evaluation studies
However, independence should not mean isolation: There appears to be a detachment between the OIE and CDB that is of concern to the Panel; on the one hand, between the OIE and operations staff, and (2) on the other, in terms of the structural arrangements between the OIE and senior management.
3) In agreeing for the OIE to concentrate on strategic and thematic, in-depth evaluations, responsibility for project monitoring and evaluation were given over to operations. The division is clear and respected. However, it has its drawbacks. With the OIE no longer systematically involved at the front-end of project design, the monitoring data needs are likely to be poorly defined. Weak monitoring data will contribute to weaker evaluations. (More on this point under the heading self and independent evaluations.)
In the reviewers’ opinion, it is a common misunderstanding to assume that providing evaluator advice on monitoring and evaluation data will comprise evaluator independence. On the contrary, evaluation input into project design is essential to assure that the logic, indicators and data needs are addressed so that at some future point in time an evaluation of the achievements can be empirically grounded.
This is not to say that the OIE no longer has any influence at the front-end design stage; it has merely shifted the point of focus. The OIE is now systematically providing such input more generally to the corporate planning teams for the tools and systems they are developing to support the MfDR framework. The monitoring data for projects and their implementation should be improved once the Project Performance Evaluation System (PPES) and the Portfolio Performance Management System (PPMS) are updated and operational.
4) In the second place, the OIE has limited formal access to the Advisory Management Team (AMT) weekly meetings where the President and senior management gather to exchange up-to-date information on the dynamics of CDB policy and practice. The OIE is not regularly invited in any capacity to these meetings or given a copy of the agenda or minutes; the OIE is occasionally invited to attend in order to discuss an evaluation report or management feedback. For the OIE, this means that it is unlikely to pick up on the ‘when’ and ‘what’ of key decisional issues or provide input into the discussion based on evaluative information. Its observer status at Loans Committee meetings, or as a participant informer at the OAC and BoD meetings and discussions do not necessarily provide the same insight as to the dynamics of management actions and/or decisions. .
To respond to this situation, the President has agreed to meet regularly with the Head of the OIE in order to keep him up to date with CDB strategic thinking. This is a welcomed change.
OIE Independence and Behavioural Issues The Panel has concerns about some behavioural issues. For example, through both the interviews and documentary review, we learned of considerable delays in processing both the independent evaluation reports as well as OIE’s validation of the CDB’s self-evaluations. Delays are generally due to receiving feedback on the independent reports from first, the relevant operational department, then from the AMT, and then on providing the OIE with a management response that is initially drafted by operations staff before being reviewed by the AMT. (OIE reports cannot be submitted to the OAC without the relevant management response). This two-layer process for preparing submissions to the Board is inefficient and could potentially be a
39
threat to evaluation’s independence in the future by delaying OIE’s timely reporting to the OAC.
OIE validations of the CDB self-evaluations are also submitted to the OAC, but it is in both sides’ interest to clear up any misunderstandings beforehand. Despite attempts to improve the timeframe for completing these validations, delays are more the norm than the exception. Table 2 below summarises our assessment of the behavioural aspects of independence.
Table 2: OIE and Behavioural Independence
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Ability and willingness to issue strong, high quality, and uncompromising reports
Extent to which the evaluation unit:
a) has issued high quality reports that invite public scrutiny (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk) of the lessons from the organization’s programs and activities;
b) proposes standards for performance that are in advance of those in current use by the organization; and
c) critiques the outcomes of the organization’s programs, activities and entities
Partially complies – paucity of data and documentation sometimes hinder the quality of reports. The OIE emphasizes the learning part of evaluation, and is cautious in its criticism recognising that management is going through a transitory stage and can still be overly defensive.
Ability to report candidly
Extent to which the organization’s mandate provides that the evaluation unit transmits its reports to the Management/Board after review and comment by relevant corporate units but without management-imposed restrictions on their scope and comments
Partially complies - as sometimes reporting to the Board is compromised by delays in the review/comment process between the OIE and the CDB. Any delay with the production of a Management Response will also mean that submitting a report to the Board in a timely manner is impaired since the two have to be submitted together.
Transparency in the reporting of evaluation findings
Extent to which the organization’s disclosure rules permit the evaluation unit to report significant findings to concerned stakeholders, both internal and external (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk).
Who determines evaluation unit’s disclosure policy and procedures: Board, relevant committee, or management.
Partially complies - The OIE’s conforms to the CDB’s disclosure policy. However, the dissemination of evaluation findings appears to be currently restricted to website publication and reports to the Board. A more targeted communication strategy to include other key stakeholders, e.g. project implementers in the BMCs should be developed and put in place.
Self-selection of items for work program
Procedures for selection of work program items are chosen, through systematic or purposive means, by the evaluation organization; consultation on work program with Management and Board
Complies - The OIE also ensures that its work program is drawn up after consultation with both CDB Management and Board to seek their input on relevant topics and themes.
Protection of administrative budget, and other budget
Line item of administrative budget for evaluation determined in accordance with a clear policy parameter, and
Partially complies - The administrative budget for supporting OIE work is protected. Access to additional sources of
40
Aspects Indicators CDB Evaluation Policy (EP) and Practice
sources, for evaluation function
preserved at an indicated level or proportion; access to additional sources of funding with only formal review of content of submissions
funding is possible if well argued and justified. But the approval process is complex and inefficient. (See Figure 1 below)
OIE and Protection from External influence or interference
Our overall assessment is provided in Table 3 below. The OIE’s independence in the design, conduct and content of its evaluations does not appear to be subjected to any external interference. But securing funding from any sources outside the OIE’s administrative budget, i.e. from the Social Development Fund, is an unduly complex and long process. As such we consider that the current funding process can affect the OIE’s choice with regard to the type of evaluations it can undertake. (See Figures 1 and 2 below)
Table 3: OIE and its Independence from External influence or interference
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Proper design and execution of an evaluation
Extent to which the evaluation unit is able to determine the design, scope, timing and conduct of evaluations without Management interference
Complies – however within limits of restricted human and financial resources available
Evaluation study funding
Extent to which the evaluation unit is unimpeded by restrictions on funds or other resources that would adversely affect its ability to carry out its responsibilities
Partially Complies - OIE must work within the limits of the agreed administrative budget wherever possible. If additional resources are needed for studies it must seek alternative funds elsewhere. The budget limitations can have an affect on the type of evaluations undertaken and therefore its independence in terms of choice.
Judgments made by the evaluators
Extent to which the evaluator’s judgment as to the appropriate content of a report is not subject to overruling or influence by an external authority
Complies – the evidence available suggests that the Board and Management accept the evaluators’ independent interpretation and conclusions Management responses are agreed to be the accepted place to raise any difference of opinion.
Evaluation unit head hiring/firing, term of office, performance review and compensation
Mandate or equivalent document specifies procedures for the
a) hiring, firing,
b) term of office,
c) performance review, and d). compensation of the evaluation unit head that ensure independence from operational management
Complies – the Head of OIE is appointed by the CDB President in agreement with the OAC for a 5 year period which is renewable x 1. The Head could be removed from Office by the President or the Board but only with the agreement of both parties.
However the Head reports to the President for all administrative and personnel matters. Even though this was not recommended in the Osvaldo Feinstein & Patrick G. Grasso report on Independence in 2011, the BoD accepted CDB’s reasons for keeping this arrangement. (e.g.most OAC members are non residents and cannot oversee day-to-day work)
. Extent to which the evaluation unit has control over:
a) staff hiring,
Partially complies - All OIE staff members are treated in the same way as other CDB staff. The Head has limited control over the hiring, firing or promotion of OIE staff.
41
Aspects Indicators CDB Evaluation Policy (EP) and Practice
b) promotion, pay increases, and
c) firing, within a merit system
Continued staff employment
Extent to which the evaluator’s continued employment is based only on reasons related to job performance, competency or the need for evaluator services
Partially complies - Whilst the EP is clear about procedures for hiring, firing and promotion, all of which must conform with CDB human resource policy, there is nothing mentioned about any difference of opinion between the CDB and the Head of the OIE with regard to continued staff employment subject to changes in the level of technical or interpersonal competencies needed to meet new demands.
Avoidance of Financial, Personal or Professional conflicts of interest
This particular aspect refers to the organisation’s Human Resources Policy; there must be provisions in place to protect against actual or potential conflict of interest. The Panel requested via the OIE, to have evidence from human resources on any such provisions but did not receive an answer. It must be assumes that this aspect of independence, past or present, does indeed form part of normal CDB Human Resource Policies
To conclude: The Panel is impressed with the measures CDB has taken to assure the organisational independence of the OIE. Its independent status is accepted and respected by senior and line management. The OIE’s budget is not independent from the overall CDB administrative budget; this affects its choice of evaluation types or approaches. Some of the behavioural issues affecting independence were also of concern, especially due to the delays in the exchange of documents, between the OIE and operations departments, which has a direct effect on timely reporting to the OAC. As for protection from outside interference, our concerns are largely to do with OIE’s independence over staffing issue; there are potential loopholes in current arrangements that could undermine OIE’s autonomy over its staff.
OIE’s Strategy, Work Practices and Work ProgrammeThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. Much of the period 2012 to 2015 has therefore been taken up with preparing OIE’s shift in focus from project-based evaluations to the high-level thematic and in-depth strategic studies. This has meant adopting a three-way approach; (1) for self-evaluations, reducing its time input to support the process and (2) for independent evaluations, taking stock of the gaps in coverage and expertise, and (3) networking to share experiences with centres of expertise and align OIE with international practices. In addition, amongst other duties, it has been supporting the development of MfDR tools and systems such as the Project Performance Assessment System by providing advice and input on programme logic and monitoring needs. The OIE plans to conduct 2-4 high-level studies per year from 2016. The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the
42
time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation pyramid27 are brought out in the remaining sections of this Review, not least given the limited resources available.
To conclude: The OIE has made a first step in proposing a strategy for establishing itself as an independent evaluation resource. But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities.
The Value / Usefulness of OIE’s Independent EvaluationsEvaluation is a powerful tool that can provide useful, evidence-based information to help inform and influence policy and practice. But useful evaluations depend not only on the evaluators’ skills, but on several other important factors as well; 1) on planning evaluations to be relevant to the priorities of the organisation’s work and for their results to be delivered in time to be useful; on the degree of 2) consultation and ultimately ownership by those who seek evaluative information; on the 3) tools used to support the evaluation process per se; and on the 4) credibility and quality of the evaluation products28.
1. Planning relevant and timely evaluationsThe OIE is now working on a 3 year rolling work plan that sets out the broad areas for enquiry. So far, there are no agreed criteria for making the selection of the specific topics for independent evaluation, although the priorities tend to reflect those of the CDB’s strategic plan. Nevertheless decision-making is rather arbitrary based on a process of dialogue between the OIE and the CDB and the OIE and the Board.
One of the OIE’s two objectives for 2015 therefore, was to define a work plan and agree priorities based on an approach that is “utilisation-focused”. This means that the studies are selected and planned to be relevant and useful to the organisation’s needs.
The OIE has achieved this objective with respect to its latest studies, which concerns the Social Development Fund (SDF) Multicycle 6&7 Evaluation, the Haiti Country Strategy evaluation and the evaluation of the CDB’s Policy Based Operations. Each of these three have been planned to deliver their results in time to provide the CDB Board of Directors with relevant information for negotiating the next round of funding. In spite of some delays due to a myriad of reasons, not least to the extra effort needed to secure essential data, the studies are expected to deliver on time.
The processes for agreeing OIE’s work plan and specific evaluations on the one hand, and, in securing alternative funding on the other, are shown in Figure 1 below. The Panel was surprised at learning how bureaucratic (the internal approval process), and inefficient (in view of the time it takes) the process seems to be. The concern here is that such a process could possibly pose a threat to assuring the Board of “timely studies.”
Figure 1: Selection of Evaluation Topics and Funding Source
27 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).28 These aspects reflect the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
Consultation with CDB Operations and OAC/Board for selection of
evaluation topic
3-year Work Programme and Budget (approved by Board)
Annual OIE report and work plan
submission to OAC
43
2. Consultation and ownership“The credibility of evaluations depends to some degree on whether and how the organization’s approach
to evaluation fosters partnership and helps build ownership and capacity in developing countries.”
(ECG good practices)
Internal review of Approach Paper
Specific Evaluation Study Design and Budgeting
OIE Draft Terms of Reference / Approach
Paper
Detailed ToR or Final Approach Paper if sufficiently detailed.
Finalise Approach Paper and submit to OAC/Board
Final Approach Paper
OAC ApprovalOAC minutes
Paper
Funding Track
Final Approach Paper/ToR
Board approval necessary If above USD
150,000
Board notification only if USD 150,000 or
below
Board Approval
Board Paper
OIE – Selection of consultants (if any) contracting
OIE Admin Budget or …
… SDF
Prepare TA Paper (content similar to Approach Paper but different
format.
TA Paper
Approval – Internal Loans Committee
OIE – Selection of consultants (if any)
contracting
44
The OIE engages with the OAC, CDB senior management and operations for agreeing its 3-year work plan and then for selecting the specific topics and themes. It also discusses the evaluation approach paper (design and implementation plan) with the CDB and OAC before completing the final version. However, preliminary and final drafts of the report are only submitted to the CDB line and senior managers for comment and factual errors. Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the results and their implications. Discussions with the OAC are more limited due to the overburdened agenda of OAC and Board meetings, as previously discussed.
In short, the OIE is to be commended for following the recommendations of professional good practices and standards on participative approaches; it has succeeded in having introduced a modus operandi that involves the key players in the selection of evaluation topics, the evaluation designs and their results. Figure 2 below provides an overview of the evaluation implementation and stakeholder engagement processes.
Figure 2: Evaluation Study Implementation and Feedback Loops
Arrangement AFully outsourced / external
consultants; oversight by OIE
Preparations:Detailed evaluation plan (incl tools,
timeline, etc.) and logistics
Production of Inception Report / Approach Paper
Arrangement BConducted by OIE
staff
Arrangement CJointly: external
consultants and OIE
Terms of Reference
Prepares Inception Report /
Approach Paper
Presentation/workshop:Interim findings and conclusions for immediate feedback and validation
Data Collection and Analysis
OIE
Summary and ppt for workshop presentation
and discussion with CDBSubmission of Draft Final
Report to OIE
Final OIE approved report to CDB Senior Management for Management Response
Board notification only if USD 150,000 or
below
Draft Final Report
Review loops – OIE and CDB (potentially also BMC)
Feedback to evaluation lead
Submission of Final Report to
OIE
45
3.4.
Notes to Figure 2
3. The OIE informed the Panel that this is an abbreviated version as there are e.g. additional steps (secondary processes) when evaluations are procured (tendering or single source), when there are additional review loops and updates to OAC etc.
4. OAC may also decide to return the report to OIE, the Panel were informed, or demand from Management specific actions based on the report.
This process is engaging and appears to have secured senior management and OAC interest and buy-in as witnessed in the latest studies. But there is the downside too! The process takes much time and, in our view, is partly unnecessary. The Panel appreciates that staff from operations as well as the AMT may both want to confer on an appropriate management response, but this should not be the case for reviewing an independent report for factual errors. The two-phase approach seems somewhat inefficient and unnecessary in our opinion.
Contact between the OIE, the CDB and/or the OAC during the actual study implementation is most often restricted to the occasional progress report, particularly when studies run behind time. There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. The OIE does, however, arrange discussions for reflecting on emerging findings, but we are not sure of how systematic this feedback loop is.
More generally speaking, outside of an evaluation study, the OIE has limited dealings with operations. The OIE has an advisory role in providing them with help, particularly with providing training, guidelines and tools to support self-evaluations. We are nevertheless concerned about the seeming distance between these two and how this has affected the perceived value of evaluation. (For further on this point, please see the section below on “Self- and Independent Evaluations”)
But the Panel also wishes to stress that this is not the case for newly appointed senior managers. A much more open attitude to evaluation and appreciation of its potential value was evident;
Prepare for disclosure and dissemination
Final Report
Final Report and Management Response submitted to
OAC/BoardFinal Report and
Mgt. Resp.
Management Response
OIE ApprovalFinal Report and Management Response considered by CDB
AMT
OAC/Board endorsed
46
they expressed interest in drawing out important lessons on what works, how, for whom, and under what conditions. In one case, interest was followed up in practice; the OIE was recently invited by a senior manager to share evaluative knowledge and experience with his staff regarding policy based operations.
Certainly, we can say that overall, the key stakeholders within the CDB are adequately integrated into the evaluation process as to foster their buy-in and ownership. But more generally, we feel that the utility of independent evaluations can be improved by fostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that evaluation can offer can add value to understanding the strengths and weaknesses of such strategies. This however cannot be done overnight and takes a long time.
3. Tools to support the evaluation processSo far, during this transitional phase, the OIE has mainly focussed on improving the tools to support the operations areas’ self-evaluations. This has left the OIE with little time to produce the checklists or tools to support its own studies. There are plans to develop an OIE Manual to guide and support the independent evaluation process. Such plans should be encouraged, as these documents will form a very important part of training, particularly for newcomers to the OIE team.
In the meantime, the OIE and operations staff refers to the Performance Assessment System (PAS) Manuals for evaluation activities. The manuals are based on DAC criteria and ECG principles. Much emphasis is given to the rating system and how and what should be rated. However we find them lengthy, unwieldy and overcomplicated. Moreover, such manuals should be used for reference, but cannot and should not replace first-hand training in how to plan, conduct and manage the evaluation process.
Quality Assessment (QA) and Quality at Entry (QaE)
There was a transition period between 2012 and 2014 to establish the OIE. Work on the PAS, QaE, PCRs, ARPP, which had started earlier, was therefore completed after OIE came into existence, but it effectively had no formal ‘home’ in operations. The Panel was told that there had been some discussions about creating a Quality Assurance unit within CDB (OPS) but the current status is unclear.
The QaE Guidance Questionnaire was developed before and completed by the OIE. It was used to assess the documents that came across to the OIE for comments at the Review Stage. The results were then sent to the Portfolio Manager/Project Coordinator indicating any gaps/issues that needed to be addressed or clarified. QaE Guidance Questionnaires were developed for all the Bank’s lending products, CSP and to assess the quality of supervision.
After the QaE was launched bank wide, several operations officers saw the merit in using the QaE Guidance Questionnaire in the field and adopted it as a tool for their use during the appraisal mission in order to cross check and test their data collection and analysis.
OIE’s use of the QaE was discontinued in 2014 due to limited resources and a stronger focus on evaluations. It still sometimes comments on specific appraisals, but very selectively.
Both QaE and QaS (quality at supervision) are also addressed in the PAS Manuals. In addition the QaE and PAS have been incorporated in Volume 2 of the Operations Manual OPPM.
The Review Panel assessed the QaE forms. They are relatively standard, adapted to the specificities of the CDB. They contribute to judging a project’s expected quality in a relatively objective way. As such, they are are helpful, as a benchmark, in the ex-post assessment of projects.
The Panel considers that the lack of an established Quality Unit in the CDB (and independent from OIE) is a weakness that should be addressed in the near future.
47
4. Credibility and Quality of Evaluation ProductsAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An independent evaluation is processed as follows; the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
Since 2012, the OIE has produced a range of studies and approach papers. This review is based on those listed below as provided by the OIE, and cover the period from May 2012 to December 2015. It includes 3 evaluations (in blue), 4 Assessment studies (in brown) 14 validations of self-evaluations (in green) and 3 Approach Papers (in purple) for upcoming evaluations. These are listed below in Table 4.
Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
Board Meeting
Date Type / Topic
251 May 2012 Ex-Post Evaluation Report on Road Improvement and Maintenance Project, Nevis -St. Kitts and Nevis.
Validation of Project Completion Report on Sites and Services – Grenada. Assessment of Effectiveness of Implementation of Poverty Reduction
Strategy 2004-09.253 Oct. 2012 Assessment of Extent and Effectiveness of Mainstreaming Environment,
Climate Change, Disaster Management at CDB.254 Dec. 2012 Assessment of the Implementation Effectiveness of the Gender Equality
Policy and Operational Strategy of the Caribbean Development Bank. Validation of Project Completion Report on Enhancement of Technical and
Vocational Education and Training – Belize. Validation of Project Completion Report on Fourth Road (Northern Coastal
Highway Improvement Section 1 of Segment II) Project – Jamaica. Assessment of the Effectiveness of the Policy-based Lending Instrument.
256 May 2013 Validation of Project Completion Report on Expansion of Grantley Adams International Airport – Barbados.
Validation of Project Completion Report on Fifth Water Supply Project – Saint Lucia.
261 May 2014 Validation of Project Completion Report on Immediate Response Loan, Tropical Storm Gustav, Jamaica.
Validation of Project Completion Report on Social Investment Fund, Jamaica.
Validation of Project Completion Report on Disaster Mitigation and Restoration – Rockfall and Landslip, Grenada.
263 Oct. 2014 Validation of Project Completion Report on Basic Education Project – Antigua and Barbuda
263 Oct. 2014 Approach Paper for SDF 6 & 7 Multicycle Evaluation
264 Dec. 2014 Validation of Project Completion Report on Policy-Based Loan – Anguilla
48
Validation of Project Completion Report on Immediate Response Loan - Tropical Storm Arthur – Belize.
Evaluation of Technical Assistance Interventions of the Caribbean Development Bank Related To Tax Administration and Tax Reform in The Borrowing Member Countries 2005-2012.
265 March
2015
Approach Paper for the Evaluation of Policy Based Operations
266 May 2015 Validation of Project Completion Report on Upgrading of Ecotourism Sites – Dominica
The Evaluation of the Caribbean Development Bank’s Intervention in Technical and Vocational Education and Training (1990-2012)
267 July 2015 Validation of Project Completion Report on The Belize Social Investment Fund I Project − Belize
268 Oct.2015 Approach Paper Country Strategy and Programme Evaluation, Haiti
The review and analysis of these documents is based on the UNEG Quality Checklist for Evaluation Reports (http://www.uneval.org/document/detail/607) as well as on ECG guidance (Big Book on Good Practice Standards).
Approach Papers
Three Approach Papers (APs) were made available to the panel (see Table [ref] above). An AP describes the rationale for the evaluation, the background to the topic evaluated, the evaluation framework (criteria and questions) and approach. It also describes the team and provides an initial planning. Being the first main deliverable of OIE’s evaluation process, APs are the starting point and therefore a major determining element in the roll-out of each evaluation. Therefore APs “have to get it right”.
The APs examined are clearly written, well-structured and of reasonable length.29 We were surprised to find, however, that they do not make explicit the objectives of the evaluated intervention(s), e.g., through a clear objective tree, or through an explicit theory of change, intervention logic or logframe. Whilst one of the APs contains, in an appendix, a results framework for the evaluation, the results framework for the intervention (PBO) itself is lacking.
Inception reports
Only one Inception Report was given to the Panel for review (SDF 6&7). This gives an in-depth description of the evaluated programme and provides a clear Theory of Change. It is good practice that this is established after a pilot field mission, which helps to amend the initial AP on the basis of field observations and sharpen the evaluation questions if needed.
However, it is still considered to be good practice to have the Theory of Change elaborated in the initial design documents . This would facilitate OIE evaluations after project completion. Establishing the Theory of Change of any intervention would be included in the QaE form more explicitly, to be developed between the Quality unit referred to above, and OIE.
Evaluations and Assessments
Three evaluations and four assessment reports completed during the review period were considered. Assessments are similar to evaluations but have a narrower scope; they focus on a limited set of aspects or judgment criteria, mainly effectiveness, i.e. achievement of objectives.
29 Opportunities remain of course to be more concise and to move parts to appendices, e.g., detailed descriptions of the evaluation team or part of the description of the evaluated intervention.
49
Evaluations generally base their judgment on the internationally recognised DAC criteria as well as aspects of the CDB and BMC’s management of the intervention.
In general, these reports are of reasonable quality. In the main, they explain the evaluated object30 and provide evaluation objectives. The findings are organised around the evaluation criteria or questions detailed in the scope and objectives section of the report. They are based on evidence derived from data collection and analysis methods as described in the methodology section. The reports tend to dwell on the limitations that the evaluation encountered, but without becoming defensive. In one case (PBL Assessment) the report starts with a summary of the reviews on the topic done by other MDBs. This was a pleasant surprise and indeed a good practice that could well be adopted in future evaluations too.
However, the reports also show several significant weaknesses:
- Reports do not always provide clear (reconstructed) intervention logics or theories of change for the intervention(s) evaluated.31 Evaluation criteria and questions are defined at a fairly general level. They are translated into more precise “research questions” (in an “Evaluation Design Matrix”, for each project for each criterion). However, it is unclear how these questions relate to the intervention logic (as this is not made explicit). This may be done in inception reports (of which, as noted above, only one was available for review), but should be done also in the final reports.
- The reports do not describe the link from the evaluation questions to the answers, how the evaluation judgments are made and how these ultimately transform into ratings for each criterion and each project. In other words, the explanation provided in the evaluation frameworks is inadequate. The “evaluation design matrix” currently used does not provide sufficient insight into how ultimately an intervention’s performance is judged.32 Links between findings, conclusions and recommendations could be improved by making this more explicit. In other words, reports should include the story on how the evaluand is credibly linked to any observed outcomes and impacts, and should be clear on how causal claims are made.
- With the exception of the PBL Assessment, reports are lengthy and detailed. One reason for this is an over-emphasis on ratings. Their detailed discussion, project by project, criterion by criterion, occupies a very prominent position in the evaluation reports’ main body of text. Although ratings are traditionally an important element in evaluations of MDBs, too strong an emphasis can be tedious and may distract the reader from the real lessons to be drawn. The detailed discussion of ratings, and their evidence base, would be better placed in an Appendix, with a brief summary in the main report. This would help give the lessons and recommendations a more prominent position than is now the case. This would also help make the evaluation reports not only shorter but also more interesting to read; this could help add value to evaluation’s image within the organisation.
- The reviewers feel that the OIE evaluations tend to over-emphasise objective-based evaluation33 and the DAC criteria to the exclusions of considering other evaluation
30 Sometimes in great length: for instance with the SDF 6&7 multicycle evaluation report it is only at page 30 that we find the beginning of the report on findings…31 Again with the SDF 6&7 evaluation, it is said to be guided by a “Logic Model” which is not explained.32 Marlène: I moreover have the idea that the methodology (often described as “visits”) is based on interviews and little hard evidence. Any view on this?.JM: My “interview-based evaluations”!!33 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.
50
approaches such as Developmental Evaluation (Patton, 201034); evaluation should be case specific and answer the actual information needs of managers and other decisions makers rather than always concentrating on final performance.
- Related to the previous point (and again with the exception of the PBL Assessment) executive summaries (approximately 8 pages) are too long. For the evaluation report to increase potential impact, they would need to be reduced to 2 to 3 pages and be more focused; again this could be done by dwelling less on the individual ratings of projects and more on key findings, lessons and conclusions. More generally, reports could be better adapted to the needs of the different audiences. Although not strictly limited to evaluations, The Health Evidence Network Reports35 are a model that could be adapted for evaluation reporting purposes; they are specifically geared towards addressing policy and decision-making.
- The “Recommendations to BMCs” are an interesting feature of the reports, (although we are unsure to what degree such recommendations could be effectively followed up by OIE or the Bank, but certainly could taken up with BMC Board members.
- Reports (e.g. the evaluation report on Technical Assistance) focus much on technical problems that were encountered during the evaluation. Although these are important issues, again to improve the report’s flow and “readability” this section would be better placed in the Appendix. What counts is the story of the intervention, not the story of the evaluation (see “Limitations” section in the TA report for instance)
OIE Validations of Project and Country Strategy Programme Completion Reports (referred to globally as PCRs hereafter)
As said above, the OIE has the mandate to validate the Project and Economic departments PCRs and CSPCRs. However, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
The validations tend to repeat the different items reported in the PCRs and then provide extensive comment on each. The PCVRs go into great depth and detail, which makes the documents rich and complete. This is their strength – but also their weakness. The depth and level of detail, as well as the repetitions from the original PCRs, makes PCVRs (overly) lengthy (20-40 pages) and difficult to read. The OIE reported spending approximately 27.2% of its time on validating PCRs in 2015 compared with 44.4% on its core work, i.e. doing or managing the higher level evaluations. That is more than half of its evaluation work is being spent on the validation process. Finally, the PCVRs now seem to be, to a great extent, a standalone output of OIE. It is not always clear to us how they are being used as the “building blocks” for the OIE’s independent evaluations. Making this clearer in the independent evaluations would help show the link and therefore the value of the time being spent on the self-evaluation validations.
To conclude, the review finds that the OIE has taken steps to improve the perceived utility of evaluation in several ways. In the first instance, by planning its work to provide relevant and timely evidence geared towards helping the Board with its oversight and decision making tasks. The topics are selected through dialogue between the OIE and key CDB stakeholders and reflect
34 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press35 See the reports available at the WHO’s Health Evidence Netowkr at http://www.euro.who.int/en/data-and-evidence/evidence-informed-policy-making/health-evidence-network-hen
51
priorities of the CDBs strategic plan. Secondly, by securing the interest and consequently the buy-in of the OAC and CDB senior management through engaging their input throughout the evaluation process. This is evidenced by the reported interest in the latest three studies, the Country strategy programme in Haiti, the evaluation of policy-based operations and the SDF 6& 7 multicycle assessment.
The OIE products are of an acceptable quality and could be even better if some of the shortcomings were addressed. However, the products themselves do not impair the utility of OIE’s work; this is undermined in several ways: (1) by the time delays in commenting on PCRs (OIE) and providing feedback to the independent evaluations (operations and management) (2) by the inefficient processes for agreeing topics and funding sources as well as providing OIE with management responses to its reports.
Putting Evaluation to Use: transparency, feedback and follow-upThere are several ways that evaluation can be, and is being used. As John Mayne has pointed out in his many publications on the issue,36 when we talk of evaluation use, we are mainly thinking about its Instrumental use—use made to directly improve programming and performance. But there is also conceptual use - use which often goes unnoticed or more precisely, unmeasured. This refers to the kind of use made to enhance knowledge about the type of intervention under study in a more general way. Or even Reflective use— this refers to using discussions or workshops to encourage and support reflection on the evaluation findings to see how they might contribute to future strategies.
In the case of the CDB there is some evidence to suggest that “use” is not only instrumental, but other types are also developing. For example, in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessons. (Reflective use)
Another important use, as recommended by the ECG, is that from time to time a synthesis of lessons is drawn from a number of evaluations and made available publically. In fact the Panel was impressed to hear that in the past, the evaluation unit had done this drawing on lessons from evaluations of the power sector. (Conceptual use) Although nothing has happened since, it is now on the “to do list” for 2016 (OIE’s 2016 Work Plan).
As for instrumental use, responsibility for using the knowledge generated through evaluation and for possibly drawing up an action plan of what should be done is up to CDB senior management and the relevant CDB department and division. Oversight on applying recommendations and picking up on the lessons drawn is the responsibility of the OAC.
Evidence on how evaluations have actually contributed to decisions or negotiations is lacking or confusing, Certainly the OIE is unaware of the extent to which its evaluations are put to use. On the one hand, the OAC minutes sometimes indicate that lessons learned are integrated into the next phase. On the other hand, the reviewers were told that often in the past, the evaluation results were “too old” to be of use as the lessons had already been drawn and used way before the report was completed. Similarly, people’s gaps in memory on how well the evaluative information from previous studies may have been used may also account for the scarcity of evidence.
In response, the Panel questioned CDB staff and the OIE about a particular study, the Technical and Vocational Education and Training Assessment. The feedback was somewhat contradictory. On the one hand, the study was criticised as “confirming” news rather than bringing “new news”. However, on the other, we learned that In October 2015, the Board of Directors approved a proposal for the revision of CDB’s Education and Training Policy and Strategy. Work on this has already begun and an external consultant has been engaged to lead the process.
36 See for example, his opening chapter to Enhancing Evaluation use: Insights from internal Evaluation Units, Läubli Loud, M. and Mayne, J. 2014, Sage Publications
52
Although it is one of the OIE’s tasks to set up a database on results and lessons learned from evaluations, so far this has not been a priority. There is also currently no systematic tracking of lessons or recommendations arising from the evaluations, or on any progress in their uptake. (The Panel has already referred above to OAC’s lack of oversight in the use of evaluation.)
The OIE’s role in supporting CDB’s organisational learning is clearly specified in the Evaluation Policy, with many good suggestions for knowledge sharing activities such as “brown-bag lunches, workshops, pamphlets and short issues papers” (p. 19). So far, however, the OIE’s lead role on the knowledge sharing side appears to be quite limited. It has provided advisory input in Loan Committee discussions, and organises workshops together with the relative operations department for discussing the implications of evaluation studies. Ultimately, of course, the uptake of evaluation results and knowledge is in the hands of management. But the evaluation unit has an important role to play in terms of knowledge broker and knowledge manager. Both have tended to be underplayed in OIE’s work plan so far.
Transparency: The Communication Strategy
In recent times and with the approval of its new Disclosure Policy, the CDB has started to post its independent evaluation reports on its website. (There is nothing on the self-evaluations). The website also presents a good overview of the role and function of the OIE and evaluation within the CDB. This is a step in the right direction for sharing information. However, in our view, the CDB’s communication strategy is the weakest part of the evaluation system to date.
The Panel has already commended the OIE in its efforts to engage the CDB and the OAC in evaluation work. But reporting and communicating the lessons seem to be entirely targeted at the Board and the CDB. Moreover, the 2015 budget provides only US$2’000 for communication – nothing of which is intended for outreach.
Reviewers feel that actively engaging with the more indirect stakeholders, for example project implementers in the BMCs, NGOs or project beneficiaries is relatively weak37. There appears to be little reflection on drawing out significant messages for the broader group of stakeholders, or on how then to transmit them to the “right” people in the “right” way (knowledge brokerage).
To conclude, evidence on the uptake of evaluation is either confusing or sparse. It is unfortunate that so far no systematic record keeping system has been put into place to track lessons learned or the uptake of recommendations (or actions agreed from management responses). The OIE plays a weak role in brokering the knowledge generated through evaluations to the benefit of external partners and in managing such knowledge. Although the Evaluation Policy specifies the need for “distilling evaluation findings and lessons learned in appropriate formats for targeted audiences both within and outside the CDB” (p.19) such a targeted communication strategy has yet to be developed and budgeted.
Strengthening Evaluation Capacities and Networking From the onset in 2012, the OIE has stressed the importance of developing and strengthening evaluation capacities within the OIE, the CDB and, subject to available resources, in borrowing member countries. Building evaluation capacity in BMCs and the CDB is one of the OIE’s mandated tasks. It has been a priority that figures on the work plan from the beginning (Work Programme and Budget 2012-2104) The idea of developing an internship programme for graduates from the Caribbean region was one idea that was advanced to help build local evaluation resources. However, the capacity-building has primarily been focused on OIE and CDB staff to date. One of the OIE’s two objectives for 2015 therefore was to take up the challenge and “strengthen evaluation capacities and networking” to include reaching out to the BMCs.
37 A broader communication strategy is one of the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
53
Developing OIE staff capacities
The change from project level to strategic and thematic evaluations does require different evaluative skills and competencies. The MDB Evaluation Pyramid presented below in Figure 3 shows the different types of evaluation and changing resource needs as one ascends the pyramid. Implicit here also is the change in the type of expertise and competencies needed as evaluation aspires to the higher levels.
Consequently for 2015, the OIE set itself the objective of networking and developing working partnerships with regional and international evaluation entities and academic institutions. The rationale was twofold: (1) secure further support and guidance as well as (2) increase its outreach and coverage through joint work and international exposure. Another implicit aim was to benefit from partners’ contacts in the BMCs wherever possible so as to improve data collection and quality.
Figure 3: The MDB Evaluation Pyramid38
The OIE has therefore linked up with Carleton University in Canada and the University of the West Indies, Barbados campus. The OIE was also approached by the Development Bank of South Africa to exchange experiences about setting up an evaluation entity in a “small” development bank. However, its attempt to become a member of the Evaluation Cooperation Group was not successful for reasons beyond its control.
The OIE is to be commended in addressing the issue of staff competencies and professional development more generally. New developments in evaluation as well as new developments in the scope of OIE’s work may necessitate new competencies. For this reason, organisations such as the International Developmental Evaluation Association have recommended that the
38 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
54
competencies of evaluators and evaluation managers should be periodically reviewed. Several publications now exist on competency requirements and suggestions for the periodic review of staff competencies.39
It is not within this remit to compare and contrast OIE’s competencies with those recommended by international and national agencies. However, what we can say is that the OIE demonstrates great forethought in taking this on board.
Capacity building within CDB
The OIE’s objective also consists of continuing to develop measures for improving the monitoring and self-evaluation side of CDB’s work. OIE’s strategy here is to use the windows of opportunity on offer through some of the training sessions that are being organised by CDB as part of its shift towards MfDR e.g. by Corporate Planning Services and Technical Assistance. For 2016 it is also planned to have the OIE present at the annual staff meeting and Learning Forum.
The OIE also organises some ad hoc training with operations, for example to help understand new tools e.g. for drawing out lessons from self-evaluation reports and, more generally, in helping staff appreciate how evaluation can add value to the organisation’s work. Measures include providing advisory services on demand, and providing training alongside the introduction of new or revised tools.
Capacity building in the BMCs
This is an ambitious task and would require additional investment; from the bi-annual work plans to be effective. A modest attempt has been made in 2015; from what we understand, the OIE has joined together with the Carleton University and the University of the West Indies, using their networks in some of the BMCs, to try to develop this aspect.
To conclude, we cannot comment on the quality or reaction to such training, but can commend the OIE for making capacity building one of its priority objectives. From both the Policy and the documents we reviewed, we note that capacity building was always seen to be an important aspect of OIE’s work, but hitherto has received little strategic focus. But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.
Adequacy of the OIE’s human and financial resources to support its work
OIE’s Human Resources;
The OIE is has a staff of 5; the head, 1 senior evaluation officer and two evaluation managers, plus one administrative assistant. Three of the five were recruited from within the CDB. The limited capacity means that it is not feasible to cover all the types of evaluation activities outlined in the Evaluation Policy. Yet there is some indication from the Board that OIE should embark on impact evaluations at some future stage. An increasing demand for evaluation and for impact evaluations in particular, would run the risk of overstretching the OIE’s capacity to deliver credible and useful evaluations. Moreover, there are many other designated OIE activities that should be recognised as valuable work; the validations, building CDB and BMC evaluation capacity, providing supervision, advice, knowledge management and brokerage as well as managing evaluation contracts, The time needs of dealing with all of these may be underestimated in OIE’s budgets; all are important for assuring best value from evaluation. The Panel is concerned that a demand for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level evaluations may undermine the importance and time needs of other 39 E.g. IDEAS, (2012) Competencies for Development Evaluation Evaluators, Managers and Commissioners, the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice (2010) and the Swiss Evaluation Society’s Evaluation Managers Competencies Framework (2014)
55
essential tasks.
Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses. (more on financial aspects and OIE practices is provided in Appendix III).
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.
Self- and independent evaluationSelf-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes. Both types of evaluation are important as they are at the very heart of the evaluation function; they are said to be the building blocks for the more strategic evaluations that the OIE is now undertaking.
The Evaluation Coordination Group recommends that the self-evaluations be carried out by the relevant operations department and in turn, reviewed and validated by the organisation’s independent evaluation office. The CDB’s Evaluation Policy therefore talks of “validating all self-evaluations” as being one of OIE’s essential oversight tasks.
56
Within CDB, the self-evaluations should provide management with performance assessments and thereby serve an accountability function to the CDB and Board. To support the process, the OIE provides operations with manuals and checklists for guidance. Once a self-evaluation report is to hand, it is given over to the OIE for the validation of its technical quality and credibility.40
However, in the CDB case, there are well-documented issues that have affected the quality and timeliness of the self-evaluations on the one hand, and therefore the quality of the foundation on which to build the independent evaluations. Paucity of documentation within CDB, paucity of data collected and available in the Borrowing Member Countries (BMCs), time delays in producing completion reports and in turn, having them validated by the OIE - all such issues were systematically raised during interviews and in some of the independent evaluation reports. There appears to be little incentive to complete self-evaluations in a timelier manner.
Generally speaking, many of the monitoring data problems appear to be due to a lack of management oversight. For example, with the introduction of results-based management, the logic frame and monitoring and data needs are systematically being built into intervention design. However, the BMCs are not delivering the data as contractually agreed at the outset. Incentives to support any significant change towards building a results-based culture seem to be weak and sanctions seem to be rarely enforced when the supply of data is lacking or lengthy delays to the projects occur. Although we can appreciate the complexities of trying to enforce monitoring compliance, this means that often, project deadlines have had to be extended, data gaps are not being satisfactorily dealt with and in turn, there has been a void in the quality and quantity of available evidence for the CDB’s self-assessment of project performance. For some time, this lack of oversight has been tolerated. Part of the problem is the low priority accorded to completing the self-evaluation reports by operations, coupled with the absence of any focal point within senior management to drive the process and deal with the problems.
No record is kept of how the self-evaluation results are actually used. They do not appear on the CDB website, but we were told that the findings are integrated into the following project designs. Hence we are somewhat unclear as to the utility of these reports at present. The situation is exacerbated by a rather confused image of evaluation: some operations staff consider OIE’s input (through validations or independent evaluations) to be sometimes over-critical, regulatory and adding little value; it is a threat rather than an opportunity for learning. Yet at the same time, evaluation is recognized as an integral part of result-based management.
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.
Some managers however seem to have started making positive changes. For example a revised and simplified template for producing project completion reports is being considered, and mid-term project reviews are expected to be more stringent in looking at monitoring plans and practices and tying disbursements to performance. In some cases we also learned of incentives being introduced to encourage project managers to complete their reports in a timelier manner. But much remains to be done and, since the OIE is no longer responsible for monitoring and project evaluations, there is a void that needs to be filled. It is up to line managers to drive this work forward.
To conclude, it is fair to say that in view of a number of “frustrations” between the OIE and operations, which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring data, the added value that evaluation might offer to the operations area is ill recognized. Moreover, the link between self-evaluation as the
40 According to the Evaluation Policy, OIE should validate all PCRs and CCRs but due to the backlog of reports and the delay in completing them (sometimes years later) since October 2015, the OIE has secured OAC agreement to validate a maximum of 6 per year, which are selected in consultation with the OAC.
57
building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
58
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”41
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
41 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
59
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”42 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 43 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
42 CDB (2011) Evaluation Policy (p.2)43
60
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
61
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
62
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
63
mendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
64
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
65
APPENDICES
Appendix I - The External Review Mandate – Terms of Reference and Approach Paper
Appendix II -Review Approach, Data collection and Analysis, and Limitations
Appendix III – Overview of OIE Evaluation Practice
Appendix IV – OIE Practice compared with ECG Good Practice Standards on Independence
Appendix V - List of Persons Interviewed
Appendix VI - List of Documents Reviewed
Appendix VII- List of Topics used to guide interviews with members of CDB Board of Directors
Appendix VIII- List of Topics used to guide interviews with CDB staff
66
Appendix III – Overview of OIE Evaluation Practice (prepared by the OIE in response to Reviewer’s request)
Caribbean Development Bank, Office of Independent Evaluation - OIE
Category Response
Percentage of projects subject to project (self-) evaluation
100% - Project Completion Reports (PCR)
Percentage of projects subject to validation by OIE
Approximately 40-50%
About 15 projects exit portfolio annually. Evaluation Policy calls for all PCR to be validated. However, OIE resources insufficient. Validation process reviewed in 2014. Now OAC (Board committee) selects a sample of 6-8 PCR for validation each year.
Percentage/number of projects subject to in-depth review by OIE
None – unless specifically requested by OAC
Due to limited resources, focus of OIE evaluation work programme is on PCR validations and high-level evaluations – including country strategy and programme evaluations (CSPE).
Number of high-level evaluations conducted by OIE (e.g. sector, thematic, geographic)
1-2 per year since 2011
Plan is 2-4 per year from 2016. This would include CSPE (1st planned for Q1 2016: Haiti)
Number of project impact evaluations conducted by OIE
None
OIE includes “impact questions” in high-level evaluations.
Number of project impact evaluations conducted by Bank staff or other non-OIE staff
OIE is not aware of any impact evaluation conducted by the Bank.
However, OIE provides technical support to the Basic Needs Trust Fund (BNTF) in its design of an M&E framework that entails impact evaluations.
Budget In USD mn: 0.78 in 2015; 0.82 in 2016. This is equivalent to about 2.5% of total CDB Administrative Budget.
75% of the budget is for Staff salaries (4 Professionals, 1 Support staff), leaving around USD 190,000 (in 2015) for other expenses, including consultants e.g. for external evaluations. Additional funding is accessed via the Special Development Fund (SDF). This varies according to type and scope of the evaluation, e.g. the ongoing SDF 6/7 Evaluation is SDF funded at USD 255,000.
Budget determined by Board, not separate from administrative budget.
SDF funding for evaluations is considered separately and subject to Bank internal approval process. SDF funding cannot be used to cover OIE expenses such as staff time or travel. Country eligibility for SDF funding is also a consideration. OIE expressed concerns about this funding track in respect to predictability, independence and eligibility limitations.
Head of OIE reports to Board, with administrative link to the President
Terms of appointment for Head
67
5 year term, renewable once. Appointed by the President with the agreement of the Board.
Right of Return for Head Not eligible for other staff positions.
Consultants as proportions of OIE budget
2015: 19% (USD 145,000)
Plus SDF funding. SDF funded evaluations are outsourced.
Last external evaluation (or peer review) of OIE
No external evaluation, though a review of the function was done in 2011, leading to the Evaluation Policy.
OIE External Review completed in April, 2016
Departments or special programmes supporting impact evaluation
None
68
Appendix V – List of Persons Interviewed
Name Function relative to OIE Type interview
Mrs. Colleen Wainwright Member CDB Board of Directors (UK)
Face to face
Mrs. Cherianne Clarke Alternate MemberCDB Board of Directors (UK)
Face to face
Mrs. Jean McCardle Member CDB Board of Directors (Canada)
Face to face
Dr. Louis Woodroofe MemberCDB Board of Directors (Barbados)
Mr. A: de Brigard Former Member CDB Board of Directors
Skype interview
Mr. H. Illi Fromer Member CDB Board ofDirectors
Telephone interview
Mrs. Claudia Reyes Nieto Member CDB Board of Directors
Telephone interview
Mr. Bu Yu alternate DirectorCDB Board of Directors
Face to face
Mr. Michael Schroll(Barbados)
Head OIE
series of interviews viaSkype and face-to-face
Mr. Mark Clayton OIE Senior Evaluation Officer Focus GroupMrs. Egene Baccus Latchman OIE Evaluation OfficerMr. Everton Clinton OIE Evaluation OfficerMrs. Valerie Pilgrim OIE Evaluation Officer
Dr. Justin Ram CDB Director Economics Department
Face to face
Mr. Ian Durant CDB Deputy Director Economics Dept Face to faceDr. Wm Warren Smith CDB President
Joint interviewFace to face
Mrs. Yvette Lemonias-Seale CDB Vice President Corporate Services & Bank Secretariat
Mr. Denis Bergevin CDB Deputy DirectorInternal Audit
Face to face
Mr. Edward Greene CDB Division Chief, Technical Cooperation Division
Face to face
Mrs. Monica La Bennett CDB Deputy Director Corporate Planning Face to faceMrs. Patricia McKenzie CDB Vice President Operations Face to faceMs. Deidre Clarendon CDB Division Chief
Social Sector DivisionFace to face
Mrs. Cheryl Dixon CDB Co-ordinator, Environmental Sustainability Unit
Focus group
Mrs. Denise Noel- Debique CDB Gender Equality Advisor Mrs. Tessa Williams-Robertson CDB Head Renewable EnergyMrs. Klao Bell-Lewis CDB Head Corporate Communications Face to faceMr. Daniel Best CDB Director
Projects DepartmentFace to face
Mr. Carlyle Assue CDB Director Face to face
69
Finance Department
70
Appendix VI - Interview Guide: Members of CDB Board of Directors
Below are a list of themes that I should like to raise with you based on your experience and knowledge of the CDB’s independent evaluation function (Office of Independent
Evaluation).
In each case, I should be grateful if you could illustrate your responses with examples or help this Review by, wherever possible, sending me (or telling me where I can find) any
documents that could support your responses.
This guide is being sent to you in advance to help prepare our meeting. However, our interview will be conducted more in the style of a conversation. The following sub-questions will be used to GUIDE the interview. Please feel encouraged to raise any
additional issues that you feel we should take into account
On the governance and Independence of CDB’s evaluation functionWhat mechanisms are there in place to support its independence?
How satisfactory are the current arrangements in your opinion?
How is the balance between independence and the need for interaction with line management dealt with by the system? For example, what mechanisms exist to ensure that the OIE is kept up to date with decisions, policy / programme changes, other contextual changes etc that could have an affect on OIE evaluation studies / evaluation planning?
On the OIE’s Evaluation PolicyThe CDB’s Evaluation Policy was established in 2011. To what degree do you feel it is adequate? Still relevant?
What suggestions do you have for any improvements?
In your opinion, how adequate is the current quality assurance system for over viewing the evaluation function?
On the quality and credibility of evaluation studiesTo what degree do you believe the reports are fair and impartial?
Do you consider them to be of good quality? Are they credible?
Are you adequately consulted/involved on evaluations of interest to you?
On the relevance and usefulness of evaluations How well does the OIE engage with you / your committee during the preparation, implementation and reporting of an evaluation study to assure that it will be useful to the CDB?
How are the priorities set for the independent evaluations? What criteria are used? Are you satisfied with the current procedure?
71
When OIE evaluation studies are outsourced to external consultants, what criteria are used to make this decision?
How are the priorities for the OIE’s 3.year rolling work plan agreed? In your opinion, is the current plan adequate in terms of coverage and diversity?
In your opinion, do the evaluations address important and pressing programs and issues?
To what extent do you feel that the OIE’s evaluations integrate the cross-cutting theme such as gender, energy efficiency/renewable energy, climate change? What improvements might be made and how?
On the dissemination and uptake of evaluation findings and recommendationsTo what extent do you feel that evaluation findings are communicated to the CDB and its stakeholders in a
a) useful, b) constructive andc) timely manner?
Are evaluation recommendations useful? Realistic?
What mechanisms are in place to assure that evaluation results are taken into account in decision making and planning? What improvements do you feel could be made?
How have you used the findings from any evaluations? Examples?
To what degree do you feel that evaluation contributes to institutional learning? And what about to institutional accountability? Any examples?
What mechanisms are in place to ensure that knowledge from evaluation is accessible toCDB staff and other relevant stakeholders? Are the current arrangements satisfactory?
How satisfied are you with current arrangements? What expectations do you have for the future?
On resourcesHow is the OIE resourced financially and is this satisfactory?
What about the OIE staff, are all the important areas of expertise represented in the team?
On this Review of the Office of Independent EvaluationWhat are your expectations? What are you particularly hoping to learn from it?
Thank you very much for your cooperation and input
72
Appendix VII : Interview Pro-Forma – CDB Staff membersThis presents a list of the topics raised during interviews. It was used to guide the open-ended
discussion – this means that the sequence and exact wording of the questions may not necessarily have followed in this order or been asked in exactly this way.
Changeover to an Independent Evaluation Office? Expectations? Advantages and disadvantages??
Satisfaction with working relations between operations and the OIE from your perspective?
Process of dealing with the PCRs and CCRs? Advantages and limitations?
Quality and credibility of the validation process?
How are the self-evaluation reports used?
Credibility and Quality of OIE’s evaluation reports
Communication of self and OIE independent evaluations? To whom, in what way? Possible improvements?
actual or potential conflict of interestThe PanelIt must be s
The PanelisThe; this affects also of y, Work PracticesThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. therefore The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation
73
pyramid44 are brought out in the remaining sections of this Review, not least given the limited resources available.
But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities. sTheThes before completing the final version. However, p are only submitted to the CDB line and senior managers.Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the following the recommendations of professional good practices and standards on participative approaches; it has succeeded in , The Panelsstaff fromas well as There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. OIE doesarrange .sthe Panel also wishes tonewly appointed rsA was evidentthey expressed interest in In one case, interest was followed up in practice; can be improvedfostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that can offer can add value to understanding the strengths and weaknesses of such strategies. Tduring this transitional phase, Manual to guide and support the independent evaluation process.and operations staff sevaluation activities. ,oOIE’sjudginga . As such, theyare thatAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An is processed as follows;the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
SThisrItand Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
The rmade
- is still considered to be good practice to have the elaborated in the initial design documents the45 such as Developmental Evaluation (Patton, 201046)
-)
said abovePCHowever, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s
44 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).45 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.46 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press
74
agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessonsthe Panelas done this ing on lessonsAlthough nothing has happened since, it is , sometimes indicate (Panel has already referred above to ’s lack of oversight in the use of evaluation.)
sThe Panelsevaluation work Moreover, the 2015 budget provides only US$2’000 for
communication – nothing of which is intended for outreach.Reviewerseither confusing or and
budgetedConsequently for 2015, theFigure 3: The MDB Evaluation Pyramid47
47 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
75
ys to be effective. A modest attempt has been made in 2015; OIE hased But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.man and financial resources to support its work
OIE’s Human Resources;
5eThree of the five were recruited from within the CDB. edfrom the Board that OIE
should embark on ee and for impact evaluations in particular,OIE’s ee Moreover, there
are many other designated OIE activities that should be recognised as valuable work; the
validations, building CDB and BMC evaluation capacity, providing supervision, advice,
knowledge management and brokerage as well as managing evaluation contracts, The
time needs of dealing with all of these may be underestimated in OIE’s budgets; all are
important for assuring best value from evaluation. The Panel is concerned that a demand
for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level
evaluations may undermine the importance and time needs of other essential
tasks.Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses.
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies
76
does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.Self-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes.types of evaluation y There appears to be little incentive to complete self-evaluations in a timelier manner.
.
; it is a threat rather than an opportunity for learning. Yis recognized as
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.a number of , which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring dataadded value that evaluation might offer to the operations area is ill recognized Moreover, the link between self-evaluation as the building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
77
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”48
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
48 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
78
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”49 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 50 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
49 CDB (2011) Evaluation Policy (p.2)50
79
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
80
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
81
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
82
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
Independence of the Office of Independent Evaluation (OIEIndependence is absolutely central to the integrity and trustworthiness of evaluation. It is an agreed requirement within the development agencies and in the evaluation community as a whole. In examining the issue of independence and good practice, reviewers are guided by the Evaluation Cooperation Group’s recommendations on good practices, the CDB’s Evaluation Policy and by the 2011 consultancy review of independence relative to the CDB’s evaluation and oversight division51. The appraisal is based on a comparison of the ECG’s recommendations on independence52 and the current OIE status.
OIE and Independence: Recommendations from the OECD Evaluation Cooperation Group (ECG)
The ECG’s considers the issue of independence according to three specific areas: organisational, or structural independence, behavioural, or functional independence and protection from outside interference, or operational independence.
Organizational independence, ensures that the evaluation unit and staff are protected against any influence or control by senior or line management, and have unrestricted access to all documents and information sources needed for conducting their evaluations. Also, that the scope of evaluations selected can cover all relevant aspects of their institution.
Behavioural independence, generally refers to the evaluation unit’s autonomy in selecting and conducting setting its work programme and in producing quality reports which can be delivered without management interference.
Protection from outside interference refers to the extent to which the evaluation function is autonomous in setting its priorities, and conducting its studies and processes and in reaching its judgments, and in managing its human and budget resources without management interference.
Conflict of interest safeguards refers to protection against staff conflict of interests be they current, immediate, future or prior professional and personal relationships and considerations or financial interests for which there should be provision in the institution’s human resource policies.
The OIE’s Independence in Practice
Organisational / structural independenceOn the whole, the Panel acknowledges and commends the efforts being made by the CDB to assure OIE’s organisational independence. The CDB’s Evaluation Policy provides for the OIE’s organisational independence from line management and the interview data suggests that there is also wide acceptance and acknowledgement of why the OIE should have such independent status. Table 1 below provides our overall assessment of this aspect of OIE’s independence when compared with ECG recommendations. 53
51 Osvaldo Feinstein & Patrick G. Grasso, Consultants, May 2011 Consultancy to Review the Independence of the Evaluation and Oversight Division of the Caribbean Development Bank52 ECG 2014 Evaluation Good Practice Standards, Template for Assessing the Independence of Evaluation Organizations, Annexe II.1 53 Based on ECG (2014) Template for Assessing the Independence of Evaluation Organizations, Evaluation Good
Practice Standards, Annexe II.1
83
Table 1: OIE organisational independence compared with ECG recommendations
Aspects Indicators CDB Evaluation Policy (EP) and Practice
The structure and role of evaluation unit
Whether the evaluation unit has a mandate statement that makes clear its scope of responsibility extends to all operations of the organization, and that its reporting line, staff, budget and functions are organizationally independent from the organization’s operational, policy, and strategy departments and related decision-making
Partially Complies The Policy is broad enough to cover the full range of MDB type of evaluations. However in practice this would not be possible without additional human and budget resources
The unit is accountable to, and reports evaluation results to, the head or deputy head of the organization or its governing Board
Whether there is a direct reporting relationship between the unit, and
a) the Management, and/or
b) Board or
c) relevant Board Committee, of the institution
Complies - OIE reports to the Board of Directors (BoD) through its Oversight Assurance Committee (OAC)
The unit is located organizationally outside the staff or line management function of the program, activity or entity being evaluated
The unit’s position in the organization relative to the program, activity or entity being evaluated
Complies - The OIE is located outside, and is therefore independent of CDB line management
The unit reports regularly to the larger organization’s audit committee or other oversight body
Reporting relationship and frequency of reporting to the oversight body
Complies - The OIE reports x 5 per year to the OAC . Board approval for an additional executive meeting between the Head of the OIE and the OAC at least once per year was given in October 2015
The unit is sufficiently removed from political pressures to be able to report findings without fear of repercussions
Extent to which the evaluation unit and its staff are not accountable to political authorities, and are insulated from participation in political activities
Complies
Unit staffers are protected by a personnel system in which compensation, training, tenure and advancement are based on merit
Extent to which a merit system covering compensation, training, tenure and advancement is in place and enforced
Partially Complies - with CDB human resource policy. However the skill needs of OIE staff ought to be regularly reviewed in light of its move towards higher-level evaluations. Appraisal of skill needs and hiring of relevant staff should be completely under the authority of the Head of Evaluation. This is not sufficiently clear in the Policy or other documents we reviewed.
84
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Unit has access to all needed information and information sources
Extent to which the evaluation unit has access to the organization’s
a) staff, records, and project sites;
b) co-financiers and other partners, clients; and
c) programs, activities, or entities it funds or sponsors
Complies –The available evidence suggests that there is no reason to doubt such access. But systematic and easily accessible documentation is lacking in the CDB; it is one of its weak points.. Delays in getting hold of the relevant documents can have consequences on the timeliness of evaluation studies
However, independence should not mean isolation: There appears to be a detachment between the OIE and CDB that is of concern to the Panel; on the one hand, between the OIE and operations staff, and (2) on the other, in terms of the structural arrangements between the OIE and senior management.
5) In agreeing for the OIE to concentrate on strategic and thematic, in-depth evaluations, responsibility for project monitoring and evaluation were given over to operations. The division is clear and respected. However, it has its drawbacks. With the OIE no longer systematically involved at the front-end of project design, the monitoring data needs are likely to be poorly defined. Weak monitoring data will contribute to weaker evaluations. (More on this point under the heading self and independent evaluations.)
In the reviewers’ opinion, it is a common misunderstanding to assume that providing evaluator advice on monitoring and evaluation data will comprise evaluator independence. On the contrary, evaluation input into project design is essential to assure that the logic, indicators and data needs are addressed so that at some future point in time an evaluation of the achievements can be empirically grounded.
This is not to say that the OIE no longer has any influence at the front-end design stage; it has merely shifted the point of focus. The OIE is now systematically providing such input more generally to the corporate planning teams for the tools and systems they are developing to support the MfDR framework. The monitoring data for projects and their implementation should be improved once the Project Performance Evaluation System (PPES) and the Portfolio Performance Management System (PPMS) are updated and operational.
6) In the second place, the OIE has limited formal access to the Advisory Management Team (AMT) weekly meetings where the President and senior management gather to exchange up-to-date information on the dynamics of CDB policy and practice. The OIE is not regularly invited in any capacity to these meetings or given a copy of the agenda or minutes; the OIE is occasionally invited to attend in order to discuss an evaluation report or management feedback. For the OIE, this means that it is unlikely to pick up on the ‘when’ and ‘what’ of key decisional issues or provide input into the discussion based on evaluative information. Its observer status at Loans Committee meetings, or as a participant informer at the OAC and BoD meetings and discussions do not necessarily provide the same insight as to the dynamics of management actions and/or decisions. .
To respond to this situation, the President has agreed to meet regularly with the Head of the OIE in order to keep him up to date with CDB strategic thinking. This is a welcomed change.
OIE Independence and Behavioural Issues The Panel has concerns about some behavioural issues. For example, through both the interviews and documentary review, we learned of considerable delays in processing both the
85
independent evaluation reports as well as OIE’s validation of the CDB’s self-evaluations. Delays are generally due to receiving feedback on the independent reports from first, the relevant operational department, then from the AMT, and then on providing the OIE with a management response that is initially drafted by operations staff before being reviewed by the AMT. (OIE reports cannot be submitted to the OAC without the relevant management response). This two-layer process for preparing submissions to the Board is inefficient and could potentially be a threat to evaluation’s independence in the future by delaying OIE’s timely reporting to the OAC.
OIE validations of the CDB self-evaluations are also submitted to the OAC, but it is in both sides’ interest to clear up any misunderstandings beforehand. Despite attempts to improve the timeframe for completing these validations, delays are more the norm than the exception. Table 2 below summarises our assessment of the behavioural aspects of independence.
Table 2: OIE and Behavioural Independence
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Ability and willingness to issue strong, high quality, and uncompromising reports
Extent to which the evaluation unit:
a) has issued high quality reports that invite public scrutiny (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk) of the lessons from the organization’s programs and activities;
b) proposes standards for performance that are in advance of those in current use by the organization; and
c) critiques the outcomes of the organization’s programs, activities and entities
Partially complies – paucity of data and documentation sometimes hinder the quality of reports. The OIE emphasizes the learning part of evaluation, and is cautious in its criticism recognising that management is going through a transitory stage and can still be overly defensive.
Ability to report candidly
Extent to which the organization’s mandate provides that the evaluation unit transmits its reports to the Management/Board after review and comment by relevant corporate units but without management-imposed restrictions on their scope and comments
Partially complies - as sometimes reporting to the Board is compromised by delays in the review/comment process between the OIE and the CDB. Any delay with the production of a Management Response will also mean that submitting a report to the Board in a timely manner is impaired since the two have to be submitted together.
Transparency in the reporting of evaluation findings
Extent to which the organization’s disclosure rules permit the evaluation unit to report significant findings to concerned stakeholders, both internal and external (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk).
Who determines evaluation unit’s disclosure policy and procedures: Board, relevant committee, or management.
Partially complies - The OIE’s conforms to the CDB’s disclosure policy. However, the dissemination of evaluation findings appears to be currently restricted to website publication and reports to the Board. A more targeted communication strategy to include other key stakeholders, e.g. project implementers in the BMCs should be developed and put in place.
Self-selection of items for work program
Procedures for selection of work program items are chosen, through systematic or purposive means, by the
Complies - The OIE also ensures that its work program is drawn up after consultation with both CDB Management
86
Aspects Indicators CDB Evaluation Policy (EP) and Practice
evaluation organization; consultation on work program with Management and Board
and Board to seek their input on relevant topics and themes.
Protection of administrative budget, and other budget sources, for evaluation function
Line item of administrative budget for evaluation determined in accordance with a clear policy parameter, and preserved at an indicated level or proportion; access to additional sources of funding with only formal review of content of submissions
Partially complies - The administrative budget for supporting OIE work is protected. Access to additional sources of funding is possible if well argued and justified. But the approval process is complex and inefficient. (See Figure 1 below)
OIE and Protection from External influence or interference
Our overall assessment is provided in Table 3 below. The OIE’s independence in the design, conduct and content of its evaluations does not appear to be subjected to any external interference. But securing funding from any sources outside the OIE’s administrative budget, i.e. from the Social Development Fund, is an unduly complex and long process. As such we consider that the current funding process can affect the OIE’s choice with regard to the type of evaluations it can undertake. (See Figures 1 and 2 below)
Table 3: OIE and its Independence from External influence or interference
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Proper design and execution of an evaluation
Extent to which the evaluation unit is able to determine the design, scope, timing and conduct of evaluations without Management interference
Complies – however within limits of restricted human and financial resources available
Evaluation study funding
Extent to which the evaluation unit is unimpeded by restrictions on funds or other resources that would adversely affect its ability to carry out its responsibilities
Partially Complies - OIE must work within the limits of the agreed administrative budget wherever possible. If additional resources are needed for studies it must seek alternative funds elsewhere. The budget limitations can have an affect on the type of evaluations undertaken and therefore its independence in terms of choice.
Judgments made by the evaluators
Extent to which the evaluator’s judgment as to the appropriate content of a report is not subject to overruling or influence by an external authority
Complies – the evidence available suggests that the Board and Management accept the evaluators’ independent interpretation and conclusions Management responses are agreed to be the accepted place to raise any difference of opinion.
Evaluation unit head hiring/firing, term of office, performance review and compensation
Mandate or equivalent document specifies procedures for the
a) hiring, firing,
b) term of office,
c) performance review, and d). compensation of the evaluation unit head that ensure independence from operational management
Complies – the Head of OIE is appointed by the CDB President in agreement with the OAC for a 5 year period which is renewable x 1. The Head could be removed from Office by the President or the Board but only with the agreement of both parties.
However the Head reports to the President for all administrative and personnel matters. Even though this was not recommended in the Osvaldo Feinstein & Patrick G. Grasso report on Independence in 2011, the BoD accepted CDB’s reasons for keeping this arrangement. (e.g.most OAC members are non residents and cannot
87
Aspects Indicators CDB Evaluation Policy (EP) and Practice
oversee day-to-day work)
. Extent to which the evaluation unit has control over:
a) staff hiring,
b) promotion, pay increases, and
c) firing, within a merit system
Partially complies - All OIE staff members are treated in the same way as other CDB staff. The Head has limited control over the hiring, firing or promotion of OIE staff.
Continued staff employment
Extent to which the evaluator’s continued employment is based only on reasons related to job performance, competency or the need for evaluator services
Partially complies - Whilst the EP is clear about procedures for hiring, firing and promotion, all of which must conform with CDB human resource policy, there is nothing mentioned about any difference of opinion between the CDB and the Head of the OIE with regard to continued staff employment subject to changes in the level of technical or interpersonal competencies needed to meet new demands.
Avoidance of Financial, Personal or Professional conflicts of interest
This particular aspect refers to the organisation’s Human Resources Policy; there must be provisions in place to protect against actual or potential conflict of interest. The Panel requested via the OIE, to have evidence from human resources on any such provisions but did not receive an answer. It must be assumes that this aspect of independence, past or present, does indeed form part of normal CDB Human Resource Policies
To conclude: The Panel is impressed with the measures CDB has taken to assure the organisational independence of the OIE. Its independent status is accepted and respected by senior and line management. The OIE’s budget is not independent from the overall CDB administrative budget; this affects its choice of evaluation types or approaches. Some of the behavioural issues affecting independence were also of concern, especially due to the delays in the exchange of documents, between the OIE and operations departments, which has a direct effect on timely reporting to the OAC. As for protection from outside interference, our concerns are largely to do with OIE’s independence over staffing issue; there are potential loopholes in current arrangements that could undermine OIE’s autonomy over its staff.
OIE’s Strategy, Work Practices and Work ProgrammeThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. Much of the period 2012 to 2015 has therefore been taken up with preparing OIE’s shift in focus from project-based evaluations to the high-level thematic and in-depth strategic studies. This has meant adopting a three-way approach; (1) for self-evaluations, reducing its time input to support the process and (2) for independent evaluations, taking stock of the gaps in coverage and expertise, and (3) networking to share experiences with centres of expertise and align OIE with international practices. In addition, amongst other duties, it has been supporting the development of MfDR tools and systems such as the Project Performance Assessment System by providing advice and input on programme logic and monitoring needs. The OIE plans to conduct 2-4 high-level studies per year from 2016. The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
88
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation pyramid54 are brought out in the remaining sections of this Review, not least given the limited resources available.
To conclude: The OIE has made a first step in proposing a strategy for establishing itself as an independent evaluation resource. But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities.
The Value / Usefulness of OIE’s Independent EvaluationsEvaluation is a powerful tool that can provide useful, evidence-based information to help inform and influence policy and practice. But useful evaluations depend not only on the evaluators’ skills, but on several other important factors as well; 1) on planning evaluations to be relevant to the priorities of the organisation’s work and for their results to be delivered in time to be useful; on the degree of 2) consultation and ultimately ownership by those who seek evaluative information; on the 3) tools used to support the evaluation process per se; and on the 4) credibility and quality of the evaluation products55.
1. Planning relevant and timely evaluationsThe OIE is now working on a 3 year rolling work plan that sets out the broad areas for enquiry. So far, there are no agreed criteria for making the selection of the specific topics for independent evaluation, although the priorities tend to reflect those of the CDB’s strategic plan. Nevertheless decision-making is rather arbitrary based on a process of dialogue between the OIE and the CDB and the OIE and the Board.
One of the OIE’s two objectives for 2015 therefore, was to define a work plan and agree priorities based on an approach that is “utilisation-focused”. This means that the studies are selected and planned to be relevant and useful to the organisation’s needs.
The OIE has achieved this objective with respect to its latest studies, which concerns the Social Development Fund (SDF) Multicycle 6&7 Evaluation, the Haiti Country Strategy evaluation and the evaluation of the CDB’s Policy Based Operations. Each of these three have been planned to deliver their results in time to provide the CDB Board of Directors with relevant information for negotiating the next round of funding. In spite of some delays due to a myriad of reasons, not least to the extra effort needed to secure essential data, the studies are expected to deliver on time.
The processes for agreeing OIE’s work plan and specific evaluations on the one hand, and, in securing alternative funding on the other, are shown in Figure 1 below. The Panel was surprised at learning how bureaucratic (the internal approval process), and inefficient (in view of the time it takes) the process seems to be. The concern here is that such a process could possibly pose a threat to assuring the Board of “timely studies.”
54 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).55 These aspects reflect the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
89
Figure 1: Selection of Evaluation Topics and Funding Source
Consultation with CDB Operations and OAC/Board for selection of
evaluation topic
Internal review of Approach Paper
Specific Evaluation Study Design and Budgeting
OIE Draft Terms of Reference / Approach
Paper
Detailed ToR or Final Approach Paper if sufficiently detailed.
Finalise Approach Paper and submit to OAC/Board
Final Approach Paper
OAC ApprovalOAC minutes
Paper
Funding Track
Final Approach Paper/ToR
3-year Work Programme and Budget (approved by Board)
Board approval necessary If above USD
150,000
Board notification only if USD 150,000 or
below
Board Approval
Board Paper
Annual OIE report and work plan
submission to OAC
OIE – Selection of consultants (if any) contracting
OIE Admin Budget or …
… SDF
Prepare TA Paper (content similar to Approach Paper but different
format.
TA Paper
Approval – Internal Loans Committee
OIE – Selection of consultants (if any)
contracting
90
2. Consultation and ownership“The credibility of evaluations depends to some degree on whether and how the organization’s approach
to evaluation fosters partnership and helps build ownership and capacity in developing countries.”
(ECG good practices)
The OIE engages with the OAC, CDB senior management and operations for agreeing its 3-year work plan and then for selecting the specific topics and themes. It also discusses the evaluation approach paper (design and implementation plan) with the CDB and OAC before completing the final version. However, preliminary and final drafts of the report are only submitted to the CDB line and senior managers for comment and factual errors. Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the results and their implications. Discussions with the OAC are more limited due to the overburdened agenda of OAC and Board meetings, as previously discussed.
In short, the OIE is to be commended for following the recommendations of professional good practices and standards on participative approaches; it has succeeded in having introduced a modus operandi that involves the key players in the selection of evaluation topics, the evaluation designs and their results. Figure 2 below provides an overview of the evaluation implementation and stakeholder engagement processes.
Figure 2: Evaluation Study Implementation and Feedback Loops
Arrangement AFully outsourced / external
consultants; oversight by OIE
Preparations:Detailed evaluation plan (incl tools,
timeline, etc.) and logistics
Production of Inception Report / Approach Paper
Arrangement BConducted by OIE
staff
Arrangement CJointly: external
consultants and OIE
Terms of Reference
Prepares Inception Report /
Approach Paper
Presentation/workshop:Interim findings and conclusions for immediate feedback and validation
Data Collection and Analysis
OIE
Summary and ppt for workshop presentation
and discussion with CDBSubmission of Draft Final
Report to OIE
Board notification only if USD 150,000 or
below
Draft Final Report
Review loops – OIE and CDB (potentially also BMC)
91
5.6.
Notes to Figure 2
5. The OIE informed the Panel that this is an abbreviated version as there are e.g. additional steps (secondary processes) when evaluations are procured (tendering or single source), when there are additional review loops and updates to OAC etc.
6. OAC may also decide to return the report to OIE, the Panel were informed, or demand from Management specific actions based on the report.
This process is engaging and appears to have secured senior management and OAC interest and buy-in as witnessed in the latest studies. But there is the downside too! The process takes much time and, in our view, is partly unnecessary. The Panel appreciates that staff from operations as well as the AMT may both want to confer on an appropriate management response, but this should not be the case for reviewing an independent report for factual errors. The two-phase approach seems somewhat inefficient and unnecessary in our opinion.
Contact between the OIE, the CDB and/or the OAC during the actual study implementation is most often restricted to the occasional progress report, particularly when studies run behind time. There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. The OIE does, however, arrange discussions for reflecting on emerging findings, but we are not sure of how systematic this feedback loop is.
Prepare for disclosure and dissemination
Final OIE approved report to CDB Senior Management for Management Response
Feedback to evaluation lead
Submission of Final Report to
OIE
Final Report
Final Report and Management Response submitted to
OAC/BoardFinal Report and
Mgt. Resp.
Management Response
OIE ApprovalFinal Report and Management Response considered by CDB
AMT
OAC/Board endorsed
92
More generally speaking, outside of an evaluation study, the OIE has limited dealings with operations. The OIE has an advisory role in providing them with help, particularly with providing training, guidelines and tools to support self-evaluations. We are nevertheless concerned about the seeming distance between these two and how this has affected the perceived value of evaluation. (For further on this point, please see the section below on “Self- and Independent Evaluations”)
But the Panel also wishes to stress that this is not the case for newly appointed senior managers. A much more open attitude to evaluation and appreciation of its potential value was evident; they expressed interest in drawing out important lessons on what works, how, for whom, and under what conditions. In one case, interest was followed up in practice; the OIE was recently invited by a senior manager to share evaluative knowledge and experience with his staff regarding policy based operations.
Certainly, we can say that overall, the key stakeholders within the CDB are adequately integrated into the evaluation process as to foster their buy-in and ownership. But more generally, we feel that the utility of independent evaluations can be improved by fostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that evaluation can offer can add value to understanding the strengths and weaknesses of such strategies. This however cannot be done overnight and takes a long time.
3. Tools to support the evaluation processSo far, during this transitional phase, the OIE has mainly focussed on improving the tools to support the operations areas’ self-evaluations. This has left the OIE with little time to produce the checklists or tools to support its own studies. There are plans to develop an OIE Manual to guide and support the independent evaluation process. Such plans should be encouraged, as these documents will form a very important part of training, particularly for newcomers to the OIE team.
In the meantime, the OIE and operations staff refers to the Performance Assessment System (PAS) Manuals for evaluation activities. The manuals are based on DAC criteria and ECG principles. Much emphasis is given to the rating system and how and what should be rated. However we find them lengthy, unwieldy and overcomplicated. Moreover, such manuals should be used for reference, but cannot and should not replace first-hand training in how to plan, conduct and manage the evaluation process.
Quality Assessment (QA) and Quality at Entry (QaE)
There was a transition period between 2012 and 2014 to establish the OIE. Work on the PAS, QaE, PCRs, ARPP, which had started earlier, was therefore completed after OIE came into existence, but it effectively had no formal ‘home’ in operations. The Panel was told that there had been some discussions about creating a Quality Assurance unit within CDB (OPS) but the current status is unclear.
The QaE Guidance Questionnaire was developed before and completed by the OIE. It was used to assess the documents that came across to the OIE for comments at the Review Stage. The results were then sent to the Portfolio Manager/Project Coordinator indicating any gaps/issues that needed to be addressed or clarified. QaE Guidance Questionnaires were developed for all the Bank’s lending products, CSP and to assess the quality of supervision.
After the QaE was launched bank wide, several operations officers saw the merit in using the QaE Guidance Questionnaire in the field and adopted it as a tool for their use during the appraisal mission in order to cross check and test their data collection and analysis.
OIE’s use of the QaE was discontinued in 2014 due to limited resources and a stronger focus on evaluations. It still sometimes comments on specific appraisals, but very selectively.
93
Both QaE and QaS (quality at supervision) are also addressed in the PAS Manuals. In addition the QaE and PAS have been incorporated in Volume 2 of the Operations Manual OPPM.
The Review Panel assessed the QaE forms. They are relatively standard, adapted to the specificities of the CDB. They contribute to judging a project’s expected quality in a relatively objective way. As such, they are are helpful, as a benchmark, in the ex-post assessment of projects.
The Panel considers that the lack of an established Quality Unit in the CDB (and independent from OIE) is a weakness that should be addressed in the near future.
4. Credibility and Quality of Evaluation ProductsAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An independent evaluation is processed as follows; the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
Since 2012, the OIE has produced a range of studies and approach papers. This review is based on those listed below as provided by the OIE, and cover the period from May 2012 to December 2015. It includes 3 evaluations (in blue), 4 Assessment studies (in brown) 14 validations of self-evaluations (in green) and 3 Approach Papers (in purple) for upcoming evaluations. These are listed below in Table 4.
Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
Board Meeting
Date Type / Topic
251 May 2012 Ex-Post Evaluation Report on Road Improvement and Maintenance Project, Nevis -St. Kitts and Nevis.
Validation of Project Completion Report on Sites and Services – Grenada. Assessment of Effectiveness of Implementation of Poverty Reduction
Strategy 2004-09.253 Oct. 2012 Assessment of Extent and Effectiveness of Mainstreaming Environment,
Climate Change, Disaster Management at CDB.254 Dec. 2012 Assessment of the Implementation Effectiveness of the Gender Equality
Policy and Operational Strategy of the Caribbean Development Bank. Validation of Project Completion Report on Enhancement of Technical and
Vocational Education and Training – Belize. Validation of Project Completion Report on Fourth Road (Northern Coastal
Highway Improvement Section 1 of Segment II) Project – Jamaica. Assessment of the Effectiveness of the Policy-based Lending Instrument.
256 May 2013 Validation of Project Completion Report on Expansion of Grantley Adams International Airport – Barbados.
94
Validation of Project Completion Report on Fifth Water Supply Project – Saint Lucia.
261 May 2014 Validation of Project Completion Report on Immediate Response Loan, Tropical Storm Gustav, Jamaica.
Validation of Project Completion Report on Social Investment Fund, Jamaica.
Validation of Project Completion Report on Disaster Mitigation and Restoration – Rockfall and Landslip, Grenada.
263 Oct. 2014 Validation of Project Completion Report on Basic Education Project – Antigua and Barbuda
263 Oct. 2014 Approach Paper for SDF 6 & 7 Multicycle Evaluation
264 Dec. 2014 Validation of Project Completion Report on Policy-Based Loan – Anguilla Validation of Project Completion Report on Immediate Response Loan -
Tropical Storm Arthur – Belize. Evaluation of Technical Assistance Interventions of the Caribbean
Development Bank Related To Tax Administration and Tax Reform in The Borrowing Member Countries 2005-2012.
265 March
2015
Approach Paper for the Evaluation of Policy Based Operations
266 May 2015 Validation of Project Completion Report on Upgrading of Ecotourism Sites – Dominica
The Evaluation of the Caribbean Development Bank’s Intervention in Technical and Vocational Education and Training (1990-2012)
267 July 2015 Validation of Project Completion Report on The Belize Social Investment Fund I Project − Belize
268 Oct.2015 Approach Paper Country Strategy and Programme Evaluation, Haiti
The review and analysis of these documents is based on the UNEG Quality Checklist for Evaluation Reports (http://www.uneval.org/document/detail/607) as well as on ECG guidance (Big Book on Good Practice Standards).
Approach Papers
Three Approach Papers (APs) were made available to the panel (see Table [ref] above). An AP describes the rationale for the evaluation, the background to the topic evaluated, the evaluation framework (criteria and questions) and approach. It also describes the team and provides an initial planning. Being the first main deliverable of OIE’s evaluation process, APs are the starting point and therefore a major determining element in the roll-out of each evaluation. Therefore APs “have to get it right”.
The APs examined are clearly written, well-structured and of reasonable length.56 We were surprised to find, however, that they do not make explicit the objectives of the evaluated intervention(s), e.g., through a clear objective tree, or through an explicit theory of change, intervention logic or logframe. Whilst one of the APs contains, in an appendix, a results framework for the evaluation, the results framework for the intervention (PBO) itself is lacking.
Inception reports
Only one Inception Report was given to the Panel for review (SDF 6&7). This gives an in-depth description of the evaluated programme and provides a clear Theory of Change. It is good
56 Opportunities remain of course to be more concise and to move parts to appendices, e.g., detailed descriptions of the evaluation team or part of the description of the evaluated intervention.
95
practice that this is established after a pilot field mission, which helps to amend the initial AP on the basis of field observations and sharpen the evaluation questions if needed.
However, it is still considered to be good practice to have the Theory of Change elaborated in the initial design documents . This would facilitate OIE evaluations after project completion. Establishing the Theory of Change of any intervention would be included in the QaE form more explicitly, to be developed between the Quality unit referred to above, and OIE.
Evaluations and Assessments
Three evaluations and four assessment reports completed during the review period were considered. Assessments are similar to evaluations but have a narrower scope; they focus on a limited set of aspects or judgment criteria, mainly effectiveness, i.e. achievement of objectives. Evaluations generally base their judgment on the internationally recognised DAC criteria as well as aspects of the CDB and BMC’s management of the intervention.
In general, these reports are of reasonable quality. In the main, they explain the evaluated object57 and provide evaluation objectives. The findings are organised around the evaluation criteria or questions detailed in the scope and objectives section of the report. They are based on evidence derived from data collection and analysis methods as described in the methodology section. The reports tend to dwell on the limitations that the evaluation encountered, but without becoming defensive. In one case (PBL Assessment) the report starts with a summary of the reviews on the topic done by other MDBs. This was a pleasant surprise and indeed a good practice that could well be adopted in future evaluations too.
However, the reports also show several significant weaknesses:
- Reports do not always provide clear (reconstructed) intervention logics or theories of change for the intervention(s) evaluated.58 Evaluation criteria and questions are defined at a fairly general level. They are translated into more precise “research questions” (in an “Evaluation Design Matrix”, for each project for each criterion). However, it is unclear how these questions relate to the intervention logic (as this is not made explicit). This may be done in inception reports (of which, as noted above, only one was available for review), but should be done also in the final reports.
- The reports do not describe the link from the evaluation questions to the answers, how the evaluation judgments are made and how these ultimately transform into ratings for each criterion and each project. In other words, the explanation provided in the evaluation frameworks is inadequate. The “evaluation design matrix” currently used does not provide sufficient insight into how ultimately an intervention’s performance is judged.59 Links between findings, conclusions and recommendations could be improved by making this more explicit. In other words, reports should include the story on how the evaluand is credibly linked to any observed outcomes and impacts, and should be clear on how causal claims are made.
- With the exception of the PBL Assessment, reports are lengthy and detailed. One reason for this is an over-emphasis on ratings. Their detailed discussion, project by project, criterion by criterion, occupies a very prominent position in the evaluation reports’ main body of text. Although ratings are traditionally an important element in evaluations of MDBs, too strong
57 Sometimes in great length: for instance with the SDF 6&7 multicycle evaluation report it is only at page 30 that we find the beginning of the report on findings…58 Again with the SDF 6&7 evaluation, it is said to be guided by a “Logic Model” which is not explained.59 Marlène: I moreover have the idea that the methodology (often described as “visits”) is based on interviews and little hard evidence. Any view on this?.JM: My “interview-based evaluations”!!
96
an emphasis can be tedious and may distract the reader from the real lessons to be drawn. The detailed discussion of ratings, and their evidence base, would be better placed in an Appendix, with a brief summary in the main report. This would help give the lessons and recommendations a more prominent position than is now the case. This would also help make the evaluation reports not only shorter but also more interesting to read; this could help add value to evaluation’s image within the organisation.
- The reviewers feel that the OIE evaluations tend to over-emphasise objective-based evaluation60 and the DAC criteria to the exclusions of considering other evaluation approaches such as Developmental Evaluation (Patton, 201061); evaluation should be case specific and answer the actual information needs of managers and other decisions makers rather than always concentrating on final performance.
- Related to the previous point (and again with the exception of the PBL Assessment) executive summaries (approximately 8 pages) are too long. For the evaluation report to increase potential impact, they would need to be reduced to 2 to 3 pages and be more focused; again this could be done by dwelling less on the individual ratings of projects and more on key findings, lessons and conclusions. More generally, reports could be better adapted to the needs of the different audiences. Although not strictly limited to evaluations, The Health Evidence Network Reports62 are a model that could be adapted for evaluation reporting purposes; they are specifically geared towards addressing policy and decision-making.
- The “Recommendations to BMCs” are an interesting feature of the reports, (although we are unsure to what degree such recommendations could be effectively followed up by OIE or the Bank, but certainly could taken up with BMC Board members.
- Reports (e.g. the evaluation report on Technical Assistance) focus much on technical problems that were encountered during the evaluation. Although these are important issues, again to improve the report’s flow and “readability” this section would be better placed in the Appendix. What counts is the story of the intervention, not the story of the evaluation (see “Limitations” section in the TA report for instance)
OIE Validations of Project and Country Strategy Programme Completion Reports (referred to globally as PCRs hereafter)
As said above, the OIE has the mandate to validate the Project and Economic departments PCRs and CSPCRs. However, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
The validations tend to repeat the different items reported in the PCRs and then provide extensive comment on each. The PCVRs go into great depth and detail, which makes the documents rich and complete. This is their strength – but also their weakness. The depth and
60 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.61 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press62 See the reports available at the WHO’s Health Evidence Netowkr at http://www.euro.who.int/en/data-and-evidence/evidence-informed-policy-making/health-evidence-network-hen
97
level of detail, as well as the repetitions from the original PCRs, makes PCVRs (overly) lengthy (20-40 pages) and difficult to read. The OIE reported spending approximately 27.2% of its time on validating PCRs in 2015 compared with 44.4% on its core work, i.e. doing or managing the higher level evaluations. That is more than half of its evaluation work is being spent on the validation process. Finally, the PCVRs now seem to be, to a great extent, a standalone output of OIE. It is not always clear to us how they are being used as the “building blocks” for the OIE’s independent evaluations. Making this clearer in the independent evaluations would help show the link and therefore the value of the time being spent on the self-evaluation validations.
To conclude, the review finds that the OIE has taken steps to improve the perceived utility of evaluation in several ways. In the first instance, by planning its work to provide relevant and timely evidence geared towards helping the Board with its oversight and decision making tasks. The topics are selected through dialogue between the OIE and key CDB stakeholders and reflect priorities of the CDBs strategic plan. Secondly, by securing the interest and consequently the buy-in of the OAC and CDB senior management through engaging their input throughout the evaluation process. This is evidenced by the reported interest in the latest three studies, the Country strategy programme in Haiti, the evaluation of policy-based operations and the SDF 6& 7 multicycle assessment.
The OIE products are of an acceptable quality and could be even better if some of the shortcomings were addressed. However, the products themselves do not impair the utility of OIE’s work; this is undermined in several ways: (1) by the time delays in commenting on PCRs (OIE) and providing feedback to the independent evaluations (operations and management) (2) by the inefficient processes for agreeing topics and funding sources as well as providing OIE with management responses to its reports.
Putting Evaluation to Use: transparency, feedback and follow-upThere are several ways that evaluation can be, and is being used. As John Mayne has pointed out in his many publications on the issue,63 when we talk of evaluation use, we are mainly thinking about its Instrumental use—use made to directly improve programming and performance. But there is also conceptual use - use which often goes unnoticed or more precisely, unmeasured. This refers to the kind of use made to enhance knowledge about the type of intervention under study in a more general way. Or even Reflective use— this refers to using discussions or workshops to encourage and support reflection on the evaluation findings to see how they might contribute to future strategies.
In the case of the CDB there is some evidence to suggest that “use” is not only instrumental, but other types are also developing. For example, in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessons. (Reflective use)
Another important use, as recommended by the ECG, is that from time to time a synthesis of lessons is drawn from a number of evaluations and made available publically. In fact the Panel was impressed to hear that in the past, the evaluation unit had done this drawing on lessons from evaluations of the power sector. (Conceptual use) Although nothing has happened since, it is now on the “to do list” for 2016 (OIE’s 2016 Work Plan).
As for instrumental use, responsibility for using the knowledge generated through evaluation and for possibly drawing up an action plan of what should be done is up to CDB senior management and the relevant CDB department and division. Oversight on applying recommendations and picking up on the lessons drawn is the responsibility of the OAC.
Evidence on how evaluations have actually contributed to decisions or negotiations is lacking or confusing, Certainly the OIE is unaware of the extent to which its evaluations are put to use. On
63 See for example, his opening chapter to Enhancing Evaluation use: Insights from internal Evaluation Units, Läubli Loud, M. and Mayne, J. 2014, Sage Publications
98
the one hand, the OAC minutes sometimes indicate that lessons learned are integrated into the next phase. On the other hand, the reviewers were told that often in the past, the evaluation results were “too old” to be of use as the lessons had already been drawn and used way before the report was completed. Similarly, people’s gaps in memory on how well the evaluative information from previous studies may have been used may also account for the scarcity of evidence.
In response, the Panel questioned CDB staff and the OIE about a particular study, the Technical and Vocational Education and Training Assessment. The feedback was somewhat contradictory. On the one hand, the study was criticised as “confirming” news rather than bringing “new news”. However, on the other, we learned that In October 2015, the Board of Directors approved a proposal for the revision of CDB’s Education and Training Policy and Strategy. Work on this has already begun and an external consultant has been engaged to lead the process.
Although it is one of the OIE’s tasks to set up a database on results and lessons learned from evaluations, so far this has not been a priority. There is also currently no systematic tracking of lessons or recommendations arising from the evaluations, or on any progress in their uptake. (The Panel has already referred above to OAC’s lack of oversight in the use of evaluation.)
The OIE’s role in supporting CDB’s organisational learning is clearly specified in the Evaluation Policy, with many good suggestions for knowledge sharing activities such as “brown-bag lunches, workshops, pamphlets and short issues papers” (p. 19). So far, however, the OIE’s lead role on the knowledge sharing side appears to be quite limited. It has provided advisory input in Loan Committee discussions, and organises workshops together with the relative operations department for discussing the implications of evaluation studies. Ultimately, of course, the uptake of evaluation results and knowledge is in the hands of management. But the evaluation unit has an important role to play in terms of knowledge broker and knowledge manager. Both have tended to be underplayed in OIE’s work plan so far.
Transparency: The Communication Strategy
In recent times and with the approval of its new Disclosure Policy, the CDB has started to post its independent evaluation reports on its website. (There is nothing on the self-evaluations). The website also presents a good overview of the role and function of the OIE and evaluation within the CDB. This is a step in the right direction for sharing information. However, in our view, the CDB’s communication strategy is the weakest part of the evaluation system to date.
The Panel has already commended the OIE in its efforts to engage the CDB and the OAC in evaluation work. But reporting and communicating the lessons seem to be entirely targeted at the Board and the CDB. Moreover, the 2015 budget provides only US$2’000 for communication – nothing of which is intended for outreach.
Reviewers feel that actively engaging with the more indirect stakeholders, for example project implementers in the BMCs, NGOs or project beneficiaries is relatively weak64. There appears to be little reflection on drawing out significant messages for the broader group of stakeholders, or on how then to transmit them to the “right” people in the “right” way (knowledge brokerage).
To conclude, evidence on the uptake of evaluation is either confusing or sparse. It is unfortunate that so far no systematic record keeping system has been put into place to track lessons learned or the uptake of recommendations (or actions agreed from management responses). The OIE plays a weak role in brokering the knowledge generated through evaluations to the benefit of external partners and in managing such knowledge. Although the Evaluation Policy specifies the need for “distilling evaluation findings and lessons learned in appropriate formats for targeted audiences both within and outside the CDB” (p.19) such a targeted communication strategy has yet to be developed and budgeted.
64 A broader communication strategy is one of the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
99
Strengthening Evaluation Capacities and Networking From the onset in 2012, the OIE has stressed the importance of developing and strengthening evaluation capacities within the OIE, the CDB and, subject to available resources, in borrowing member countries. Building evaluation capacity in BMCs and the CDB is one of the OIE’s mandated tasks. It has been a priority that figures on the work plan from the beginning (Work Programme and Budget 2012-2104) The idea of developing an internship programme for graduates from the Caribbean region was one idea that was advanced to help build local evaluation resources. However, the capacity-building has primarily been focused on OIE and CDB staff to date. One of the OIE’s two objectives for 2015 therefore was to take up the challenge and “strengthen evaluation capacities and networking” to include reaching out to the BMCs.
Developing OIE staff capacities
The change from project level to strategic and thematic evaluations does require different evaluative skills and competencies. The MDB Evaluation Pyramid presented below in Figure 3 shows the different types of evaluation and changing resource needs as one ascends the pyramid. Implicit here also is the change in the type of expertise and competencies needed as evaluation aspires to the higher levels.
Consequently for 2015, the OIE set itself the objective of networking and developing working partnerships with regional and international evaluation entities and academic institutions. The rationale was twofold: (1) secure further support and guidance as well as (2) increase its outreach and coverage through joint work and international exposure. Another implicit aim was to benefit from partners’ contacts in the BMCs wherever possible so as to improve data collection and quality.
Figure 3: The MDB Evaluation Pyramid65
65 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
100
The OIE has therefore linked up with Carleton University in Canada and the University of the West Indies, Barbados campus. The OIE was also approached by the Development Bank of South Africa to exchange experiences about setting up an evaluation entity in a “small” development bank. However, its attempt to become a member of the Evaluation Cooperation Group was not successful for reasons beyond its control.
The OIE is to be commended in addressing the issue of staff competencies and professional development more generally. New developments in evaluation as well as new developments in the scope of OIE’s work may necessitate new competencies. For this reason, organisations such as the International Developmental Evaluation Association have recommended that the competencies of evaluators and evaluation managers should be periodically reviewed. Several publications now exist on competency requirements and suggestions for the periodic review of staff competencies.66
It is not within this remit to compare and contrast OIE’s competencies with those recommended by international and national agencies. However, what we can say is that the OIE demonstrates great forethought in taking this on board.
Capacity building within CDB
The OIE’s objective also consists of continuing to develop measures for improving the monitoring and self-evaluation side of CDB’s work. OIE’s strategy here is to use the windows of opportunity on offer through some of the training sessions that are being organised by CDB as part of its shift towards MfDR e.g. by Corporate Planning Services and Technical Assistance. For 2016 it is also planned to have the OIE present at the annual staff meeting and Learning Forum.
The OIE also organises some ad hoc training with operations, for example to help understand new tools e.g. for drawing out lessons from self-evaluation reports and, more generally, in helping staff appreciate how evaluation can add value to the organisation’s work. Measures include providing advisory services on demand, and providing training alongside the introduction of new or revised tools.
Capacity building in the BMCs
This is an ambitious task and would require additional investment; from the bi-annual work plans to be effective. A modest attempt has been made in 2015; from what we understand, the OIE has joined together with the Carleton University and the University of the West Indies, using their networks in some of the BMCs, to try to develop this aspect.
To conclude, we cannot comment on the quality or reaction to such training, but can commend the OIE for making capacity building one of its priority objectives. From both the Policy and the documents we reviewed, we note that capacity building was always seen to be an important aspect of OIE’s work, but hitherto has received little strategic focus. But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.
Adequacy of the OIE’s human and financial resources to support its work
OIE’s Human Resources;
The OIE is has a staff of 5; the head, 1 senior evaluation officer and two evaluation managers, plus one administrative assistant. Three of the five were recruited from within the CDB. The
66 E.g. IDEAS, (2012) Competencies for Development Evaluation Evaluators, Managers and Commissioners, the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice (2010) and the Swiss Evaluation Society’s Evaluation Managers Competencies Framework (2014)
101
limited capacity means that it is not feasible to cover all the types of evaluation activities outlined in the Evaluation Policy. Yet there is some indication from the Board that OIE should embark on impact evaluations at some future stage. An increasing demand for evaluation and for impact evaluations in particular, would run the risk of overstretching the OIE’s capacity to deliver credible and useful evaluations. Moreover, there are many other designated OIE activities that should be recognised as valuable work; the validations, building CDB and BMC evaluation capacity, providing supervision, advice, knowledge management and brokerage as well as managing evaluation contracts, The time needs of dealing with all of these may be underestimated in OIE’s budgets; all are important for assuring best value from evaluation. The Panel is concerned that a demand for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level evaluations may undermine the importance and time needs of other essential tasks.
Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses.
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.
102
Self- and independent evaluationSelf-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes. Both types of evaluation are important as they are at the very heart of the evaluation function; they are said to be the building blocks for the more strategic evaluations that the OIE is now undertaking.
The Evaluation Coordination Group recommends that the self-evaluations be carried out by the relevant operations department and in turn, reviewed and validated by the organisation’s independent evaluation office. The CDB’s Evaluation Policy therefore talks of “validating all self-evaluations” as being one of OIE’s essential oversight tasks.
Within CDB, the self-evaluations should provide management with performance assessments and thereby serve an accountability function to the CDB and Board. To support the process, the OIE provides operations with manuals and checklists for guidance. Once a self-evaluation report is to hand, it is given over to the OIE for the validation of its technical quality and credibility.67
However, in the CDB case, there are well-documented issues that have affected the quality and timeliness of the self-evaluations on the one hand, and therefore the quality of the foundation on which to build the independent evaluations. Paucity of documentation within CDB, paucity of data collected and available in the Borrowing Member Countries (BMCs), time delays in producing completion reports and in turn, having them validated by the OIE - all such issues were systematically raised during interviews and in some of the independent evaluation reports. There appears to be little incentive to complete self-evaluations in a timelier manner.
Generally speaking, many of the monitoring data problems appear to be due to a lack of management oversight. For example, with the introduction of results-based management, the logic frame and monitoring and data needs are systematically being built into intervention design. However, the BMCs are not delivering the data as contractually agreed at the outset. Incentives to support any significant change towards building a results-based culture seem to be weak and sanctions seem to be rarely enforced when the supply of data is lacking or lengthy delays to the projects occur. Although we can appreciate the complexities of trying to enforce monitoring compliance, this means that often, project deadlines have had to be extended, data gaps are not being satisfactorily dealt with and in turn, there has been a void in the quality and quantity of available evidence for the CDB’s self-assessment of project performance. For some time, this lack of oversight has been tolerated. Part of the problem is the low priority accorded to completing the self-evaluation reports by operations, coupled with the absence of any focal point within senior management to drive the process and deal with the problems.
No record is kept of how the self-evaluation results are actually used. They do not appear on the CDB website, but we were told that the findings are integrated into the following project designs. Hence we are somewhat unclear as to the utility of these reports at present. The situation is exacerbated by a rather confused image of evaluation: some operations staff consider OIE’s input (through validations or independent evaluations) to be sometimes over-critical, regulatory and adding little value; it is a threat rather than an opportunity for learning. Yet at the same time, evaluation is recognized as an integral part of result-based management.
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.
Some managers however seem to start changing the status quo. For example a revised and simplified template for producing project completion reports is being considered, and mid-term
67 According to the Evaluation Policy, OIE should validate all PCRs and CCRs but due to the backlog of reports and the delay in completing them (sometimes years later) since October 2015, the OIE has secured OAC agreement to validate a maximum of 6 per year, which are selected in consultation with the OAC.
103
project reviews are expected to be more stringent in looking at monitoring plans and practices and tying disbursements to performance. In some cases we also learned of incentives being introduced to encourage project managers to complete their reports in a timelier manner. But much remains to be done and, since the OIE is no longer responsible for monitoring and project evaluations, there is a void that needs to be filled. It is up to line managers to drive this work forward.
To conclude, it is fair to say that in view of a number of “frustrations” between the OIE and operations, which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring data, the added value that evaluation might offer to the operations area is ill recognized. Moreover, the link between self-evaluation as the building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
104
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”68
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
68 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
105
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”69 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 70 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
69 CDB (2011) Evaluation Policy (p.2)70
106
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
107
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
108
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
109
mendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
110
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
111
APPENDICES
Appendix I - The External Review Mandate – Terms of Reference and Approach Paper
Appendix II -Review Approach, Data collection and Analysis, and Limitations
Appendix III – Overview of OIE Evaluation Practice
Appendix IV - List of Persons Interviewed
Appendix V - List of Documents Reviewed
Appendix VI- List of Topics used to guide interviews with members of CDB Board of Directors
Appendix VII - List of Topics used to guide interviews with CDB staff
112
Appendix III – Overview of OIE Evaluation Practice (prepared by the OIE in response to Reviewer’s request)
Caribbean Development Bank, Office of Independent Evaluation - OIE
Category Response
Percentage of projects subject to project (self-) evaluation
100% - Project Completion Reports (PCR)
Percentage of projects subject to validation by OIE
Approximately 40-50%
About 15 projects exit portfolio annually. Evaluation Policy calls for all PCR to be validated. However, OIE resources insufficient. Validation process reviewed in 2014. Now OAC (Board committee) selects a sample of 6-8 PCR for validation each year.
Percentage/number of projects subject to in-depth review by OIE
None – unless specifically requested by OAC
Due to limited resources, focus of OIE evaluation work programme is on PCR validations and high-level evaluations – including country strategy and programme evaluations (CSPE).
Number of high-level evaluations conducted by OIE (e.g. sector, thematic, geographic)
1-2 per year since 2011
Plan is 2-4 per year from 2016. This would include CSPE (1st planned for Q1 2016: Haiti)
Number of project impact evaluations conducted by OIE
None
OIE includes “impact questions” in high-level evaluations.
Number of project impact evaluations conducted by Bank staff or other non-OIE staff
OIE is not aware of any impact evaluation conducted by the Bank.
However, OIE provides technical support to the Basic Needs Trust Fund (BNTF) in its design of an M&E framework that entails impact evaluations.
Budget In USD mn: 0.78 in 2015; 0.82 in 2016. This is equivalent to about 2.5% of total CDB Administrative Budget.
75% of the budget is for Staff salaries (4 Professionals, 1 Support staff), leaving around USD 190,000 (in 2015) for other expenses, including consultants e.g. for external evaluations. Additional funding is accessed via the Special Development Fund (SDF). This varies according to type and scope of the evaluation, e.g. the ongoing SDF 6/7 Evaluation is SDF funded at USD 255,000.
Budget determined by Board, not separate from administrative budget.
SDF funding for evaluations is considered separately and subject to Bank internal approval process. SDF funding cannot be used to cover OIE expenses such as staff time or travel. Country eligibility for SDF funding is also a consideration. OIE expressed concerns about this funding track in respect to predictability, independence and eligibility limitations.
Head of OIE reports to Board, with administrative link to the President
Terms of appointment for Head
113
5 year term, renewable once. Appointed by the President with the agreement of the Board.
Right of Return for Head Not eligible for other staff positions.
Consultants as proportions of OIE budget
2015: 19% (USD 145,000)
Plus SDF funding. SDF funded evaluations are outsourced.
Last external evaluation (or peer review) of OIE
No external evaluation, though a review of the function was done in 2011, leading to the Evaluation Policy.
OIE External Review completed in April, 2016
Departments or special programmes supporting impact evaluation
None
114
Appendix IV – List of Persons Interviewed
Name Function relative to OIE Type interview
Mrs. Colleen Wainwright Member CDB Board of Directors (UK)
Face to face
Mrs. Cherianne Clarke Alternate MemberCDB Board of Directors (UK)
Face to face
Mrs. Jean McCardle Member CDB Board of Directors (Canada)
Face to face
Dr. Louis Woodroofe MemberCDB Board of Directors (Barbados)
Mr. A: de Brigard Former Member CDB Board of Directors
Skype interview
Mr. H. Illi Fromer Member CDB Board ofDirectors
Telephone interview
Mrs. Claudia Reyes Nieto Member CDB Board of Directors
Telephone interview
Mr. Bu Yu alternate DirectorCDB Board of Directors
Face to face
Mr. Michael Schroll(Barbados)
Head OIE
series of interviews viaSkype and face-to-face
Mr. Mark Clayton OIE Senior Evaluation Officer Focus GroupMrs. Egene Baccus Latchman OIE Evaluation OfficerMr. Everton Clinton OIE Evaluation OfficerMrs. Valerie Pilgrim OIE Evaluation Officer
Dr. Justin Ram CDB Director Economics Department
Face to face
Mr. Ian Durant CDB Deputy Director Economics Dept Face to faceDr. Wm Warren Smith CDB President
Joint interviewFace to face
Mrs. Yvette Lemonias-Seale CDB Vice President Corporate Services & Bank Secretariat
Mr. Denis Bergevin CDB Deputy DirectorInternal Audit
Face to face
Mr. Edward Greene CDB Division Chief, Technical Cooperation Division
Face to face
Mrs. Monica La Bennett CDB Deputy Director Corporate Planning Face to faceMrs. Patricia McKenzie CDB Vice President Operations Face to faceMs. Deidre Clarendon CDB Division Chief
Social Sector DivisionFace to face
Mrs. Cheryl Dixon CDB Co-ordinator, Environmental Sustainability Unit
Focus group
Mrs. Denise Noel- Debique CDB Gender Equality Advisor Mrs. Tessa Williams-Robertson CDB Head Renewable EnergyMrs. Klao Bell-Lewis CDB Head Corporate Communications Face to faceMr. Daniel Best CDB Director
Projects DepartmentFace to face
Mr. Carlyle Assue CDB Director Face to face
115
Finance Department
116
Appendix VI - Interview Guide: Members of CDB Board of Directors
Below are a list of themes that I should like to raise with you based on your experience and knowledge of the CDB’s independent evaluation function (Office of Independent
Evaluation).
In each case, I should be grateful if you could illustrate your responses with examples or help this Review by, wherever possible, sending me (or telling me where I can find) any
documents that could support your responses.
This guide is being sent to you in advance to help prepare our meeting. However, our interview will be conducted more in the style of a conversation. The following sub-questions will be used to GUIDE the interview. Please feel encouraged to raise any
additional issues that you feel we should take into account
On the governance and Independence of CDB’s evaluation functionWhat mechanisms are there in place to support its independence?
How satisfactory are the current arrangements in your opinion?
How is the balance between independence and the need for interaction with line management dealt with by the system? For example, what mechanisms exist to ensure that the OIE is kept up to date with decisions, policy / programme changes, other contextual changes etc that could have an affect on OIE evaluation studies / evaluation planning?
On the OIE’s Evaluation PolicyThe CDB’s Evaluation Policy was established in 2011. To what degree do you feel it is adequate? Still relevant?
What suggestions do you have for any improvements?
In your opinion, how adequate is the current quality assurance system for over viewing the evaluation function?
On the quality and credibility of evaluation studiesTo what degree do you believe the reports are fair and impartial?
Do you consider them to be of good quality? Are they credible?
Are you adequately consulted/involved on evaluations of interest to you?
On the relevance and usefulness of evaluations How well does the OIE engage with you / your committee during the preparation, implementation and reporting of an evaluation study to assure that it will be useful to the CDB?
How are the priorities set for the independent evaluations? What criteria are used? Are you satisfied with the current procedure?
117
When OIE evaluation studies are outsourced to external consultants, what criteria are used to make this decision?
How are the priorities for the OIE’s 3.year rolling work plan agreed? In your opinion, is the current plan adequate in terms of coverage and diversity?
In your opinion, do the evaluations address important and pressing programs and issues?
To what extent do you feel that the OIE’s evaluations integrate the cross-cutting theme such as gender, energy efficiency/renewable energy, climate change? What improvements might be made and how?
On the dissemination and uptake of evaluation findings and recommendationsTo what extent do you feel that evaluation findings are communicated to the CDB and its stakeholders in a
a) useful, b) constructive andc) timely manner?
Are evaluation recommendations useful? Realistic?
What mechanisms are in place to assure that evaluation results are taken into account in decision making and planning? What improvements do you feel could be made?
How have you used the findings from any evaluations? Examples?
To what degree do you feel that evaluation contributes to institutional learning? And what about to institutional accountability? Any examples?
What mechanisms are in place to ensure that knowledge from evaluation is accessible toCDB staff and other relevant stakeholders? Are the current arrangements satisfactory?
How satisfied are you with current arrangements? What expectations do you have for the future?
On resourcesHow is the OIE resourced financially and is this satisfactory?
What about the OIE staff, are all the important areas of expertise represented in the team?
On this Review of the Office of Independent EvaluationWhat are your expectations? What are you particularly hoping to learn from it?
Thank you very much for your cooperation and input
118
Appendix VII : Interview Pro-Forma – CDB Staff membersThis presents a list of the topics raised during interviews. It was used to guide the open-ended
discussion – this means that the sequence and exact wording of the questions may not necessarily have followed in this order or been asked in exactly this way.
Changeover to an Independent Evaluation Office? Expectations? Advantages and disadvantages??
Satisfaction with working relations between operations and the OIE from your perspective?
Process of dealing with the PCRs and CCRs? Advantages and limitations?
Quality and credibility of the validation process?
How are the self-evaluation reports used?
Credibility and Quality of OIE’s evaluation reports
Communication of self and OIE independent evaluations? To whom, in what way? Possible improvements?
Independence of the Office of Independent Evaluation (OIEIndependence is absolutely central to the integrity and trustworthiness of evaluation. It is an agreed requirement within the development agencies and in the evaluation community as a whole. In examining the issue of independence and good practice, reviewers are guided by the Evaluation Cooperation Group’s recommendations on good practices, the CDB’s Evaluation Policy and by the 2011 consultancy review of independence relative to the CDB’s evaluation and oversight division71. The appraisal is based on a comparison of the ECG’s recommendations on independence72 and the current OIE status.
OIE and Independence: Recommendations from the OECD Evaluation Cooperation Group (ECG)
The ECG’s considers the issue of independence according to three specific areas: organisational, or structural independence, behavioural, or functional independence and protection from outside interference, or operational independence.
71 Osvaldo Feinstein & Patrick G. Grasso, Consultants, May 2011 Consultancy to Review the Independence of the Evaluation and Oversight Division of the Caribbean Development Bank72 ECG 2014 Evaluation Good Practice Standards, Template for Assessing the Independence of Evaluation Organizations, Annexe II.1
119
Organizational independence, ensures that the evaluation unit and staff are protected against any influence or control by senior or line management, and have unrestricted access to all documents and information sources needed for conducting their evaluations. Also, that the scope of evaluations selected can cover all relevant aspects of their institution.
Behavioural independence, generally refers to the evaluation unit’s autonomy in selecting and conducting setting its work programme and in producing quality reports which can be delivered without management interference.
Protection from outside interference refers to the extent to which the evaluation function is autonomous in setting its priorities, and conducting its studies and processes and in reaching its judgments, and in managing its human and budget resources without management interference.
Conflict of interest safeguards refers to protection against staff conflict of interests be they current, immediate, future or prior professional and personal relationships and considerations or financial interests for which there should be provision in the institution’s human resource policies.
The OIE’s Independence in Practice
Organisational / structural independenceOn the whole, the Panel acknowledges and commends the efforts being made by the CDB to assure OIE’s organisational independence. The CDB’s Evaluation Policy provides for the OIE’s organisational independence from line management and the interview data suggests that there is also wide acceptance and acknowledgement of why the OIE should have such independent status. Table 1 below provides our overall assessment of this aspect of OIE’s independence when compared with ECG recommendations. 73
Table 1: OIE organisational independence compared with ECG recommendations
Aspects Indicators CDB Evaluation Policy (EP) and Practice
The structure and role of evaluation unit
Whether the evaluation unit has a mandate statement that makes clear its scope of responsibility extends to all operations of the organization, and that its reporting line, staff, budget and functions are organizationally independent from the organization’s operational, policy, and strategy departments and related decision-making
Partially Complies The Policy is broad enough to cover the full range of MDB type of evaluations. However in practice this would not be possible without additional human and budget resources
The unit is accountable to, and reports evaluation results to, the head or deputy head of the organization or its governing Board
Whether there is a direct reporting relationship between the unit, and
a) the Management, and/or
b) Board or
c) relevant Board Committee, of the institution
Complies - OIE reports to the Board of Directors (BoD) through its Oversight Assurance Committee (OAC)
The unit is located organizationally outside the staff or line management function of
The unit’s position in the organization relative to the program, activity or entity being evaluated
Complies - The OIE is located outside, and is therefore independent of CDB line management
73 Based on ECG (2014) Template for Assessing the Independence of Evaluation Organizations, Evaluation Good
Practice Standards, Annexe II.1
120
Aspects Indicators CDB Evaluation Policy (EP) and Practice
the program, activity or entity being evaluated
The unit reports regularly to the larger organization’s audit committee or other oversight body
Reporting relationship and frequency of reporting to the oversight body
Complies - The OIE reports x 5 per year to the OAC . Board approval for an additional executive meeting between the Head of the OIE and the OAC at least once per year was given in October 2015
The unit is sufficiently removed from political pressures to be able to report findings without fear of repercussions
Extent to which the evaluation unit and its staff are not accountable to political authorities, and are insulated from participation in political activities
Complies
Unit staffers are protected by a personnel system in which compensation, training, tenure and advancement are based on merit
Extent to which a merit system covering compensation, training, tenure and advancement is in place and enforced
Partially Complies - with CDB human resource policy. However the skill needs of OIE staff ought to be regularly reviewed in light of its move towards higher-level evaluations. Appraisal of skill needs and hiring of relevant staff should be completely under the authority of the Head of Evaluation. This is not sufficiently clear in the Policy or other documents we reviewed.
Unit has access to all needed information and information sources
Extent to which the evaluation unit has access to the organization’s
a) staff, records, and project sites;
b) co-financiers and other partners, clients; and
c) programs, activities, or entities it funds or sponsors
Complies –The available evidence suggests that there is no reason to doubt such access. But systematic and easily accessible documentation is lacking in the CDB; it is one of its weak points.. Delays in getting hold of the relevant documents can have consequences on the timeliness of evaluation studies
However, independence should not mean isolation: There appears to be a detachment between the OIE and CDB that is of concern to the Panel; on the one hand, between the OIE and operations staff, and (2) on the other, in terms of the structural arrangements between the OIE and senior management.
7) In agreeing for the OIE to concentrate on strategic and thematic, in-depth evaluations, responsibility for project monitoring and evaluation were given over to operations. The division is clear and respected. However, it has its drawbacks. With the OIE no longer systematically involved at the front-end of project design, the monitoring data needs are likely to be poorly defined. Weak monitoring data will contribute to weaker evaluations. (More on this point under the heading self and independent evaluations.)
In the reviewers’ opinion, it is a common misunderstanding to assume that providing evaluator advice on monitoring and evaluation data will comprise evaluator independence. On the contrary, evaluation input into project design is essential to assure that the logic, indicators and data needs are addressed so that at some future point in time an evaluation of the achievements can be empirically grounded.
121
This is not to say that the OIE no longer has any influence at the front-end design stage; it has merely shifted the point of focus. The OIE is now systematically providing such input more generally to the corporate planning teams for the tools and systems they are developing to support the MfDR framework. The monitoring data for projects and their implementation should be improved once the Project Performance Evaluation System (PPES) and the Portfolio Performance Management System (PPMS) are updated and operational.
8) In the second place, the OIE has limited formal access to the Advisory Management Team (AMT) weekly meetings where the President and senior management gather to exchange up-to-date information on the dynamics of CDB policy and practice. The OIE is not regularly invited in any capacity to these meetings or given a copy of the agenda or minutes; the OIE is occasionally invited to attend in order to discuss an evaluation report or management feedback. For the OIE, this means that it is unlikely to pick up on the ‘when’ and ‘what’ of key decisional issues or provide input into the discussion based on evaluative information. Its observer status at Loans Committee meetings, or as a participant informer at the OAC and BoD meetings and discussions do not necessarily provide the same insight as to the dynamics of management actions and/or decisions. .
To respond to this situation, the President has agreed to meet regularly with the Head of the OIE in order to keep him up to date with CDB strategic thinking. This is a welcomed change.
OIE Independence and Behavioural Issues The Panel has concerns about some behavioural issues. For example, through both the interviews and documentary review, we learned of considerable delays in processing both the independent evaluation reports as well as OIE’s validation of the CDB’s self-evaluations. Delays are generally due to receiving feedback on the independent reports from first, the relevant operational department, then from the AMT, and then on providing the OIE with a management response that is initially drafted by operations staff before being reviewed by the AMT. (OIE reports cannot be submitted to the OAC without the relevant management response). This two-layer process for preparing submissions to the Board is inefficient and could potentially be a threat to evaluation’s independence in the future by delaying OIE’s timely reporting to the OAC.
OIE validations of the CDB self-evaluations are also submitted to the OAC, but it is in both sides’ interest to clear up any misunderstandings beforehand. Despite attempts to improve the timeframe for completing these validations, delays are more the norm than the exception. Table 2 below summarises our assessment of the behavioural aspects of independence.
Table 2: OIE and Behavioural Independence
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Ability and willingness to issue strong, high quality, and uncompromising reports
Extent to which the evaluation unit:
a) has issued high quality reports that invite public scrutiny (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk) of the lessons from the organization’s programs and activities;
b) proposes standards for performance that are in advance of those in current use by the organization; and
c) critiques the outcomes of the
Partially complies – paucity of data and documentation sometimes hinder the quality of reports. The OIE emphasizes the learning part of evaluation, and is cautious in its criticism recognising that management is going through a transitory stage and can still be overly defensive.
122
Aspects Indicators CDB Evaluation Policy (EP) and Practice
organization’s programs, activities and entities
Ability to report candidly
Extent to which the organization’s mandate provides that the evaluation unit transmits its reports to the Management/Board after review and comment by relevant corporate units but without management-imposed restrictions on their scope and comments
Partially complies - as sometimes reporting to the Board is compromised by delays in the review/comment process between the OIE and the CDB. Any delay with the production of a Management Response will also mean that submitting a report to the Board in a timely manner is impaired since the two have to be submitted together.
Transparency in the reporting of evaluation findings
Extent to which the organization’s disclosure rules permit the evaluation unit to report significant findings to concerned stakeholders, both internal and external (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk).
Who determines evaluation unit’s disclosure policy and procedures: Board, relevant committee, or management.
Partially complies - The OIE’s conforms to the CDB’s disclosure policy. However, the dissemination of evaluation findings appears to be currently restricted to website publication and reports to the Board. A more targeted communication strategy to include other key stakeholders, e.g. project implementers in the BMCs should be developed and put in place.
Self-selection of items for work program
Procedures for selection of work program items are chosen, through systematic or purposive means, by the evaluation organization; consultation on work program with Management and Board
Complies - The OIE also ensures that its work program is drawn up after consultation with both CDB Management and Board to seek their input on relevant topics and themes.
Protection of administrative budget, and other budget sources, for evaluation function
Line item of administrative budget for evaluation determined in accordance with a clear policy parameter, and preserved at an indicated level or proportion; access to additional sources of funding with only formal review of content of submissions
Partially complies - The administrative budget for supporting OIE work is protected. Access to additional sources of funding is possible if well argued and justified. But the approval process is complex and inefficient. (See Figure 1 below)
OIE and Protection from External influence or interference
Our overall assessment is provided in Table 3 below. The OIE’s independence in the design, conduct and content of its evaluations does not appear to be subjected to any external interference. But securing funding from any sources outside the OIE’s administrative budget, i.e. from the Social Development Fund, is an unduly complex and long process. As such we consider that the current funding process can affect the OIE’s choice with regard to the type of evaluations it can undertake. (See Figures 1 and 2 below)
Table 3: OIE and its Independence from External influence or interference
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Proper design and execution of an evaluation
Extent to which the evaluation unit is able to determine the design, scope, timing and conduct of evaluations without Management
Complies – however within limits of restricted human and financial resources available
123
Aspects Indicators CDB Evaluation Policy (EP) and Practice
interference
Evaluation study funding
Extent to which the evaluation unit is unimpeded by restrictions on funds or other resources that would adversely affect its ability to carry out its responsibilities
Partially Complies - OIE must work within the limits of the agreed administrative budget wherever possible. If additional resources are needed for studies it must seek alternative funds elsewhere. The budget limitations can have an affect on the type of evaluations undertaken and therefore its independence in terms of choice.
Judgments made by the evaluators
Extent to which the evaluator’s judgment as to the appropriate content of a report is not subject to overruling or influence by an external authority
Complies – the evidence available suggests that the Board and Management accept the evaluators’ independent interpretation and conclusions Management responses are agreed to be the accepted place to raise any difference of opinion.
Evaluation unit head hiring/firing, term of office, performance review and compensation
Mandate or equivalent document specifies procedures for the
a) hiring, firing,
b) term of office,
c) performance review, and d). compensation of the evaluation unit head that ensure independence from operational management
Complies – the Head of OIE is appointed by the CDB President in agreement with the OAC for a 5 year period which is renewable x 1. The Head could be removed from Office by the President or the Board but only with the agreement of both parties.
However the Head reports to the President for all administrative and personnel matters. Even though this was not recommended in the Osvaldo Feinstein & Patrick G. Grasso report on Independence in 2011, the BoD accepted CDB’s reasons for keeping this arrangement. (e.g.most OAC members are non residents and cannot oversee day-to-day work)
. Extent to which the evaluation unit has control over:
a) staff hiring,
b) promotion, pay increases, and
c) firing, within a merit system
Partially complies - All OIE staff members are treated in the same way as other CDB staff. The Head has limited control over the hiring, firing or promotion of OIE staff.
Continued staff employment
Extent to which the evaluator’s continued employment is based only on reasons related to job performance, competency or the need for evaluator services
Partially complies - Whilst the EP is clear about procedures for hiring, firing and promotion, all of which must conform with CDB human resource policy, there is nothing mentioned about any difference of opinion between the CDB and the Head of the OIE with regard to continued staff employment subject to changes in the level of technical or interpersonal competencies needed to meet new demands.
Avoidance of Financial, Personal or Professional conflicts of interest
This particular aspect refers to the organisation’s Human Resources Policy; there must be provisions in place to protect against actual or potential conflict of interest. The Panel requested via the OIE, to have evidence from human resources on any such provisions but did not receive an answer. It must be assumes that this aspect of independence, past or present, does indeed form part of normal CDB Human Resource Policies
To conclude: The Panel is impressed with the measures CDB has taken to assure the organisational independence of the OIE. Its independent status is accepted and respected by
124
senior and line management. The OIE’s budget is not independent from the overall CDB administrative budget; this affects its choice of evaluation types or approaches. Some of the behavioural issues affecting independence were also of concern, especially due to the delays in the exchange of documents, between the OIE and operations departments, which has a direct effect on timely reporting to the OAC. As for protection from outside interference, our concerns are largely to do with OIE’s independence over staffing issue; there are potential loopholes in current arrangements that could undermine OIE’s autonomy over its staff.
OIE’s Strategy, Work Practices and Work ProgrammeThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. Much of the period 2012 to 2015 has therefore been taken up with preparing OIE’s shift in focus from project-based evaluations to the high-level thematic and in-depth strategic studies. This has meant adopting a three-way approach; (1) for self-evaluations, reducing its time input to support the process and (2) for independent evaluations, taking stock of the gaps in coverage and expertise, and (3) networking to share experiences with centres of expertise and align OIE with international practices. In addition, amongst other duties, it has been supporting the development of MfDR tools and systems such as the Project Performance Assessment System by providing advice and input on programme logic and monitoring needs. The OIE plans to conduct 2-4 high-level studies per year from 2016. The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation pyramid74 are brought out in the remaining sections of this Review, not least given the limited resources available.
To conclude: The OIE has made a first step in proposing a strategy for establishing itself as an independent evaluation resource. But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities.
The Value / Usefulness of OIE’s Independent EvaluationsEvaluation is a powerful tool that can provide useful, evidence-based information to help inform and influence policy and practice. But useful evaluations depend not only on the evaluators’ skills, but on several other important factors as well; 1) on planning evaluations to be relevant to the priorities of the organisation’s work and for their results to be delivered in time to be useful; on the degree of 2) consultation and ultimately ownership by those who seek evaluative information; on the 3) tools used to support the evaluation process per se; and on the 4) credibility and quality of the evaluation products75.
74 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).75 These aspects reflect the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
125
1. Planning relevant and timely evaluationsThe OIE is now working on a 3 year rolling work plan that sets out the broad areas for enquiry. So far, there are no agreed criteria for making the selection of the specific topics for independent evaluation, although the priorities tend to reflect those of the CDB’s strategic plan. Nevertheless decision-making is rather arbitrary based on a process of dialogue between the OIE and the CDB and the OIE and the Board.
One of the OIE’s two objectives for 2015 therefore, was to define a work plan and agree priorities based on an approach that is “utilisation-focused”. This means that the studies are selected and planned to be relevant and useful to the organisation’s needs.
The OIE has achieved this objective with respect to its latest studies, which concerns the Social Development Fund (SDF) Multicycle 6&7 Evaluation, the Haiti Country Strategy evaluation and the evaluation of the CDB’s Policy Based Operations. Each of these three have been planned to deliver their results in time to provide the CDB Board of Directors with relevant information for negotiating the next round of funding. In spite of some delays due to a myriad of reasons, not least to the extra effort needed to secure essential data, the studies are expected to deliver on time.
The processes for agreeing OIE’s work plan and specific evaluations on the one hand, and, in securing alternative funding on the other, are shown in Figure 1 below. The Panel was surprised at learning how bureaucratic (the internal approval process), and inefficient (in view of the time it takes) the process seems to be. The concern here is that such a process could possibly pose a threat to assuring the Board of “timely studies.”
Figure 1: Selection of Evaluation Topics and Funding Source
Consultation with CDB Operations and OAC/Board for selection of
evaluation topic
Internal review of Approach Paper
Specific Evaluation Study Design and Budgeting
OIE Draft Terms of Reference / Approach
Paper
Finalise Approach Paper and submit to OAC/Board
Final Approach Paper
OAC ApprovalOAC minutes
Paper
3-year Work Programme and Budget (approved by Board)
Board approval necessary If above USD
150,000
Board notification only if USD 150,000 or
below
Board Paper
Annual OIE report and work plan
submission to OAC
126
2. Consultation and ownership“The credibility of evaluations depends to some degree on whether and how the organization’s approach
to evaluation fosters partnership and helps build ownership and capacity in developing countries.”
(ECG good practices)
The OIE engages with the OAC, CDB senior management and operations for agreeing its 3-year work plan and then for selecting the specific topics and themes. It also discusses the evaluation approach paper (design and implementation plan) with the CDB and OAC before completing the final version. However, preliminary and final drafts of the report are only submitted to the CDB line and senior managers for comment and factual errors. Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the results and their implications. Discussions with the OAC are more limited due to the overburdened agenda of OAC and Board meetings, as previously discussed.
In short, the OIE is to be commended for following the recommendations of professional good practices and standards on participative approaches; it has succeeded in having introduced a modus operandi that involves the key players in the selection of evaluation topics, the evaluation designs and their results. Figure 2 below provides an overview of the evaluation implementation and stakeholder engagement processes.
Figure 2: Evaluation Study Implementation and Feedback Loops
Detailed ToR or Final Approach Paper if sufficiently detailed.
Funding Track
Final Approach Paper/ToR Board
Approval
OIE – Selection of consultants (if any) contracting
OIE Admin Budget or …
… SDF
Prepare TA Paper (content similar to Approach Paper but different
format.
TA Paper
Approval – Internal Loans Committee
Arrangement AFully outsourced / external
consultants; oversight by OIE
Arrangement BConducted by OIE
staff
Arrangement CJointly: external
consultants and OIE
Terms of Reference
OIE – Selection of consultants (if any)
contracting
127
7.8.
Preparations:Detailed evaluation plan (incl tools,
timeline, etc.) and logistics
Production of Inception Report / Approach Paper
Prepares Inception Report /
Approach Paper
Prepare for disclosure and dissemination
Presentation/workshop:Interim findings and conclusions for immediate feedback and validation
Data Collection and Analysis
OIE
Summary and ppt for workshop presentation
and discussion with CDBSubmission of Draft Final
Report to OIE
Final OIE approved report to CDB Senior Management for Management Response
Board notification only if USD 150,000 or
below
Draft Final Report
Review loops – OIE and CDB (potentially also BMC)
Feedback to evaluation lead
Submission of Final Report to
OIE
Final Report
Final Report and Management Response submitted to
OAC/BoardFinal Report and
Mgt. Resp.
Management Response
OIE ApprovalFinal Report and Management Response considered by CDB
AMT
OAC/Board endorsed
128
Notes to Figure 2
7. The OIE informed the Panel that this is an abbreviated version as there are e.g. additional steps (secondary processes) when evaluations are procured (tendering or single source), when there are additional review loops and updates to OAC etc.
8. OAC may also decide to return the report to OIE, the Panel were informed, or demand from Management specific actions based on the report.
This process is engaging and appears to have secured senior management and OAC interest and buy-in as witnessed in the latest studies. But there is the downside too! The process takes much time and, in our view, is partly unnecessary. The Panel appreciates that staff from operations as well as the AMT may both want to confer on an appropriate management response, but this should not be the case for reviewing an independent report for factual errors. The two-phase approach seems somewhat inefficient and unnecessary in our opinion.
Contact between the OIE, the CDB and/or the OAC during the actual study implementation is most often restricted to the occasional progress report, particularly when studies run behind time. There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. The OIE does, however, arrange discussions for reflecting on emerging findings, but we are not sure of how systematic this feedback loop is.
More generally speaking, outside of an evaluation study, the OIE has limited dealings with operations. The OIE has an advisory role in providing them with help, particularly with providing training, guidelines and tools to support self-evaluations. We are nevertheless concerned about the seeming distance between these two and how this has affected the perceived value of evaluation. (For further on this point, please see the section below on “Self- and Independent Evaluations”)
But the Panel also wishes to stress that this is not the case for newly appointed senior managers. A much more open attitude to evaluation and appreciation of its potential value was evident; they expressed interest in drawing out important lessons on what works, how, for whom, and under what conditions. In one case, interest was followed up in practice; the OIE was recently invited by a senior manager to share evaluative knowledge and experience with his staff regarding policy based operations.
Certainly, we can say that overall, the key stakeholders within the CDB are adequately integrated into the evaluation process as to foster their buy-in and ownership. But more generally, we feel that the utility of independent evaluations can be improved by fostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that evaluation can offer can add value to understanding the strengths and weaknesses of such strategies. This however cannot be done overnight and takes a long time.
3. Tools to support the evaluation processSo far, during this transitional phase, the OIE has mainly focussed on improving the tools to support the operations areas’ self-evaluations. This has left the OIE with little time to produce the checklists or tools to support its own studies. There are plans to develop an OIE Manual to guide and support the independent evaluation process. Such plans should be encouraged, as these documents will form a very important part of training, particularly for newcomers to the OIE team.
In the meantime, the OIE and operations staff refers to the Performance Assessment System (PAS) Manuals for evaluation activities. The manuals are based on DAC criteria and ECG principles. Much emphasis is given to the rating system and how and what should be rated.
129
However we find them lengthy, unwieldy and overcomplicated. Moreover, such manuals should be used for reference, but cannot and should not replace first-hand training in how to plan, conduct and manage the evaluation process.
Quality Assessment (QA) and Quality at Entry (QaE)
There was a transition period between 2012 and 2014 to establish the OIE. Work on the PAS, QaE, PCRs, ARPP, which had started earlier, was therefore completed after OIE came into existence, but it effectively had no formal ‘home’ in operations. The Panel was told that there had been some discussions about creating a Quality Assurance unit within CDB (OPS) but the current status is unclear.
The QaE Guidance Questionnaire was developed before and completed by the OIE. It was used to assess the documents that came across to the OIE for comments at the Review Stage. The results were then sent to the Portfolio Manager/Project Coordinator indicating any gaps/issues that needed to be addressed or clarified. QaE Guidance Questionnaires were developed for all the Bank’s lending products, CSP and to assess the quality of supervision.
After the QaE was launched bank wide, several operations officers saw the merit in using the QaE Guidance Questionnaire in the field and adopted it as a tool for their use during the appraisal mission in order to cross check and test their data collection and analysis.
OIE’s use of the QaE was discontinued in 2014 due to limited resources and a stronger focus on evaluations. It still sometimes comments on specific appraisals, but very selectively.
Both QaE and QaS (quality at supervision) are also addressed in the PAS Manuals. In addition the QaE and PAS have been incorporated in Volume 2 of the Operations Manual OPPM.
The Review Panel assessed the QaE forms. They are relatively standard, adapted to the specificities of the CDB. They contribute to judging a project’s expected quality in a relatively objective way. As such, they are are helpful, as a benchmark, in the ex-post assessment of projects.
The Panel considers that the lack of an established Quality Unit in the CDB (and independent from OIE) is a weakness that should be addressed in the near future.
4. Credibility and Quality of Evaluation ProductsAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An independent evaluation is processed as follows; the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
Since 2012, the OIE has produced a range of studies and approach papers. This review is based on those listed below as provided by the OIE, and cover the period from May 2012 to December 2015. It includes 3 evaluations (in blue), 4 Assessment studies (in brown) 14 validations of self-evaluations (in green) and 3 Approach Papers (in purple) for upcoming evaluations. These are listed below in Table 4.
130
Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
Board Meeting
Date Type / Topic
251 May 2012 Ex-Post Evaluation Report on Road Improvement and Maintenance Project, Nevis -St. Kitts and Nevis.
Validation of Project Completion Report on Sites and Services – Grenada. Assessment of Effectiveness of Implementation of Poverty Reduction
Strategy 2004-09.253 Oct. 2012 Assessment of Extent and Effectiveness of Mainstreaming Environment,
Climate Change, Disaster Management at CDB.254 Dec. 2012 Assessment of the Implementation Effectiveness of the Gender Equality
Policy and Operational Strategy of the Caribbean Development Bank. Validation of Project Completion Report on Enhancement of Technical and
Vocational Education and Training – Belize. Validation of Project Completion Report on Fourth Road (Northern Coastal
Highway Improvement Section 1 of Segment II) Project – Jamaica. Assessment of the Effectiveness of the Policy-based Lending Instrument.
256 May 2013 Validation of Project Completion Report on Expansion of Grantley Adams International Airport – Barbados.
Validation of Project Completion Report on Fifth Water Supply Project – Saint Lucia.
261 May 2014 Validation of Project Completion Report on Immediate Response Loan, Tropical Storm Gustav, Jamaica.
Validation of Project Completion Report on Social Investment Fund, Jamaica.
Validation of Project Completion Report on Disaster Mitigation and Restoration – Rockfall and Landslip, Grenada.
263 Oct. 2014 Validation of Project Completion Report on Basic Education Project – Antigua and Barbuda
263 Oct. 2014 Approach Paper for SDF 6 & 7 Multicycle Evaluation
264 Dec. 2014 Validation of Project Completion Report on Policy-Based Loan – Anguilla Validation of Project Completion Report on Immediate Response Loan -
Tropical Storm Arthur – Belize. Evaluation of Technical Assistance Interventions of the Caribbean
Development Bank Related To Tax Administration and Tax Reform in The Borrowing Member Countries 2005-2012.
265 March
2015
Approach Paper for the Evaluation of Policy Based Operations
266 May 2015 Validation of Project Completion Report on Upgrading of Ecotourism Sites – Dominica
The Evaluation of the Caribbean Development Bank’s Intervention in Technical and Vocational Education and Training (1990-2012)
267 July 2015 Validation of Project Completion Report on The Belize Social Investment Fund I Project − Belize
268 Oct.2015 Approach Paper Country Strategy and Programme Evaluation, Haiti
The review and analysis of these documents is based on the UNEG Quality Checklist for Evaluation Reports (http://www.uneval.org/document/detail/607) as well as on ECG guidance (Big Book on Good Practice Standards).
131
Approach Papers
Three Approach Papers (APs) were made available to the panel (see Table [ref] above). An AP describes the rationale for the evaluation, the background to the topic evaluated, the evaluation framework (criteria and questions) and approach. It also describes the team and provides an initial planning. Being the first main deliverable of OIE’s evaluation process, APs are the starting point and therefore a major determining element in the roll-out of each evaluation. Therefore APs “have to get it right”.
The APs examined are clearly written, well-structured and of reasonable length.76 We were surprised to find, however, that they do not make explicit the objectives of the evaluated intervention(s), e.g., through a clear objective tree, or through an explicit theory of change, intervention logic or logframe. Whilst one of the APs contains, in an appendix, a results framework for the evaluation, the results framework for the intervention (PBO) itself is lacking.
Inception reports
Only one Inception Report was given to the Panel for review (SDF 6&7). This gives an in-depth description of the evaluated programme and provides a clear Theory of Change. It is good practice that this is established after a pilot field mission, which helps to amend the initial AP on the basis of field observations and sharpen the evaluation questions if needed.
However, it is still considered to be good practice to have the Theory of Change elaborated in the initial design documents . This would facilitate OIE evaluations after project completion. Establishing the Theory of Change of any intervention would be included in the QaE form more explicitly, to be developed between the Quality unit referred to above, and OIE.
Evaluations and Assessments
Three evaluations and four assessment reports completed during the review period were considered. Assessments are similar to evaluations but have a narrower scope; they focus on a limited set of aspects or judgment criteria, mainly effectiveness, i.e. achievement of objectives. Evaluations generally base their judgment on the internationally recognised DAC criteria as well as aspects of the CDB and BMC’s management of the intervention.
In general, these reports are of reasonable quality. In the main, they explain the evaluated object77 and provide evaluation objectives. The findings are organised around the evaluation criteria or questions detailed in the scope and objectives section of the report. They are based on evidence derived from data collection and analysis methods as described in the methodology section. The reports tend to dwell on the limitations that the evaluation encountered, but without becoming defensive. In one case (PBL Assessment) the report starts with a summary of the reviews on the topic done by other MDBs. This was a pleasant surprise and indeed a good practice that could well be adopted in future evaluations too.
However, the reports also show several significant weaknesses:
- Reports do not always provide clear (reconstructed) intervention logics or theories of change for the intervention(s) evaluated.78 Evaluation criteria and questions are defined at a fairly general level. They are translated into more precise “research questions” (in an “Evaluation Design Matrix”, for each project for each criterion). However, it is unclear how these questions relate to the intervention logic (as this is not made explicit). This may be
76 Opportunities remain of course to be more concise and to move parts to appendices, e.g., detailed descriptions of the evaluation team or part of the description of the evaluated intervention.77 Sometimes in great length: for instance with the SDF 6&7 multicycle evaluation report it is only at page 30 that we find the beginning of the report on findings…78 Again with the SDF 6&7 evaluation, it is said to be guided by a “Logic Model” which is not explained.
132
done in inception reports (of which, as noted above, only one was available for review), but should be done also in the final reports.
- The reports do not describe the link from the evaluation questions to the answers, how the evaluation judgments are made and how these ultimately transform into ratings for each criterion and each project. In other words, the explanation provided in the evaluation frameworks is inadequate. The “evaluation design matrix” currently used does not provide sufficient insight into how ultimately an intervention’s performance is judged.79 Links between findings, conclusions and recommendations could be improved by making this more explicit. In other words, reports should include the story on how the evaluand is credibly linked to any observed outcomes and impacts, and should be clear on how causal claims are made.
- With the exception of the PBL Assessment, reports are lengthy and detailed. One reason for this is an over-emphasis on ratings. Their detailed discussion, project by project, criterion by criterion, occupies a very prominent position in the evaluation reports’ main body of text. Although ratings are traditionally an important element in evaluations of MDBs, too strong an emphasis can be tedious and may distract the reader from the real lessons to be drawn. The detailed discussion of ratings, and their evidence base, would be better placed in an Appendix, with a brief summary in the main report. This would help give the lessons and recommendations a more prominent position than is now the case. This would also help make the evaluation reports not only shorter but also more interesting to read; this could help add value to evaluation’s image within the organisation.
- The reviewers feel that the OIE evaluations tend to over-emphasise objective-based evaluation80 and the DAC criteria to the exclusions of considering other evaluation approaches such as Developmental Evaluation (Patton, 201081); evaluation should be case specific and answer the actual information needs of managers and other decisions makers rather than always concentrating on final performance.
- Related to the previous point (and again with the exception of the PBL Assessment) executive summaries (approximately 8 pages) are too long. For the evaluation report to increase potential impact, they would need to be reduced to 2 to 3 pages and be more focused; again this could be done by dwelling less on the individual ratings of projects and more on key findings, lessons and conclusions. More generally, reports could be better adapted to the needs of the different audiences. Although not strictly limited to evaluations, The Health Evidence Network Reports82 are a model that could be adapted for evaluation reporting purposes; they are specifically geared towards addressing policy and decision-making.
- The “Recommendations to BMCs” are an interesting feature of the reports, (although we are unsure to what degree such recommendations could be effectively followed up by OIE or the Bank, but certainly could taken up with BMC Board members.
79 Marlène: I moreover have the idea that the methodology (often described as “visits”) is based on interviews and little hard evidence. Any view on this?.JM: My “interview-based evaluations”!!80 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.81 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press82 See the reports available at the WHO’s Health Evidence Netowkr at http://www.euro.who.int/en/data-and-evidence/evidence-informed-policy-making/health-evidence-network-hen
133
- Reports (e.g. the evaluation report on Technical Assistance) focus much on technical problems that were encountered during the evaluation. Although these are important issues, again to improve the report’s flow and “readability” this section would be better placed in the Appendix. What counts is the story of the intervention, not the story of the evaluation (see “Limitations” section in the TA report for instance)
OIE Validations of Project and Country Strategy Programme Completion Reports (referred to globally as PCRs hereafter)
As said above, the OIE has the mandate to validate the Project and Economic departments PCRs and CSPCRs. However, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
The validations tend to repeat the different items reported in the PCRs and then provide extensive comment on each. The PCVRs go into great depth and detail, which makes the documents rich and complete. This is their strength – but also their weakness. The depth and level of detail, as well as the repetitions from the original PCRs, makes PCVRs (overly) lengthy (20-40 pages) and difficult to read. The OIE reported spending approximately 27.2% of its time on validating PCRs in 2015 compared with 44.4% on its core work, i.e. doing or managing the higher level evaluations. That is more than half of its evaluation work is being spent on the validation process. Finally, the PCVRs now seem to be, to a great extent, a standalone output of OIE. It is not always clear to us how they are being used as the “building blocks” for the OIE’s independent evaluations. Making this clearer in the independent evaluations would help show the link and therefore the value of the time being spent on the self-evaluation validations.
To conclude, the review finds that the OIE has taken steps to improve the perceived utility of evaluation in several ways. In the first instance, by planning its work to provide relevant and timely evidence geared towards helping the Board with its oversight and decision making tasks. The topics are selected through dialogue between the OIE and key CDB stakeholders and reflect priorities of the CDBs strategic plan. Secondly, by securing the interest and consequently the buy-in of the OAC and CDB senior management through engaging their input throughout the evaluation process. This is evidenced by the reported interest in the latest three studies, the Country strategy programme in Haiti, the evaluation of policy-based operations and the SDF 6& 7 multicycle assessment.
The OIE products are of an acceptable quality and could be even better if some of the shortcomings were addressed. However, the products themselves do not impair the utility of OIE’s work; this is undermined in several ways: (1) by the time delays in commenting on PCRs (OIE) and providing feedback to the independent evaluations (operations and management) (2) by the inefficient processes for agreeing topics and funding sources as well as providing OIE with management responses to its reports.
Putting Evaluation to Use: transparency, feedback and follow-upThere are several ways that evaluation can be, and is being used. As John Mayne has pointed out in his many publications on the issue,83 when we talk of evaluation use, we are mainly thinking about its Instrumental use—use made to directly improve programming and performance. But there is also conceptual use - use which often goes unnoticed or more precisely, unmeasured. This refers to the kind of use made to enhance knowledge about the type of intervention under study in a more general way. Or even Reflective use— this refers to using discussions or
83 See for example, his opening chapter to Enhancing Evaluation use: Insights from internal Evaluation Units, Läubli Loud, M. and Mayne, J. 2014, Sage Publications
134
workshops to encourage and support reflection on the evaluation findings to see how they might contribute to future strategies.
In the case of the CDB there is some evidence to suggest that “use” is not only instrumental, but other types are also developing. For example, in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessons. (Reflective use)
Another important use, as recommended by the ECG, is that from time to time a synthesis of lessons is drawn from a number of evaluations and made available publically. In fact the Panel was impressed to hear that in the past, the evaluation unit had done this drawing on lessons from evaluations of the power sector. (Conceptual use) Although nothing has happened since, it is now on the “to do list” for 2016 (OIE’s 2016 Work Plan).
As for instrumental use, responsibility for using the knowledge generated through evaluation and for possibly drawing up an action plan of what should be done is up to CDB senior management and the relevant CDB department and division. Oversight on applying recommendations and picking up on the lessons drawn is the responsibility of the OAC.
Evidence on how evaluations have actually contributed to decisions or negotiations is lacking or confusing, Certainly the OIE is unaware of the extent to which its evaluations are put to use. On the one hand, the OAC minutes sometimes indicate that lessons learned are integrated into the next phase. On the other hand, the reviewers were told that often in the past, the evaluation results were “too old” to be of use as the lessons had already been drawn and used way before the report was completed. Similarly, people’s gaps in memory on how well the evaluative information from previous studies may have been used may also account for the scarcity of evidence.
In response, the Panel questioned CDB staff and the OIE about a particular study, the Technical and Vocational Education and Training Assessment. The feedback was somewhat contradictory. On the one hand, the study was criticised as “confirming” news rather than bringing “new news”. However, on the other, we learned that In October 2015, the Board of Directors approved a proposal for the revision of CDB’s Education and Training Policy and Strategy. Work on this has already begun and an external consultant has been engaged to lead the process.
Although it is one of the OIE’s tasks to set up a database on results and lessons learned from evaluations, so far this has not been a priority. There is also currently no systematic tracking of lessons or recommendations arising from the evaluations, or on any progress in their uptake. (The Panel has already referred above to OAC’s lack of oversight in the use of evaluation.)
The OIE’s role in supporting CDB’s organisational learning is clearly specified in the Evaluation Policy, with many good suggestions for knowledge sharing activities such as “brown-bag lunches, workshops, pamphlets and short issues papers” (p. 19). So far, however, the OIE’s lead role on the knowledge sharing side appears to be quite limited. It has provided advisory input in Loan Committee discussions, and organises workshops together with the relative operations department for discussing the implications of evaluation studies. Ultimately, of course, the uptake of evaluation results and knowledge is in the hands of management. But the evaluation unit has an important role to play in terms of knowledge broker and knowledge manager. Both have tended to be underplayed in OIE’s work plan so far.
Transparency: The Communication Strategy
In recent times and with the approval of its new Disclosure Policy, the CDB has started to post its independent evaluation reports on its website. (There is nothing on the self-evaluations). The website also presents a good overview of the role and function of the OIE and evaluation within the CDB. This is a step in the right direction for sharing information. However, in our view, the CDB’s communication strategy is the weakest part of the evaluation system to date.
135
The Panel has already commended the OIE in its efforts to engage the CDB and the OAC in evaluation work. But reporting and communicating the lessons seem to be entirely targeted at the Board and the CDB. Moreover, the 2015 budget provides only US$2’000 for communication – nothing of which is intended for outreach.
Reviewers feel that actively engaging with the more indirect stakeholders, for example project implementers in the BMCs, NGOs or project beneficiaries is relatively weak84. There appears to be little reflection on drawing out significant messages for the broader group of stakeholders, or on how then to transmit them to the “right” people in the “right” way (knowledge brokerage).
To conclude, evidence on the uptake of evaluation is either confusing or sparse. It is unfortunate that so far no systematic record keeping system has been put into place to track lessons learned or the uptake of recommendations (or actions agreed from management responses). The OIE plays a weak role in brokering the knowledge generated through evaluations to the benefit of external partners and in managing such knowledge. Although the Evaluation Policy specifies the need for “distilling evaluation findings and lessons learned in appropriate formats for targeted audiences both within and outside the CDB” (p.19) such a targeted communication strategy has yet to be developed and budgeted.
Strengthening Evaluation Capacities and Networking From the onset in 2012, the OIE has stressed the importance of developing and strengthening evaluation capacities within the OIE, the CDB and, subject to available resources, in borrowing member countries. Building evaluation capacity in BMCs and the CDB is one of the OIE’s mandated tasks. It has been a priority that figures on the work plan from the beginning (Work Programme and Budget 2012-2104) The idea of developing an internship programme for graduates from the Caribbean region was one idea that was advanced to help build local evaluation resources. However, the capacity-building has primarily been focused on OIE and CDB staff to date. One of the OIE’s two objectives for 2015 therefore was to take up the challenge and “strengthen evaluation capacities and networking” to include reaching out to the BMCs.
Developing OIE staff capacities
The change from project level to strategic and thematic evaluations does require different evaluative skills and competencies. The MDB Evaluation Pyramid presented below in Figure 3 shows the different types of evaluation and changing resource needs as one ascends the pyramid. Implicit here also is the change in the type of expertise and competencies needed as evaluation aspires to the higher levels.
Consequently for 2015, the OIE set itself the objective of networking and developing working partnerships with regional and international evaluation entities and academic institutions. The rationale was twofold: (1) secure further support and guidance as well as (2) increase its outreach and coverage through joint work and international exposure. Another implicit aim was to benefit from partners’ contacts in the BMCs wherever possible so as to improve data collection and quality.
Figure 3: The MDB Evaluation Pyramid85
84 A broader communication strategy is one of the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.85 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
136
The OIE has therefore linked up with Carleton University in Canada and the University of the West Indies, Barbados campus. The OIE was also approached by the Development Bank of South Africa to exchange experiences about setting up an evaluation entity in a “small” development bank. However, its attempt to become a member of the Evaluation Cooperation Group was not successful for reasons beyond its control.
The OIE is to be commended in addressing the issue of staff competencies and professional development more generally. New developments in evaluation as well as new developments in the scope of OIE’s work may necessitate new competencies. For this reason, organisations such as the International Developmental Evaluation Association have recommended that the competencies of evaluators and evaluation managers should be periodically reviewed. Several publications now exist on competency requirements and suggestions for the periodic review of staff competencies.86
It is not within this remit to compare and contrast OIE’s competencies with those recommended by international and national agencies. However, what we can say is that the OIE demonstrates great forethought in taking this on board.
Capacity building within CDB
The OIE’s objective also consists of continuing to develop measures for improving the monitoring and self-evaluation side of CDB’s work. OIE’s strategy here is to use the windows of opportunity on offer through some of the training sessions that are being organised by CDB as part of its shift towards MfDR e.g. by Corporate Planning Services and Technical Assistance. For 2016 it is also planned to have the OIE present at the annual staff meeting and Learning Forum.
The OIE also organises some ad hoc training with operations, for example to help understand new tools e.g. for drawing out lessons from self-evaluation reports and, more generally, in
86 E.g. IDEAS, (2012) Competencies for Development Evaluation Evaluators, Managers and Commissioners, the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice (2010) and the Swiss Evaluation Society’s Evaluation Managers Competencies Framework (2014)
137
helping staff appreciate how evaluation can add value to the organisation’s work. Measures include providing advisory services on demand, and providing training alongside the introduction of new or revised tools.
Capacity building in the BMCs
This is an ambitious task and would require additional investment; from the bi-annual work plans to be effective. A modest attempt has been made in 2015; from what we understand, the OIE has joined together with the Carleton University and the University of the West Indies, using their networks in some of the BMCs, to try to develop this aspect.
To conclude, we cannot comment on the quality or reaction to such training, but can commend the OIE for making capacity building one of its priority objectives. From both the Policy and the documents we reviewed, we note that capacity building was always seen to be an important aspect of OIE’s work, but hitherto has received little strategic focus. But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.
Adequacy of the OIE’s human and financial resources to support its work
OIE’s Human Resources;
The OIE is has a staff of 5; the head, 1 senior evaluation officer and two evaluation managers, plus one administrative assistant. Three of the five were recruited from within the CDB. The limited capacity means that it is not feasible to cover all the types of evaluation activities outlined in the Evaluation Policy. Yet there is some indication from the Board that OIE should embark on impact evaluations at some future stage. An increasing demand for evaluation and for impact evaluations in particular, would run the risk of overstretching the OIE’s capacity to deliver credible and useful evaluations. Moreover, there are many other designated OIE activities that should be recognised as valuable work; the validations, building CDB and BMC evaluation capacity, providing supervision, advice, knowledge management and brokerage as well as managing evaluation contracts, The time needs of dealing with all of these may be underestimated in OIE’s budgets; all are important for assuring best value from evaluation. The Panel is concerned that a demand for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level evaluations may undermine the importance and time needs of other essential tasks.
Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses.
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the
138
backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.
Self- and independent evaluationSelf-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes. Both types of evaluation are important as they are at the very heart of the evaluation function; they are said to be the building blocks for the more strategic evaluations that the OIE is now undertaking.
The Evaluation Coordination Group recommends that the self-evaluations be carried out by the relevant operations department and in turn, reviewed and validated by the organisation’s independent evaluation office. The CDB’s Evaluation Policy therefore talks of “validating all self-evaluations” as being one of OIE’s essential oversight tasks.
Within CDB, the self-evaluations should provide management with performance assessments and thereby serve an accountability function to the CDB and Board. To support the process, the OIE provides operations with manuals and checklists for guidance. Once a self-evaluation report is to hand, it is given over to the OIE for the validation of its technical quality and credibility.87
However, in the CDB case, there are well-documented issues that have affected the quality and timeliness of the self-evaluations on the one hand, and therefore the quality of the foundation on which to build the independent evaluations. Paucity of documentation within CDB, paucity of data collected and available in the Borrowing Member Countries (BMCs), time delays in producing completion reports and in turn, having them validated by the OIE - all such issues were systematically raised during interviews and in some of the independent evaluation reports. There appears to be little incentive to complete self-evaluations in a timelier manner.
Generally speaking, many of the monitoring data problems appear to be due to a lack of management oversight. For example, with the introduction of results-based management, the logic frame and monitoring and data needs are systematically being built into intervention design. However, the BMCs are not delivering the data as contractually agreed at the outset.
87 According to the Evaluation Policy, OIE should validate all PCRs and CCRs but due to the backlog of reports and the delay in completing them (sometimes years later) since October 2015, the OIE has secured OAC agreement to validate a maximum of 6 per year, which are selected in consultation with the OAC.
139
Incentives to support any significant change towards building a results-based culture seem to be weak and sanctions seem to be rarely enforced when the supply of data is lacking or lengthy delays to the projects occur. Although we can appreciate the complexities of trying to enforce monitoring compliance, this means that often, project deadlines have had to be extended, data gaps are not being satisfactorily dealt with and in turn, there has been a void in the quality and quantity of available evidence for the CDB’s self-assessment of project performance. For some time, this lack of oversight has been tolerated. Part of the problem is the low priority accorded to completing the self-evaluation reports by operations, coupled with the absence of any focal point within senior management to drive the process and deal with the problems.
No record is kept of how the self-evaluation results are actually used. They do not appear on the CDB website, but we were told that the findings are integrated into the following project designs. Hence we are somewhat unclear as to the utility of these reports at present. The situation is exacerbated by a rather confused image of evaluation: some operations staff consider OIE’s input (through validations or independent evaluations) to be sometimes over-critical, regulatory and adding little value; it is a threat rather than an opportunity for learning. Yet at the same time, evaluation is recognized as an integral part of result-based management.
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.
Some managers however seem to start changing the status quo. For example a revised and simplified template for producing project completion reports is being considered, and mid-term project reviews are expected to be more stringent in looking at monitoring plans and practices and tying disbursements to performance. In some cases we also learned of incentives being introduced to encourage project managers to complete their reports in a timelier manner. But much remains to be done and, since the OIE is no longer responsible for monitoring and project evaluations, there is a void that needs to be filled. It is up to line managers to drive this work forward.
To conclude, it is fair to say that in view of a number of “frustrations” between the OIE and operations, which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring data, the added value that evaluation might offer to the operations area is ill recognized. Moreover, the link between self-evaluation as the building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
140
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”88
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
88 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
141
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”89 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 90 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
89 CDB (2011) Evaluation Policy (p.2)90
142
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
143
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
144
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
145
mendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
146
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
147
APPENDICES
Appendix I - The External Review Mandate – Terms of Reference and Approach Paper
Appendix II -Review Approach, Data collection and Analysis, and Limitations
Appendix III – Overview of OIE Evaluation Practice
Appendix IV - List of Persons Interviewed
Appendix V - List of Documents Reviewed
Appendix VI- List of Topics used to guide interviews with members of CDB Board of Directors
Appendix VII - List of Topics used to guide interviews with CDB staff
148
Appendix III – Overview of OIE Evaluation Practice (prepared by the OIE in response to Reviewer’s request)
Caribbean Development Bank, Office of Independent Evaluation - OIE
Category Response
Percentage of projects subject to project (self-) evaluation
100% - Project Completion Reports (PCR)
Percentage of projects subject to validation by OIE
Approximately 40-50%
About 15 projects exit portfolio annually. Evaluation Policy calls for all PCR to be validated. However, OIE resources insufficient. Validation process reviewed in 2014. Now OAC (Board committee) selects a sample of 6-8 PCR for validation each year.
Percentage/number of projects subject to in-depth review by OIE
None – unless specifically requested by OAC
Due to limited resources, focus of OIE evaluation work programme is on PCR validations and high-level evaluations – including country strategy and programme evaluations (CSPE).
Number of high-level evaluations conducted by OIE (e.g. sector, thematic, geographic)
1-2 per year since 2011
Plan is 2-4 per year from 2016. This would include CSPE (1st planned for Q1 2016: Haiti)
Number of project impact evaluations conducted by OIE
None
OIE includes “impact questions” in high-level evaluations.
Number of project impact evaluations conducted by Bank staff or other non-OIE staff
OIE is not aware of any impact evaluation conducted by the Bank.
However, OIE provides technical support to the Basic Needs Trust Fund (BNTF) in its design of an M&E framework that entails impact evaluations.
Budget In USD mn: 0.78 in 2015; 0.82 in 2016. This is equivalent to about 2.5% of total CDB Administrative Budget.
75% of the budget is for Staff salaries (4 Professionals, 1 Support staff), leaving around USD 190,000 (in 2015) for other expenses, including consultants e.g. for external evaluations. Additional funding is accessed via the Special Development Fund (SDF). This varies according to type and scope of the evaluation, e.g. the ongoing SDF 6/7 Evaluation is SDF funded at USD 255,000.
Budget determined by Board, not separate from administrative budget.
SDF funding for evaluations is considered separately and subject to Bank internal approval process. SDF funding cannot be used to cover OIE expenses such as staff time or travel. Country eligibility for SDF funding is also a consideration. OIE expressed concerns about this funding track in respect to predictability, independence and eligibility limitations.
Head of OIE reports to Board, with administrative link to the President
Terms of appointment for Head
149
5 year term, renewable once. Appointed by the President with the agreement of the Board.
Right of Return for Head Not eligible for other staff positions.
Consultants as proportions of OIE budget
2015: 19% (USD 145,000)
Plus SDF funding. SDF funded evaluations are outsourced.
Last external evaluation (or peer review) of OIE
No external evaluation, though a review of the function was done in 2011, leading to the Evaluation Policy.
OIE External Review completed in April, 2016
Departments or special programmes supporting impact evaluation
None
150
Appendix IV – List of Persons Interviewed
Name Function relative to OIE Type interview
Mrs. Colleen Wainwright Member CDB Board of Directors (UK)
Face to face
Mrs. Cherianne Clarke Alternate MemberCDB Board of Directors (UK)
Face to face
Mrs. Jean McCardle Member CDB Board of Directors (Canada)
Face to face
Dr. Louis Woodroofe MemberCDB Board of Directors (Barbados)
Mr. A: de Brigard Former Member CDB Board of Directors
Skype interview
Mr. H. Illi Fromer Member CDB Board ofDirectors
Telephone interview
Mrs. Claudia Reyes Nieto Member CDB Board of Directors
Telephone interview
Mr. Bu Yu alternate DirectorCDB Board of Directors
Face to face
Mr. Michael Schroll(Barbados)
Head OIE
series of interviews viaSkype and face-to-face
Mr. Mark Clayton OIE Senior Evaluation Officer Focus GroupMrs. Egene Baccus Latchman OIE Evaluation OfficerMr. Everton Clinton OIE Evaluation OfficerMrs. Valerie Pilgrim OIE Evaluation Officer
Dr. Justin Ram CDB Director Economics Department
Face to face
Mr. Ian Durant CDB Deputy Director Economics Dept Face to faceDr. Wm Warren Smith CDB President
Joint interviewFace to face
Mrs. Yvette Lemonias-Seale CDB Vice President Corporate Services & Bank Secretariat
Mr. Denis Bergevin CDB Deputy DirectorInternal Audit
Face to face
Mr. Edward Greene CDB Division Chief, Technical Cooperation Division
Face to face
Mrs. Monica La Bennett CDB Deputy Director Corporate Planning Face to faceMrs. Patricia McKenzie CDB Vice President Operations Face to faceMs. Deidre Clarendon CDB Division Chief
Social Sector DivisionFace to face
Mrs. Cheryl Dixon CDB Co-ordinator, Environmental Sustainability Unit
Focus group
Mrs. Denise Noel- Debique CDB Gender Equality Advisor Mrs. Tessa Williams-Robertson CDB Head Renewable EnergyMrs. Klao Bell-Lewis CDB Head Corporate Communications Face to faceMr. Daniel Best CDB Director
Projects DepartmentFace to face
Mr. Carlyle Assue CDB Director Face to face
151
Finance Department
152
Appendix VI - Interview Guide: Members of CDB Board of Directors
Below are a list of themes that I should like to raise with you based on your experience and knowledge of the CDB’s independent evaluation function (Office of Independent
Evaluation).
In each case, I should be grateful if you could illustrate your responses with examples or help this Review by, wherever possible, sending me (or telling me where I can find) any
documents that could support your responses.
This guide is being sent to you in advance to help prepare our meeting. However, our interview will be conducted more in the style of a conversation. The following sub-questions will be used to GUIDE the interview. Please feel encouraged to raise any
additional issues that you feel we should take into account
On the governance and Independence of CDB’s evaluation functionWhat mechanisms are there in place to support its independence?
How satisfactory are the current arrangements in your opinion?
How is the balance between independence and the need for interaction with line management dealt with by the system? For example, what mechanisms exist to ensure that the OIE is kept up to date with decisions, policy / programme changes, other contextual changes etc that could have an affect on OIE evaluation studies / evaluation planning?
On the OIE’s Evaluation PolicyThe CDB’s Evaluation Policy was established in 2011. To what degree do you feel it is adequate? Still relevant?
What suggestions do you have for any improvements?
In your opinion, how adequate is the current quality assurance system for over viewing the evaluation function?
On the quality and credibility of evaluation studiesTo what degree do you believe the reports are fair and impartial?
Do you consider them to be of good quality? Are they credible?
Are you adequately consulted/involved on evaluations of interest to you?
On the relevance and usefulness of evaluations How well does the OIE engage with you / your committee during the preparation, implementation and reporting of an evaluation study to assure that it will be useful to the CDB?
How are the priorities set for the independent evaluations? What criteria are used? Are you satisfied with the current procedure?
153
When OIE evaluation studies are outsourced to external consultants, what criteria are used to make this decision?
How are the priorities for the OIE’s 3.year rolling work plan agreed? In your opinion, is the current plan adequate in terms of coverage and diversity?
In your opinion, do the evaluations address important and pressing programs and issues?
To what extent do you feel that the OIE’s evaluations integrate the cross-cutting theme such as gender, energy efficiency/renewable energy, climate change? What improvements might be made and how?
On the dissemination and uptake of evaluation findings and recommendationsTo what extent do you feel that evaluation findings are communicated to the CDB and its stakeholders in a
a) useful, b) constructive andc) timely manner?
Are evaluation recommendations useful? Realistic?
What mechanisms are in place to assure that evaluation results are taken into account in decision making and planning? What improvements do you feel could be made?
How have you used the findings from any evaluations? Examples?
To what degree do you feel that evaluation contributes to institutional learning? And what about to institutional accountability? Any examples?
What mechanisms are in place to ensure that knowledge from evaluation is accessible toCDB staff and other relevant stakeholders? Are the current arrangements satisfactory?
How satisfied are you with current arrangements? What expectations do you have for the future?
On resourcesHow is the OIE resourced financially and is this satisfactory?
What about the OIE staff, are all the important areas of expertise represented in the team?
On this Review of the Office of Independent EvaluationWhat are your expectations? What are you particularly hoping to learn from it?
Thank you very much for your cooperation and input
154
Appendix VII : Interview Pro-Forma – CDB Staff membersThis presents a list of the topics raised during interviews. It was used to guide the open-ended
discussion – this means that the sequence and exact wording of the questions may not necessarily have followed in this order or been asked in exactly this way.
Changeover to an Independent Evaluation Office? Expectations? Advantages and disadvantages??
Satisfaction with working relations between operations and the OIE from your perspective?
Process of dealing with the PCRs and CCRs? Advantages and limitations?
Quality and credibility of the validation process?
How are the self-evaluation reports used?
Credibility and Quality of OIE’s evaluation reports
Communication of self and OIE independent evaluations? To whom, in what way? Possible improvements?
External Review of the Office of Independent Evaluation
Caribbean Development Bank
April, 2016
155
Principal ReviewerMarlène Läubli Loud,
Review Panel MembersJohn Mayne
Bastiaan de Laat
156
Primary audiencesMembers of the Oversight Assurance Committee
Members of the Board of Directors
Independent Evaluation Office
Caribbean Development Bank Management and Staff
Review Panel Members
Marlène Läubli Loud (DPhil) is currently an independent consultant and trainer in public sector evaluation. She has worked with a range of organizations, small and big including the European Commission, the World Health Organisation, the United Nations Evaluation Group, the UK Employment Department, UK Health Promotion Agency (now merged and become NICE), and the English Nursing Board. She was head of the Research and Evaluation Unit at the Swiss Federal Office of Public Health for nearly twenty years where she gained much experience in evaluation management, and especially in the ways and means for improving the use and utility of evaluation in organisations. She continues to have a keen theoretical and practical interest in this area. Prior to this, she was an independent evaluator in the UK, specializing in the evaluation of developmental programmes in health and general education. She was also a research fellow at the Department of Education, University of Surrey and in the Social Science Faculty, University of Oxford, UK.
John Mayne (PhD) is an independent advisor on public sector performance. He has been working with a number of organizations and jurisdictions, including several agencies of the UN, the Challenge Program on Water and Food, the European Union, the Scottish Government, the United Nations Secretariat, the International Development Research Centre, the Asian Development Bank and several Canadian federal departments on results management, evaluation and accountability issues. Until 2004, he was at the Office of the Auditor General where he led efforts at developing practices for effective managing for results and performance reporting in the government of Canada, as well as leading the Office’s audit efforts in accountability and governance. Prior to 1995, John was with the Canadian Treasury Board Secretariat and Office of the Comptroller General. He has authored numerous articles and reports, and edited five books in the areas of program evaluation, public administration and performance monitoring. In 1989 and in 1995, he was awarded the Canadian Evaluation Society Award for Contribution to Evaluation in Canada. In 2006, he became a Canadian Evaluation Society Fellow.
Bastiaan de Laat (PhD) is Evaluation Expert and Team
157
Leader at the European Investment Bank (EIB) where over the past two of years he has been in charge of major evaluations in important areas such as Climate Action, SME support and Technical Assistance. He has a longstanding experience in evaluation as well as in foresight. Founder-director of the French subsidiary of the Technopolis Group (1998-2006) he led many evaluations for and provided policy advice to a great variety of local, national and international public bodies. He trained several hundreds of European Commission staff and national government officials in evaluation and designed monitoring and evaluation systems for various public organisations. Before joining the EIB he worked as Evaluator at the Council of Europe Development Bank. He has developed tools and performed programme, policy and regulatory evaluations, both ex ante and ex post, in a variety of fields. He has also made several academic contributions, most recently with articles on evaluation use and on the "Tricky Triangle", on the relationships between evaluator, evaluation commissioner and evaluand. In his private capacity, Bastiaan served as Secretary General of the European Evaluation Society and was recently elected Vice-President.
Acknowledgements The Review exercise could not have been possible without the support and commitment of the OAC, the OIE and the CDB. The exploratory discussions with the OIE staff, members of the Board of Directors as well as with the CDB management and staff provided great insight and were a valuable contribution to the Review.
We are indebted to the Head of OIE, Michael Schroll, and his team for their cooperation, insight, and readiness to provide us with any information requests. We are also grateful to them for their useful comments regarding the first draft of our report, and for their suggested improvements.
We are especially appreciative of OIE’s administrative assist, Denise xxxx, for her help in coordinating the interviews during our 10-day field study in Barbados and in providing us with all documents we requested.
158
AcronymsLIST OF ABBREVIATIONS
AMT Advisory Management TeamAPECAPAR
Audit and Post Evaluation CommitteeApproach PaperAppraisal Report
BOD Board of DirectorsBMCs Borrowing Member CountriesBNTF Basic Needs Trust FundCDB Caribbean Development BankCSPDAC
Country Strategy PaperDevelopment Assistance Committee
DFI Development Financial InstitutionED Economics DepartmentFI Financial InstitutionIRLMDB
Immediate Response LoanMultilateral Development Bank
mn millionM&E Monitoring and EvaluationMfDR Managing for Development ResultsOAC Oversight and Assurance CommitteeOIE Office of Independent EvaluationPAS Performance Assessment SystemPBG Policy-Based GrantPBL Policy-Based Loan PCR Project Completion Report PCVR Project Completion Validation ReportPPES Project Performance Evaluation SystemPPMS Portfolio Performance Management SystemSDF Special Development FundTA Technical Assistance WB World Bank
159
Preface Evaluation work at the Caribbean Development Bank (CDB) has been ongoing since the early 1990s, although initially it was mainly focused on the ex-post evaluation of projects. However, in 2011, the CDB reviewed its evaluation system to bring it up to date with the good practices of international development organisations. In December that year, it produced its comprehensive Evaluation Policy (December 2011) setting out the aim and objectives and guiding principles for CDB’s evaluation system.
The Policy provides for the establishment of the Office of Independent Evaluation (OIE). Its main objective is to provide “CDB’s Board of Directors, President, Advisory Management Team, CDB staff and other stakeholders and partners with timely, credible and evidence-based information on the relevance and performance of CDB’s projects, programs, policies and other development activities.” (Evaluation Policy, 2011, p. 1).
To oversee and assess good practice, the Evaluation Cooperation Group (ECG) for Multilateral Development Banks (MDBs) recommends that the MDBs’ evaluation system and independent evaluation units be the subject of a review on a regular basis. The aim here is to help the institutions adopt recognised evaluation standards and practices so that its policies may benefit from evidence-based assessments.
In mid-2014, a new Head of the OIE was appointed and, following an initial learning period, he called for a peer review of the evaluation system. Even though the OIE had only been in existence since 2012, it was considered timely to take stock of what had been done so far in order to tease out the priorities for the next 3-4 years.
It was originally anticipated that such an assessment could be done by the ECG as part of the OIE’s application for ECG membership. This did not prove possible, since the CDB’s operation is considered too small for such membership. A review was therefore commissioned to independent experts in evaluation who are knowledgeable and experienced in the management of internal evaluation units.
Main AimThe Review’s main aim is to provide the CDB’s Board of Directors with an independent assessment of the OIE and CDB’s evaluation system. The intention is to highlight the factors that help or hinder the OIE’s independence and performance in order to identify where improvements could be made. This report will be presented together with a Management Response to the CDB’s Oversight Assurance Committee and its Board of Directors at its meeting in May 2016. It is anticipated that an action plan will be drawn up on the basis of the Board’s decision on how to address the recommendations put forward.
Report StructureThe Review starts with some general background information about the CDB and the setting up of an independent evaluation function. It also sets out the reasons for an external review and why this was requested at this particular point in time. Part One also outlines the Review, which, is presented in more detail in Appendix II. Part Two provides the Review’s findings and conclusions according to each of the criteria presented. The Panels general conclusions and recommendations are the subject of Part Three.
The Panel is grateful for the complete freedom it was given to form its own opinions and to reach conclusions based on its analysis. The findings, conclusions and recommendations
presented in this paper are those of the Peer Review Panel members. The views of the CDB, are provided separately in the Management Response that accompanies this Report.
160
Executive Summary
161
Part One: Introduction and BackgroundIn order to understand the development of the Office of Independent Evaluation’s (OIE) work, a brief description of the CDB’s current reforms is needed. First, there has been a change over the last decade from the nature of the programmes the bank supports; for example it has become increasingly engaged in funding policy-based operations and social development issues. Similarly, there have been changes in the whole of the development field, which is grappling to deal with complex issues such as gender or climate change. To meet today’s challenges and ensure that its work practices reflect the international standards of Multilateral Development Banks (MDBs), the CDB has introduced a number of measures aimed at improving its effectiveness and efficiency. For example, in line with international standards for Management for Development Results (MfDR), it has introduced a Results Based Management Framework for organising and assessing its performance.
In 2011, the CDB commissioned an external consultancy to undertake an assessment of its evaluation function in order to develop a policy that took account of good practices within the international development community.91 The CDB’s Evaluation Policy (referred to hereafter as the Policy) is a direct response to that review; it reflects the standards and good practices of the Evaluation Cooperation Group (ECG) of the Multilateral Development Banks (MDB) as well as the evaluation principles and standards of many professional associations. Similarly, the Bank showed its commitment to having evaluation as a core function by establishing an independent evaluation unit that reports to the Board of Directors.It is responsible for assessing the Bank’s activities and interventions, but especially for drawing out the key lessons and recommendations for improving the Bank’s performance.
As such, the monitoring tasks formally under the responsibility of the Evaluation and Oversight Division (EOV) were handed over to the Bank’s Operations and Economic Divisions. The OIE, in its advisory capacity, is expected to provide the sareaoperations area with the necessary support manuals, tools and guidance; the OIE then validates the credibility and rigour of the self-evaluations.
In addition to the OIE, the Bank also set up other independent functions: internal audit, risk assessment and management, integrity, compliance and accountability. The mainstreaming of three cross-cutting themes (gender, energy and climate issues) into CDB’s work has also been initiated. At the same time, there are limited funds available as the CDB is working within a Board-sanctioned policy based on the principle of a zero real growth, which is in line with the budget policy of other MDBs.
In short, the bank has taken many important steps towards updating CDB’s management practices in line with other MDBs. However, the introduction of many innovations in parallel requires coordination and a shift in working practices and thinking. There is also the need to engage in different types of evaluation; evaluations that take into account cross-cutting themes and different levels of complexity. As such, whilst this review is particularly focused on the CDB’s Office of Independent Evaluation (OIE), its work and utility depend to a large degree on the development of other management practices and the degree to which evaluation is able to link to their work.
A full description of the Review’s mandate, approach, process and methods are provided in Appendices I and II. It was designed to address the following four key questions as set out in the appended Terms of Reference and Approach Paper:
91 Osvaldo Feinstein & Patrick G. Grasso, Consultants, May 2011 Consultancy to Review the Independence of the Evaluation and Oversight Division of the Caribbean Development Bank
162
To what degree is the Office of Independent Evaluation independent at the strategic, functional and operational levels? Which measures help or hinder such independence?
To what extent is the OIE achieving its 2 strategic objectives? (which are (1) the timely delivery of good quality evaluations and PCR Reviews and (2) strengthening capacity building, networking and communication) How useful are the OIE’s procedures and products towards this end?
How adequate are the financial and human resources of the OIE for carrying out its tasks and achieving its objectives?
How effective is OIE in relating with its internal partners to develop evaluation capacity?
This Review of the OIE is based on the recommended criteria of the Evaluation Cooperation Group for Multilateral Development Banks; governance and independence, credibility, use and transparency.
The data used for analysing and interpreting the findings relied on exploratory, semi structured interviews with OIE staff as well as with CDB senior and middle managers and members of its Board of Directors. Whilst much of the interview data was collected during a 10-day intensive, on-site visit to the Bank, the majority of the Board members were interviewed through Skype. The interview data was complemented by a review of a range of key documents including the Bank’s Evaluation Policy, various kinds of reports on, or about evaluation, the complete set of minutes of meetings between the OIE and the Oversight Assurance Commission92 and the subsequent chairman’s report to the Board for the study period 2012 to 2015, OIE staff biographies as well as a number of other organisations’ evaluation principles, good practices and standards. A full list can be found in the Appendices (Appendix V). Not least, the Reviewers have also drawn on their own knowledge and experience of evaluation management to complement data analysis and interpretation.
Scope and Limitations
The Panel was asked to concentrate on the 4-year period since the establishment of the OIE, January 2012 to December 2015, but more particularly on the changes introduced since the new Head of the OIE was appointed (June 2014 to December 2015).
It has mainly focussed on the strategic role of the OIE within the CDB as well as its functional and operational roles and responsibilities.
It was planned as a Review and not a fully-fledged evaluation; this was due to the limited time and resources available for the exercise as well as the fact that a ”light” review is in keeping with the spirit of the OIE’s Terms of Reference. The Review could not undertake any in-depth analysis of documents or consult with country level stakeholders or other external sources of expertise. Moreover, of the 29 people identified for interview, despite several reminders (by email or telephone) , the Panel were unable to either contact or secure the agreement of 5 of the 14 Board members, and 1 CDB senior manager. In light of this experience, as well as the time invested in securing the interviews “at a distance”, the planned on-line survey to follow-up on face-to-face interview data was abandoned.
We regret that in the time available, full justice could not be done to all the material provided to the Panel by the OIE. Nevertheless, the documentary review and interviews focussed on addressing the key questions, and we are therefore confident that the main issues raised in the Terms of Reference have been addressed in this report.92 The Audit and Post Evaluation Committee, now the Oversight Assurance Committee, is a Board Committee responsible for the oversight of evaluation and other key management functions.
163
164
Part Two: What the Review Found In the first place, the Panel should like to commend the CDB for its efforts to establish an independent evaluation function.. Similarly, in spite of some of the challenges raised in this Review, the current Head of OIE and his team are to be commended for their efforts in advancing evaluation in the right direction of the UNEG Norms and Standards and ECG guidelines on Good Practices. The Panel presents its findings and analysis in a spirit of constructive criticism, highlighting the strengths of the current situation as well as several challenges that need to be addressed in order to bring out the full value of evaluation to the CDB.
In this part of the report, we present our findings and conclusions relative to the following criteria used to assess and respond to the four TOR questions:
the evaluation policy governance independence the OIE strategy, practices and work programme usefulness of evaluation, evaluation use communicating evaluation results (transparency) adequacy of resources, and finally the working relationship between self and independent evaluation
The Evaluation PolicyThe CDB Board agreed an Evaluation Policy (the Policy) in December 2011. It sets out the guiding principles and provisions for the OIE. It also aims at guaranteeing the independent functioning of the Office of Independent Evaluation (OIE) by having it report to the Board of Directors through the Oversight Assurance Committee, OAC. However, the President retains oversight on administrative matters for management of day-to-day activities such as travel approval..
Generally speaking, the Policy reflects many of the ECG’s recommendations on evaluation independence and good practices. Similarly, the evaluation criteria for judging outcomes are the five developed by the DAC, that is relevance, effectiveness, efficiency, impact and sustainability. In general, the Policy is intended to maximize the strategic value, timeliness and the learning aspect of evaluation.
Yet in reality, the Policy provides a framework for what could be achieved under optimal conditions. It is overambitious in terms of what could be done with the current level of resources. For example undertaking the validation of all Project and Country Completion Reports as well as engaging in the full range of evaluation types undertaken within the MDBs have proven to be simply not feasible at this stage. (More on this later in the report.)
Many important tasks outlined in the evaluation policy have not been done so far by either the OAC or the OIE. For instance, the OAC has yet to produce an annual report on OIE’s performance and the OIE has yet to establish a database of evaluation lessons, recommendations, actions and management responses.
To conclude: The Evaluation Policy is a mission statement of what could be achieved in time with sufficient financial and human resourcing. It reflects the internationally recognized evaluation principles and standards, but is probably somewhat ambitious for the OIE to fully put into practice for a number of years
165
Governance IssuesOversight of the OIE is entrusted to a Committee of the Board of Directors (originally called the Audit and Post-Evaluation Committee, APEC, and now the Oversight Assurance Committee (OAC).The OIE reports to the Board through the OAC. There are 5 members, of which only 2 are located in Barbados.
The OAC meets 5 times per year, the day before Board meetings. It has oversight responsibility for external and internal audit, independent evaluation, risk management and integrity, compliance and accountability in relation to CDB’s work.
The OAC Chairperson prepares a very brief resume of the day before’s meeting to present to the Board for approval. The report generally covers progress, shortcomings and risks but is only a small part of the Board meeting so that generally there is little discussion; evaluation is only one of many items on the agenda. (We were told that the report to the Board averages approximately 10 minutes). Some of our interviewees could not recall any discussion about evaluation during Board meetings or remember reference being made to any evaluation report.
Given the breadth of its oversight responsibilities, there is now provision for the OAC to hire in consultants to provide it with technical expertise as needs be. Resources would have to be provided out of the CDB’s administrative budget. Another novelty is the provision to meet with the head of the OIE in an executive session at least once per year. So far, our understanding is that neither of these opportunities has yet been taken up by the OAC.
The major problem for the OAC is the volume of paperwork and length of individual documents received in parallel from the CDB and its independent offices, generally very shortly ahead of its meetings. Both Board and OAC members expressed their deep concern about the need for the more timely delivery of reports and background papers.The OAC members fear they are unable to do justice to their oversight responsibilities. Hence, based on the Panel’s review of the minutes and comments from the OIE, the meetings appear to be more formalisticbut with little in-depth discussion or systematic follow up on the recommendations, agreed actions or the lessons drawn, The “follow up on actions agreed” does not appear to be a systematic item on each OAC meeting’s agenda..93 Similarly any attempt to identify key messages for various stakeholders other than the CDB is not mentioned in the minutes or reports to the Board.
In response, the OIE has greatly improved the presentation of technical reports by summarising the main points in its “Brief Reports” (e.g. the Tax Administration and Tax Reform and Technical and Vocational Education and Training evaluation). This is commendable and certainly a step in the right direction although the Panel considers that they should have a sharper focus on the strategic issues (which are the end of the brief rather than the beginning), be condensed and be made more “reader friendly”.
The Panel was also surprised to find that despite expressions of support for rigorous evaluation and its importance to the CDB, the OAC do not appear to be taking any firm position with regard to the paucity of available data. OAC has been made aware of the data problems in the BMCs (e.g. lack of rigorous monitoring and statistical data and the consequent effect on the rigour of OIE’s evaluations) as well as the delays in the submission of self-evaluations and their validations, yet there appears to be no OAC attempt to deal with such problems e.g. exerting any pressure on the CDB or on the BMCs through their representatives on the Board.
To conclude: The OAC has an expressed interest in advancing the role of evaluation as a strategic tool for CDB management. However, it is not performing its oversight function with sufficient firmness to bring about any change to the problems raised through evaluations, especially with regard to data issues and reporting delays. More generally there is a lack of any systematic
93 At the APEC meeting in May 2012, it was agreed that the OIE would prepare a Management Action Record to highlight the follow up actions taken to the recommendations of all evaluation reports, every two years, with the first report presented to APEC at the March 2013 Board Meeting. There is no record of this having ever been done or of the APEC / OAC’s following up on such request.
166
report on “follow up of actions agreed” which could be particularly useful for tracking changes as a consequence of an evaluation and management’s response. The OIE could do better justice to its oversight responsibly if it were to receive all background documents systematically at least two weeks before its meetings. Moreover, the volume and length of documents received at any one time is considered to be overwhelming. The number and/or importance of agenda items competing for attention at any one session is an additional handicap.
the Panelss
the PanelssUnit has access to all needed information and information sources
Extent to which the evaluation unit has access to the organization’s
a) staff, records, and project sites;
b) co-financiers and other partners, clients; and
c) programs, activities, or entities it funds or sponsors
Complies –The available evidence suggests that there is no reason to doubt such access. But systematic and easily accessible documentation is lacking in the CDB; it is one of its weak points.. Delays in getting hold of the relevant documents can have consequences on the timeliness of evaluation studies
9) There appears to be a that is of concern to the Panels (More on this point under the heading self and independent evaluations.)The OIE is not regularly invited in any capacity to these meetings or given a copy of the agenda or minutes; the OIE is occasionally invited to attend in order to discuss an evaluation report or management feedback. Its observer status at meetingsor as a do not necessarily provide the same insight as to the dynamics of management actions and/or decisions.
. Extent to which the evaluation unit has control over:
a) staff hiring,
b) promotion, pay increases, and
c) firing, within a merit system
Partially complies - All OIE staff members are treated in the same way as other CDB staff. The Head has limited control over the hiring, firing or promotion of OIE staff.
actual or potential conflict of interestThe PanelIt must be s
The PanelisThe; this affects also of y, Work PracticesThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. therefore The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation pyramid94 are brought out in the remaining sections of this Review, not least given the limited resources available.
But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities. sTheThes before completing the final
94 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
167
version. However, p are only submitted to the CDB line and senior managers.Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the following the recommendations of professional good practices and standards on participative approaches; it has succeeded in , The Panelsstaff fromas well as There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. OIE doesarrange .sthe Panel also wishes tonewly appointed rsA was evidentthey expressed interest in In one case, interest was followed up in practice; can be improvedfostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that can offer can add value to understanding the strengths and weaknesses of such strategies. Tduring this transitional phase, Manual to guide and support the independent evaluation process.and operations staff sevaluation activities. ,oOIE’sjudginga . As such, theyare thatAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An is processed as follows;the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
SThisrItand Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
The rmade
- is still considered to be good practice to have the elaborated in the initial design documents the95 such as Developmental Evaluation (Patton, 201096)
-)
said abovePCHowever, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessonsthe Panelas done this ing
95 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.96 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press
168
on lessonsAlthough nothing has happened since, it is , sometimes indicate (Panel has already referred above to ’s lack of oversight in the use of evaluation.)
sThe Panelsevaluation work Moreover, the 2015 budget provides only US$2’000 for
communication – nothing of which is intended for outreach.Reviewerseither confusing or and
budgetedConsequently for 2015, theFigure 3: The MDB Evaluation Pyramid97
97 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
169
ys to be effective. A modest attempt has been made in 2015; OIE hased But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.man and financial resources to support its work
OIE’s Human Resources;
5eThree of the five were recruited from within the CDB. edfrom the Board that OIE
should embark on ee and for impact evaluations in particular,OIE’s ee Moreover, there
are many other designated OIE activities that should be recognised as valuable work; the
validations, building CDB and BMC evaluation capacity, providing supervision, advice,
knowledge management and brokerage as well as managing evaluation contracts, The
time needs of dealing with all of these may be underestimated in OIE’s budgets; all are
important for assuring best value from evaluation. The Panel is concerned that a demand
for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level
evaluations may undermine the importance and time needs of other essential
tasks.Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses.
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies
170
does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.Self-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes.types of evaluation y There appears to be little incentive to complete self-evaluations in a timelier manner.
.
; it is a threat rather than an opportunity for learning. Yis recognized as
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.a number of , which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring dataadded value that evaluation might offer to the operations area is ill recognized Moreover, the link between self-evaluation as the building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
171
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”98
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
98 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
172
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”99 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 100 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
99 CDB (2011) Evaluation Policy (p.2)100
173
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
174
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
175
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
176
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
Independence of the Office of Independent Evaluation (OIEIndependence is absolutely central to the integrity and trustworthiness of evaluation. It is an agreed requirement within the development agencies and in the evaluation community as a whole. In examining the issue of independence and good practice, reviewers are guided by the Evaluation Cooperation Group’s recommendations on good practices, the CDB’s Evaluation Policy and by the 2011 consultancy review of independence relative to the CDB’s evaluation and oversight division101. The appraisal is based on a comparison of the ECG’s recommendations on independence102 and the current OIE status.
OIE and Independence: Recommendations from the OECD Evaluation Cooperation Group (ECG)
The ECG’s considers the issue of independence according to three specific areas: organisational, or structural independence, behavioural, or functional independence and protection from outside interference, or operational independence.
Organizational independence, ensures that the evaluation unit and staff are protected against any influence or control by senior or line management, and have unrestricted access to all documents and information sources needed for conducting their evaluations. Also, that the scope of evaluations selected can cover all relevant aspects of their institution.
Behavioural independence, generally refers to the evaluation unit’s autonomy in selecting and conducting setting its work programme and in producing quality reports which can be delivered without management interference.
Protection from outside interference refers to the extent to which the evaluation function is autonomous in setting its priorities, and conducting its studies and processes and in reaching its judgments, and in managing its human and budget resources without management interference.
Conflict of interest safeguards refers to protection against staff conflict of interests be they current, immediate, future or prior professional and personal relationships and considerations or financial interests for which there should be provision in the institution’s human resource policies.
The OIE’s Independence in Practice
Organisational / structural independenceOn the whole, the Panel acknowledges and commends the efforts being made by the CDB to assure OIE’s organisational independence. The CDB’s Evaluation Policy provides for the OIE’s organisational independence from line management and the interview data suggests that there is also wide acceptance and acknowledgement of why the OIE should have such independent status. Table 1 below provides our overall assessment of this aspect of OIE’s independence when compared with ECG recommendations. 103
101 Osvaldo Feinstein & Patrick G. Grasso, Consultants, May 2011 Consultancy to Review the Independence of the Evaluation and Oversight Division of the Caribbean Development Bank102 ECG 2014 Evaluation Good Practice Standards, Template for Assessing the Independence of Evaluation Organizations, Annexe II.1 103 Based on ECG (2014) Template for Assessing the Independence of Evaluation Organizations, Evaluation Good
Practice Standards, Annexe II.1
177
Table 1: OIE organisational independence compared with ECG recommendations
Aspects Indicators CDB Evaluation Policy (EP) and Practice
The structure and role of evaluation unit
Whether the evaluation unit has a mandate statement that makes clear its scope of responsibility extends to all operations of the organization, and that its reporting line, staff, budget and functions are organizationally independent from the organization’s operational, policy, and strategy departments and related decision-making
Partially Complies The Policy is broad enough to cover the full range of MDB type of evaluations. However in practice this would not be possible without additional human and budget resources
The unit is accountable to, and reports evaluation results to, the head or deputy head of the organization or its governing Board
Whether there is a direct reporting relationship between the unit, and
a) the Management, and/or
b) Board or
c) relevant Board Committee, of the institution
Complies - OIE reports to the Board of Directors (BoD) through its Oversight Assurance Committee (OAC)
The unit is located organizationally outside the staff or line management function of the program, activity or entity being evaluated
The unit’s position in the organization relative to the program, activity or entity being evaluated
Complies - The OIE is located outside, and is therefore independent of CDB line management
The unit reports regularly to the larger organization’s audit committee or other oversight body
Reporting relationship and frequency of reporting to the oversight body
Complies - The OIE reports x 5 per year to the OAC . Board approval for an additional executive meeting between the Head of the OIE and the OAC at least once per year was given in October 2015
The unit is sufficiently removed from political pressures to be able to report findings without fear of repercussions
Extent to which the evaluation unit and its staff are not accountable to political authorities, and are insulated from participation in political activities
Complies
Unit staffers are protected by a personnel system in which compensation, training, tenure and advancement are based on merit
Extent to which a merit system covering compensation, training, tenure and advancement is in place and enforced
Partially Complies - with CDB human resource policy. However the skill needs of OIE staff ought to be regularly reviewed in light of its move towards higher-level evaluations. Appraisal of skill needs and hiring of relevant staff should be completely under the authority of the Head of Evaluation. This is not sufficiently clear in the Policy or other documents we reviewed.
Unit has access to all needed information and information sources
Extent to which the evaluation unit has access to the organization’s
a) staff, records, and project sites;
b) co-financiers and other partners, clients; and
Complies –The available evidence suggests that there is no reason to doubt such access. But systematic and easily accessible documentation is lacking in the CDB; it is one of its weak points.. Delays in getting hold of the relevant documents can have consequences on the timeliness of
178
Aspects Indicators CDB Evaluation Policy (EP) and Practice
c) programs, activities, or entities it funds or sponsors
evaluation studies
However, independence should not mean isolation: There appears to be a detachment between the OIE and CDB that is of concern to the Panel; on the one hand, between the OIE and operations staff, and (2) on the other, in terms of the structural arrangements between the OIE and senior management.
10) In agreeing for the OIE to concentrate on strategic and thematic, in-depth evaluations, responsibility for project monitoring and evaluation were given over to operations. The division is clear and respected. However, it has its drawbacks. With the OIE no longer systematically involved at the front-end of project design, the monitoring data needs are likely to be poorly defined. Weak monitoring data will contribute to weaker evaluations. (More on this point under the heading self and independent evaluations.)
In the reviewers’ opinion, it is a common misunderstanding to assume that providing evaluator advice on monitoring and evaluation data will comprise evaluator independence. On the contrary, evaluation input into project design is essential to assure that the logic, indicators and data needs are addressed so that at some future point in time an evaluation of the achievements can be empirically grounded.
This is not to say that the OIE no longer has any influence at the front-end design stage; it has merely shifted the point of focus. The OIE is now systematically providing such input more generally to the corporate planning teams for the tools and systems they are developing to support the MfDR framework. The monitoring data for projects and their implementation should be improved once the Project Performance Evaluation System (PPES) and the Portfolio Performance Management System (PPMS) are updated and operational.
11) In the second place, the OIE has limited formal access to the Advisory Management Team (AMT) weekly meetings where the President and senior management gather to exchange up-to-date information on the dynamics of CDB policy and practice. The OIE is not regularly invited in any capacity to these meetings or given a copy of the agenda or minutes; the OIE is occasionally invited to attend in order to discuss an evaluation report or management feedback. For the OIE, this means that it is unlikely to pick up on the ‘when’ and ‘what’ of key decisional issues or provide input into the discussion based on evaluative information. Its observer status at Loans Committee meetings, or as a participant informer at the OAC and BoD meetings and discussions do not necessarily provide the same insight as to the dynamics of management actions and/or decisions. .
To respond to this situation, the President has agreed to meet regularly with the Head of the OIE in order to keep him up to date with CDB strategic thinking. This is a welcomed change.
OIE Independence and Behavioural Issues The Panel has concerns about some behavioural issues. For example, through both the interviews and documentary review, we learned of considerable delays in processing both the independent evaluation reports as well as OIE’s validation of the CDB’s self-evaluations. Delays are generally due to receiving feedback on the independent reports from first, the relevant operational department, then from the AMT, and then on providing the OIE with a management response that is initially drafted by operations staff before being reviewed by the AMT. (OIE reports cannot be submitted to the OAC without the relevant management response). This two-layer process for preparing submissions to the Board is inefficient and could potentially be a
179
threat to evaluation’s independence in the future by delaying OIE’s timely reporting to the OAC.
OIE validations of the CDB self-evaluations are also submitted to the OAC, but it is in both sides’ interest to clear up any misunderstandings beforehand. Despite attempts to improve the timeframe for completing these validations, delays are more the norm than the exception. Table 2 below summarises our assessment of the behavioural aspects of independence.
Table 2: OIE and Behavioural Independence
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Ability and willingness to issue strong, high quality, and uncompromising reports
Extent to which the evaluation unit:
a) has issued high quality reports that invite public scrutiny (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk) of the lessons from the organization’s programs and activities;
b) proposes standards for performance that are in advance of those in current use by the organization; and
c) critiques the outcomes of the organization’s programs, activities and entities
Partially complies – paucity of data and documentation sometimes hinder the quality of reports. The OIE emphasizes the learning part of evaluation, and is cautious in its criticism recognising that management is going through a transitory stage and can still be overly defensive.
Ability to report candidly
Extent to which the organization’s mandate provides that the evaluation unit transmits its reports to the Management/Board after review and comment by relevant corporate units but without management-imposed restrictions on their scope and comments
Partially complies - as sometimes reporting to the Board is compromised by delays in the review/comment process between the OIE and the CDB. Any delay with the production of a Management Response will also mean that submitting a report to the Board in a timely manner is impaired since the two have to be submitted together.
Transparency in the reporting of evaluation findings
Extent to which the organization’s disclosure rules permit the evaluation unit to report significant findings to concerned stakeholders, both internal and external (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk).
Who determines evaluation unit’s disclosure policy and procedures: Board, relevant committee, or management.
Partially complies - The OIE’s conforms to the CDB’s disclosure policy. However, the dissemination of evaluation findings appears to be currently restricted to website publication and reports to the Board. A more targeted communication strategy to include other key stakeholders, e.g. project implementers in the BMCs should be developed and put in place.
Self-selection of items for work program
Procedures for selection of work program items are chosen, through systematic or purposive means, by the evaluation organization; consultation on work program with Management and Board
Complies - The OIE also ensures that its work program is drawn up after consultation with both CDB Management and Board to seek their input on relevant topics and themes.
Protection of administrative budget, and other budget
Line item of administrative budget for evaluation determined in accordance with a clear policy parameter, and
Partially complies - The administrative budget for supporting OIE work is protected. Access to additional sources of
180
Aspects Indicators CDB Evaluation Policy (EP) and Practice
sources, for evaluation function
preserved at an indicated level or proportion; access to additional sources of funding with only formal review of content of submissions
funding is possible if well argued and justified. But the approval process is complex and inefficient. (See Figure 1 below)
OIE and Protection from External influence or interference
Our overall assessment is provided in Table 3 below. The OIE’s independence in the design, conduct and content of its evaluations does not appear to be subjected to any external interference. But securing funding from any sources outside the OIE’s administrative budget, i.e. from the Social Development Fund, is an unduly complex and long process. As such we consider that the current funding process can affect the OIE’s choice with regard to the type of evaluations it can undertake. (See Figures 1 and 2 below)
Table 3: OIE and its Independence from External influence or interference
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Proper design and execution of an evaluation
Extent to which the evaluation unit is able to determine the design, scope, timing and conduct of evaluations without Management interference
Complies – however within limits of restricted human and financial resources available
Evaluation study funding
Extent to which the evaluation unit is unimpeded by restrictions on funds or other resources that would adversely affect its ability to carry out its responsibilities
Partially Complies - OIE must work within the limits of the agreed administrative budget wherever possible. If additional resources are needed for studies it must seek alternative funds elsewhere. The budget limitations can have an affect on the type of evaluations undertaken and therefore its independence in terms of choice.
Judgments made by the evaluators
Extent to which the evaluator’s judgment as to the appropriate content of a report is not subject to overruling or influence by an external authority
Complies – the evidence available suggests that the Board and Management accept the evaluators’ independent interpretation and conclusions Management responses are agreed to be the accepted place to raise any difference of opinion.
Evaluation unit head hiring/firing, term of office, performance review and compensation
Mandate or equivalent document specifies procedures for the
a) hiring, firing,
b) term of office,
c) performance review, and d). compensation of the evaluation unit head that ensure independence from operational management
Complies – the Head of OIE is appointed by the CDB President in agreement with the OAC for a 5 year period which is renewable x 1. The Head could be removed from Office by the President or the Board but only with the agreement of both parties.
However the Head reports to the President for all administrative and personnel matters. Even though this was not recommended in the Osvaldo Feinstein & Patrick G. Grasso report on Independence in 2011, the BoD accepted CDB’s reasons for keeping this arrangement. (e.g.most OAC members are non residents and cannot oversee day-to-day work)
. Extent to which the evaluation unit has control over:
a) staff hiring,
Partially complies - All OIE staff members are treated in the same way as other CDB staff. The Head has limited control over the hiring, firing or promotion of OIE staff.
181
Aspects Indicators CDB Evaluation Policy (EP) and Practice
b) promotion, pay increases, and
c) firing, within a merit system
Continued staff employment
Extent to which the evaluator’s continued employment is based only on reasons related to job performance, competency or the need for evaluator services
Partially complies - Whilst the EP is clear about procedures for hiring, firing and promotion, all of which must conform with CDB human resource policy, there is nothing mentioned about any difference of opinion between the CDB and the Head of the OIE with regard to continued staff employment subject to changes in the level of technical or interpersonal competencies needed to meet new demands.
Avoidance of Financial, Personal or Professional conflicts of interest
This particular aspect refers to the organisation’s Human Resources Policy; there must be provisions in place to protect against actual or potential conflict of interest. The Panel requested via the OIE, to have evidence from human resources on any such provisions but did not receive an answer. It must be assumes that this aspect of independence, past or present, does indeed form part of normal CDB Human Resource Policies
To conclude: The Panel is impressed with the measures CDB has taken to assure the organisational independence of the OIE. Its independent status is accepted and respected by senior and line management. The OIE’s budget is not independent from the overall CDB administrative budget; this affects its choice of evaluation types or approaches. Some of the behavioural issues affecting independence were also of concern, especially due to the delays in the exchange of documents, between the OIE and operations departments, which has a direct effect on timely reporting to the OAC. As for protection from outside interference, our concerns are largely to do with OIE’s independence over staffing issue; there are potential loopholes in current arrangements that could undermine OIE’s autonomy over its staff.
OIE’s Strategy, Work Practices and Work ProgrammeThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. Much of the period 2012 to 2015 has therefore been taken up with preparing OIE’s shift in focus from project-based evaluations to the high-level thematic and in-depth strategic studies. This has meant adopting a three-way approach; (1) for self-evaluations, reducing its time input to support the process and (2) for independent evaluations, taking stock of the gaps in coverage and expertise, and (3) networking to share experiences with centres of expertise and align OIE with international practices. In addition, amongst other duties, it has been supporting the development of MfDR tools and systems such as the Project Performance Assessment System by providing advice and input on programme logic and monitoring needs. The OIE plans to conduct 2-4 high-level studies per year from 2016. The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the
182
time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation pyramid104 are brought out in the remaining sections of this Review, not least given the limited resources available.
To conclude: The OIE has made a first step in proposing a strategy for establishing itself as an independent evaluation resource. But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities.
The Value / Usefulness of OIE’s Independent EvaluationsEvaluation is a powerful tool that can provide useful, evidence-based information to help inform and influence policy and practice. But useful evaluations depend not only on the evaluators’ skills, but on several other important factors as well; 1) on planning evaluations to be relevant to the priorities of the organisation’s work and for their results to be delivered in time to be useful; on the degree of 2) consultation and ultimately ownership by those who seek evaluative information; on the 3) tools used to support the evaluation process per se; and on the 4) credibility and quality of the evaluation products105.
1. Planning relevant and timely evaluationsThe OIE is now working on a 3 year rolling work plan that sets out the broad areas for enquiry. So far, there are no agreed criteria for making the selection of the specific topics for independent evaluation, although the priorities tend to reflect those of the CDB’s strategic plan. Nevertheless decision-making is rather arbitrary based on a process of dialogue between the OIE and the CDB and the OIE and the Board.
One of the OIE’s two objectives for 2015 therefore, was to define a work plan and agree priorities based on an approach that is “utilisation-focused”. This means that the studies are selected and planned to be relevant and useful to the organisation’s needs.
The OIE has achieved this objective with respect to its latest studies, which concerns the Social Development Fund (SDF) Multicycle 6&7 Evaluation, the Haiti Country Strategy evaluation and the evaluation of the CDB’s Policy Based Operations. Each of these three have been planned to deliver their results in time to provide the CDB Board of Directors with relevant information for negotiating the next round of funding. In spite of some delays due to a myriad of reasons, not least to the extra effort needed to secure essential data, the studies are expected to deliver on time.
The processes for agreeing OIE’s work plan and specific evaluations on the one hand, and, in securing alternative funding on the other, are shown in Figure 1 below. The Panel was surprised at learning how bureaucratic (the internal approval process), and inefficient (in view of the time it takes) the process seems to be. The concern here is that such a process could possibly pose a threat to assuring the Board of “timely studies.”
Figure 1: Selection of Evaluation Topics and Funding Source
104 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).105 These aspects reflect the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
Consultation with CDB Operations and OAC/Board for selection of
evaluation topic
3-year Work Programme and Budget (approved by Board)
Annual OIE report and work plan
submission to OAC
183
2. Consultation and ownership“The credibility of evaluations depends to some degree on whether and how the organization’s approach
to evaluation fosters partnership and helps build ownership and capacity in developing countries.”
(ECG good practices)
Internal review of Approach Paper
Specific Evaluation Study Design and Budgeting
OIE Draft Terms of Reference / Approach
Paper
Detailed ToR or Final Approach Paper if sufficiently detailed.
Finalise Approach Paper and submit to OAC/Board
Final Approach Paper
OAC ApprovalOAC minutes
Paper
Funding Track
Final Approach Paper/ToR
Board approval necessary If above USD
150,000
Board notification only if USD 150,000 or
below
Board Approval
Board Paper
OIE – Selection of consultants (if any) contracting
OIE Admin Budget or …
… SDF
Prepare TA Paper (content similar to Approach Paper but different
format.
TA Paper
Approval – Internal Loans Committee
OIE – Selection of consultants (if any)
contracting
184
The OIE engages with the OAC, CDB senior management and operations for agreeing its 3-year work plan and then for selecting the specific topics and themes. It also discusses the evaluation approach paper (design and implementation plan) with the CDB and OAC before completing the final version. However, preliminary and final drafts of the report are only submitted to the CDB line and senior managers for comment and factual errors. Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the results and their implications. Discussions with the OAC are more limited due to the overburdened agenda of OAC and Board meetings, as previously discussed.
In short, the OIE is to be commended for following the recommendations of professional good practices and standards on participative approaches; it has succeeded in having introduced a modus operandi that involves the key players in the selection of evaluation topics, the evaluation designs and their results. Figure 2 below provides an overview of the evaluation implementation and stakeholder engagement processes.
Figure 2: Evaluation Study Implementation and Feedback Loops
Arrangement AFully outsourced / external
consultants; oversight by OIE
Preparations:Detailed evaluation plan (incl tools,
timeline, etc.) and logistics
Production of Inception Report / Approach Paper
Arrangement BConducted by OIE
staff
Arrangement CJointly: external
consultants and OIE
Terms of Reference
Prepares Inception Report /
Approach Paper
Presentation/workshop:Interim findings and conclusions for immediate feedback and validation
Data Collection and Analysis
OIE
Summary and ppt for workshop presentation
and discussion with CDBSubmission of Draft Final
Report to OIE
Final OIE approved report to CDB Senior Management for Management Response
Board notification only if USD 150,000 or
below
Draft Final Report
Review loops – OIE and CDB (potentially also BMC)
Feedback to evaluation lead
Submission of Final Report to
OIE
185
9.10.
Notes to Figure 2
9. The OIE informed the Panel that this is an abbreviated version as there are e.g. additional steps (secondary processes) when evaluations are procured (tendering or single source), when there are additional review loops and updates to OAC etc.
10. OAC may also decide to return the report to OIE, the Panel were informed, or demand from Management specific actions based on the report.
This process is engaging and appears to have secured senior management and OAC interest and buy-in as witnessed in the latest studies. But there is the downside too! The process takes much time and, in our view, is partly unnecessary. The Panel appreciates that staff from operations as well as the AMT may both want to confer on an appropriate management response, but this should not be the case for reviewing an independent report for factual errors. The two-phase approach seems somewhat inefficient and unnecessary in our opinion.
Contact between the OIE, the CDB and/or the OAC during the actual study implementation is most often restricted to the occasional progress report, particularly when studies run behind time. There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. The OIE does, however, arrange discussions for reflecting on emerging findings, but we are not sure of how systematic this feedback loop is.
More generally speaking, outside of an evaluation study, the OIE has limited dealings with operations. The OIE has an advisory role in providing them with help, particularly with providing training, guidelines and tools to support self-evaluations. We are nevertheless concerned about the seeming distance between these two and how this has affected the perceived value of evaluation. (For further on this point, please see the section below on “Self- and Independent Evaluations”)
But the Panel also wishes to stress that this is not the case for newly appointed senior managers. A much more open attitude to evaluation and appreciation of its potential value was evident;
Prepare for disclosure and dissemination
Final Report
Final Report and Management Response submitted to
OAC/BoardFinal Report and
Mgt. Resp.
Management Response
OIE ApprovalFinal Report and Management Response considered by CDB
AMT
OAC/Board endorsed
186
they expressed interest in drawing out important lessons on what works, how, for whom, and under what conditions. In one case, interest was followed up in practice; the OIE was recently invited by a senior manager to share evaluative knowledge and experience with his staff regarding policy based operations.
Certainly, we can say that overall, the key stakeholders within the CDB are adequately integrated into the evaluation process as to foster their buy-in and ownership. But more generally, we feel that the utility of independent evaluations can be improved by fostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that evaluation can offer can add value to understanding the strengths and weaknesses of such strategies. This however cannot be done overnight and takes a long time.
3. Tools to support the evaluation processSo far, during this transitional phase, the OIE has mainly focussed on improving the tools to support the operations areas’ self-evaluations. This has left the OIE with little time to produce the checklists or tools to support its own studies. There are plans to develop an OIE Manual to guide and support the independent evaluation process. Such plans should be encouraged, as these documents will form a very important part of training, particularly for newcomers to the OIE team.
In the meantime, the OIE and operations staff refers to the Performance Assessment System (PAS) Manuals for evaluation activities. The manuals are based on DAC criteria and ECG principles. Much emphasis is given to the rating system and how and what should be rated. However we find them lengthy, unwieldy and overcomplicated. Moreover, such manuals should be used for reference, but cannot and should not replace first-hand training in how to plan, conduct and manage the evaluation process.
Quality Assessment (QA) and Quality at Entry (QaE)
There was a transition period between 2012 and 2014 to establish the OIE. Work on the PAS, QaE, PCRs, ARPP, which had started earlier, was therefore completed after OIE came into existence, but it effectively had no formal ‘home’ in operations. The Panel was told that there had been some discussions about creating a Quality Assurance unit within CDB (OPS) but the current status is unclear.
The QaE Guidance Questionnaire was developed before and completed by the OIE. It was used to assess the documents that came across to the OIE for comments at the Review Stage. The results were then sent to the Portfolio Manager/Project Coordinator indicating any gaps/issues that needed to be addressed or clarified. QaE Guidance Questionnaires were developed for all the Bank’s lending products, CSP and to assess the quality of supervision.
After the QaE was launched bank wide, several operations officers saw the merit in using the QaE Guidance Questionnaire in the field and adopted it as a tool for their use during the appraisal mission in order to cross check and test their data collection and analysis.
OIE’s use of the QaE was discontinued in 2014 due to limited resources and a stronger focus on evaluations. It still sometimes comments on specific appraisals, but very selectively.
Both QaE and QaS (quality at supervision) are also addressed in the PAS Manuals. In addition the QaE and PAS have been incorporated in Volume 2 of the Operations Manual OPPM.
The Review Panel assessed the QaE forms. They are relatively standard, adapted to the specificities of the CDB. They contribute to judging a project’s expected quality in a relatively objective way. As such, they are are helpful, as a benchmark, in the ex-post assessment of projects.
The Panel considers that the lack of an established Quality Unit in the CDB (and independent from OIE) is a weakness that should be addressed in the near future.
187
4. Credibility and Quality of Evaluation ProductsAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An independent evaluation is processed as follows; the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
Since 2012, the OIE has produced a range of studies and approach papers. This review is based on those listed below as provided by the OIE, and cover the period from May 2012 to December 2015. It includes 3 evaluations (in blue), 4 Assessment studies (in brown) 14 validations of self-evaluations (in green) and 3 Approach Papers (in purple) for upcoming evaluations. These are listed below in Table 4.
Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
Board Meeting
Date Type / Topic
251 May 2012 Ex-Post Evaluation Report on Road Improvement and Maintenance Project, Nevis -St. Kitts and Nevis.
Validation of Project Completion Report on Sites and Services – Grenada. Assessment of Effectiveness of Implementation of Poverty Reduction
Strategy 2004-09.253 Oct. 2012 Assessment of Extent and Effectiveness of Mainstreaming Environment,
Climate Change, Disaster Management at CDB.254 Dec. 2012 Assessment of the Implementation Effectiveness of the Gender Equality
Policy and Operational Strategy of the Caribbean Development Bank. Validation of Project Completion Report on Enhancement of Technical and
Vocational Education and Training – Belize. Validation of Project Completion Report on Fourth Road (Northern Coastal
Highway Improvement Section 1 of Segment II) Project – Jamaica. Assessment of the Effectiveness of the Policy-based Lending Instrument.
256 May 2013 Validation of Project Completion Report on Expansion of Grantley Adams International Airport – Barbados.
Validation of Project Completion Report on Fifth Water Supply Project – Saint Lucia.
261 May 2014 Validation of Project Completion Report on Immediate Response Loan, Tropical Storm Gustav, Jamaica.
Validation of Project Completion Report on Social Investment Fund, Jamaica.
Validation of Project Completion Report on Disaster Mitigation and Restoration – Rockfall and Landslip, Grenada.
263 Oct. 2014 Validation of Project Completion Report on Basic Education Project – Antigua and Barbuda
263 Oct. 2014 Approach Paper for SDF 6 & 7 Multicycle Evaluation
264 Dec. 2014 Validation of Project Completion Report on Policy-Based Loan – Anguilla
188
Validation of Project Completion Report on Immediate Response Loan - Tropical Storm Arthur – Belize.
Evaluation of Technical Assistance Interventions of the Caribbean Development Bank Related To Tax Administration and Tax Reform in The Borrowing Member Countries 2005-2012.
265 March
2015
Approach Paper for the Evaluation of Policy Based Operations
266 May 2015 Validation of Project Completion Report on Upgrading of Ecotourism Sites – Dominica
The Evaluation of the Caribbean Development Bank’s Intervention in Technical and Vocational Education and Training (1990-2012)
267 July 2015 Validation of Project Completion Report on The Belize Social Investment Fund I Project − Belize
268 Oct.2015 Approach Paper Country Strategy and Programme Evaluation, Haiti
The review and analysis of these documents is based on the UNEG Quality Checklist for Evaluation Reports (http://www.uneval.org/document/detail/607) as well as on ECG guidance (Big Book on Good Practice Standards).
Approach Papers
Three Approach Papers (APs) were made available to the panel (see Table [ref] above). An AP describes the rationale for the evaluation, the background to the topic evaluated, the evaluation framework (criteria and questions) and approach. It also describes the team and provides an initial planning. Being the first main deliverable of OIE’s evaluation process, APs are the starting point and therefore a major determining element in the roll-out of each evaluation. Therefore APs “have to get it right”.
The APs examined are clearly written, well-structured and of reasonable length.106 We were surprised to find, however, that they do not make explicit the objectives of the evaluated intervention(s), e.g., through a clear objective tree, or through an explicit theory of change, intervention logic or logframe. Whilst one of the APs contains, in an appendix, a results framework for the evaluation, the results framework for the intervention (PBO) itself is lacking.
Inception reports
Only one Inception Report was given to the Panel for review (SDF 6&7). This gives an in-depth description of the evaluated programme and provides a clear Theory of Change. It is good practice that this is established after a pilot field mission, which helps to amend the initial AP on the basis of field observations and sharpen the evaluation questions if needed.
However, it is still considered to be good practice to have the Theory of Change elaborated in the initial design documents . This would facilitate OIE evaluations after project completion. Establishing the Theory of Change of any intervention would be included in the QaE form more explicitly, to be developed between the Quality unit referred to above, and OIE.
Evaluations and Assessments
Three evaluations and four assessment reports completed during the review period were considered. Assessments are similar to evaluations but have a narrower scope; they focus on a limited set of aspects or judgment criteria, mainly effectiveness, i.e. achievement of objectives.
106 Opportunities remain of course to be more concise and to move parts to appendices, e.g., detailed descriptions of the evaluation team or part of the description of the evaluated intervention.
189
Evaluations generally base their judgment on the internationally recognised DAC criteria as well as aspects of the CDB and BMC’s management of the intervention.
In general, these reports are of reasonable quality. In the main, they explain the evaluated object107 and provide evaluation objectives. The findings are organised around the evaluation criteria or questions detailed in the scope and objectives section of the report. They are based on evidence derived from data collection and analysis methods as described in the methodology section. The reports tend to dwell on the limitations that the evaluation encountered, but without becoming defensive. In one case (PBL Assessment) the report starts with a summary of the reviews on the topic done by other MDBs. This was a pleasant surprise and indeed a good practice that could well be adopted in future evaluations too.
However, the reports also show several significant weaknesses:
- Reports do not always provide clear (reconstructed) intervention logics or theories of change for the intervention(s) evaluated.108 Evaluation criteria and questions are defined at a fairly general level. They are translated into more precise “research questions” (in an “Evaluation Design Matrix”, for each project for each criterion). However, it is unclear how these questions relate to the intervention logic (as this is not made explicit). This may be done in inception reports (of which, as noted above, only one was available for review), but should be done also in the final reports.
- The reports do not describe the link from the evaluation questions to the answers, how the evaluation judgments are made and how these ultimately transform into ratings for each criterion and each project. In other words, the explanation provided in the evaluation frameworks is inadequate. The “evaluation design matrix” currently used does not provide sufficient insight into how ultimately an intervention’s performance is judged.109 Links between findings, conclusions and recommendations could be improved by making this more explicit. In other words, reports should include the story on how the evaluand is credibly linked to any observed outcomes and impacts, and should be clear on how causal claims are made.
- With the exception of the PBL Assessment, reports are lengthy and detailed. One reason for this is an over-emphasis on ratings. Their detailed discussion, project by project, criterion by criterion, occupies a very prominent position in the evaluation reports’ main body of text. Although ratings are traditionally an important element in evaluations of MDBs, too strong an emphasis can be tedious and may distract the reader from the real lessons to be drawn. The detailed discussion of ratings, and their evidence base, would be better placed in an Appendix, with a brief summary in the main report. This would help give the lessons and recommendations a more prominent position than is now the case. This would also help make the evaluation reports not only shorter but also more interesting to read; this could help add value to evaluation’s image within the organisation.
- The reviewers feel that the OIE evaluations tend to over-emphasise objective-based evaluation110 and the DAC criteria to the exclusions of considering other evaluation
107 Sometimes in great length: for instance with the SDF 6&7 multicycle evaluation report it is only at page 30 that we find the beginning of the report on findings…108 Again with the SDF 6&7 evaluation, it is said to be guided by a “Logic Model” which is not explained.109 Marlène: I moreover have the idea that the methodology (often described as “visits”) is based on interviews and little hard evidence. Any view on this?.JM: My “interview-based evaluations”!!110 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.
190
approaches such as Developmental Evaluation (Patton, 2010111); evaluation should be case specific and answer the actual information needs of managers and other decisions makers rather than always concentrating on final performance.
- Related to the previous point (and again with the exception of the PBL Assessment) executive summaries (approximately 8 pages) are too long. For the evaluation report to increase potential impact, they would need to be reduced to 2 to 3 pages and be more focused; again this could be done by dwelling less on the individual ratings of projects and more on key findings, lessons and conclusions. More generally, reports could be better adapted to the needs of the different audiences. Although not strictly limited to evaluations, The Health Evidence Network Reports112 are a model that could be adapted for evaluation reporting purposes; they are specifically geared towards addressing policy and decision-making.
- The “Recommendations to BMCs” are an interesting feature of the reports, (although we are unsure to what degree such recommendations could be effectively followed up by OIE or the Bank, but certainly could taken up with BMC Board members.
- Reports (e.g. the evaluation report on Technical Assistance) focus much on technical problems that were encountered during the evaluation. Although these are important issues, again to improve the report’s flow and “readability” this section would be better placed in the Appendix. What counts is the story of the intervention, not the story of the evaluation (see “Limitations” section in the TA report for instance)
OIE Validations of Project and Country Strategy Programme Completion Reports (referred to globally as PCRs hereafter)
As said above, the OIE has the mandate to validate the Project and Economic departments PCRs and CSPCRs. However, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
The validations tend to repeat the different items reported in the PCRs and then provide extensive comment on each. The PCVRs go into great depth and detail, which makes the documents rich and complete. This is their strength – but also their weakness. The depth and level of detail, as well as the repetitions from the original PCRs, makes PCVRs (overly) lengthy (20-40 pages) and difficult to read. The OIE reported spending approximately 27.2% of its time on validating PCRs in 2015 compared with 44.4% on its core work, i.e. doing or managing the higher level evaluations. That is more than half of its evaluation work is being spent on the validation process. Finally, the PCVRs now seem to be, to a great extent, a standalone output of OIE. It is not always clear to us how they are being used as the “building blocks” for the OIE’s independent evaluations. Making this clearer in the independent evaluations would help show the link and therefore the value of the time being spent on the self-evaluation validations.
To conclude, the review finds that the OIE has taken steps to improve the perceived utility of evaluation in several ways. In the first instance, by planning its work to provide relevant and timely evidence geared towards helping the Board with its oversight and decision making tasks. The topics are selected through dialogue between the OIE and key CDB stakeholders and reflect
111 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press112 See the reports available at the WHO’s Health Evidence Netowkr at http://www.euro.who.int/en/data-and-evidence/evidence-informed-policy-making/health-evidence-network-hen
191
priorities of the CDBs strategic plan. Secondly, by securing the interest and consequently the buy-in of the OAC and CDB senior management through engaging their input throughout the evaluation process. This is evidenced by the reported interest in the latest three studies, the Country strategy programme in Haiti, the evaluation of policy-based operations and the SDF 6& 7 multicycle assessment.
The OIE products are of an acceptable quality and could be even better if some of the shortcomings were addressed. However, the products themselves do not impair the utility of OIE’s work; this is undermined in several ways: (1) by the time delays in commenting on PCRs (OIE) and providing feedback to the independent evaluations (operations and management) (2) by the inefficient processes for agreeing topics and funding sources as well as providing OIE with management responses to its reports.
Putting Evaluation to Use: transparency, feedback and follow-upThere are several ways that evaluation can be, and is being used. As John Mayne has pointed out in his many publications on the issue,113 when we talk of evaluation use, we are mainly thinking about its Instrumental use—use made to directly improve programming and performance. But there is also conceptual use - use which often goes unnoticed or more precisely, unmeasured. This refers to the kind of use made to enhance knowledge about the type of intervention under study in a more general way. Or even Reflective use— this refers to using discussions or workshops to encourage and support reflection on the evaluation findings to see how they might contribute to future strategies.
In the case of the CDB there is some evidence to suggest that “use” is not only instrumental, but other types are also developing. For example, in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessons. (Reflective use)
Another important use, as recommended by the ECG, is that from time to time a synthesis of lessons is drawn from a number of evaluations and made available publically. In fact the Panel was impressed to hear that in the past, the evaluation unit had done this drawing on lessons from evaluations of the power sector. (Conceptual use) Although nothing has happened since, it is now on the “to do list” for 2016 (OIE’s 2016 Work Plan).
As for instrumental use, responsibility for using the knowledge generated through evaluation and for possibly drawing up an action plan of what should be done is up to CDB senior management and the relevant CDB department and division. Oversight on applying recommendations and picking up on the lessons drawn is the responsibility of the OAC.
Evidence on how evaluations have actually contributed to decisions or negotiations is lacking or confusing, Certainly the OIE is unaware of the extent to which its evaluations are put to use. On the one hand, the OAC minutes sometimes indicate that lessons learned are integrated into the next phase. On the other hand, the reviewers were told that often in the past, the evaluation results were “too old” to be of use as the lessons had already been drawn and used way before the report was completed. Similarly, people’s gaps in memory on how well the evaluative information from previous studies may have been used may also account for the scarcity of evidence.
In response, the Panel questioned CDB staff and the OIE about a particular study, the Technical and Vocational Education and Training Assessment. The feedback was somewhat contradictory. On the one hand, the study was criticised as “confirming” news rather than bringing “new news”. However, on the other, we learned that In October 2015, the Board of Directors approved a proposal for the revision of CDB’s Education and Training Policy and Strategy. Work on this has already begun and an external consultant has been engaged to lead the process.
113 See for example, his opening chapter to Enhancing Evaluation use: Insights from internal Evaluation Units, Läubli Loud, M. and Mayne, J. 2014, Sage Publications
192
Although it is one of the OIE’s tasks to set up a database on results and lessons learned from evaluations, so far this has not been a priority. There is also currently no systematic tracking of lessons or recommendations arising from the evaluations, or on any progress in their uptake. (The Panel has already referred above to OAC’s lack of oversight in the use of evaluation.)
The OIE’s role in supporting CDB’s organisational learning is clearly specified in the Evaluation Policy, with many good suggestions for knowledge sharing activities such as “brown-bag lunches, workshops, pamphlets and short issues papers” (p. 19). So far, however, the OIE’s lead role on the knowledge sharing side appears to be quite limited. It has provided advisory input in Loan Committee discussions, and organises workshops together with the relative operations department for discussing the implications of evaluation studies. Ultimately, of course, the uptake of evaluation results and knowledge is in the hands of management. But the evaluation unit has an important role to play in terms of knowledge broker and knowledge manager. Both have tended to be underplayed in OIE’s work plan so far.
Transparency: The Communication Strategy
In recent times and with the approval of its new Disclosure Policy, the CDB has started to post its independent evaluation reports on its website. (There is nothing on the self-evaluations). The website also presents a good overview of the role and function of the OIE and evaluation within the CDB. This is a step in the right direction for sharing information. However, in our view, the CDB’s communication strategy is the weakest part of the evaluation system to date.
The Panel has already commended the OIE in its efforts to engage the CDB and the OAC in evaluation work. But reporting and communicating the lessons seem to be entirely targeted at the Board and the CDB. Moreover, the 2015 budget provides only US$2’000 for communication – nothing of which is intended for outreach.
Reviewers feel that actively engaging with the more indirect stakeholders, for example project implementers in the BMCs, NGOs or project beneficiaries is relatively weak114. There appears to be little reflection on drawing out significant messages for the broader group of stakeholders, or on how then to transmit them to the “right” people in the “right” way (knowledge brokerage).
To conclude, evidence on the uptake of evaluation is either confusing or sparse. It is unfortunate that so far no systematic record keeping system has been put into place to track lessons learned or the uptake of recommendations (or actions agreed from management responses). The OIE plays a weak role in brokering the knowledge generated through evaluations to the benefit of external partners and in managing such knowledge. Although the Evaluation Policy specifies the need for “distilling evaluation findings and lessons learned in appropriate formats for targeted audiences both within and outside the CDB” (p.19) such a targeted communication strategy has yet to be developed and budgeted.
Strengthening Evaluation Capacities and Networking From the onset in 2012, the OIE has stressed the importance of developing and strengthening evaluation capacities within the OIE, the CDB and, subject to available resources, in borrowing member countries. Building evaluation capacity in BMCs and the CDB is one of the OIE’s mandated tasks. It has been a priority that figures on the work plan from the beginning (Work Programme and Budget 2012-2104) The idea of developing an internship programme for graduates from the Caribbean region was one idea that was advanced to help build local evaluation resources. However, the capacity-building has primarily been focused on OIE and CDB staff to date. One of the OIE’s two objectives for 2015 therefore was to take up the challenge and “strengthen evaluation capacities and networking” to include reaching out to the BMCs.
114 A broader communication strategy is one of the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
193
Developing OIE staff capacities
The change from project level to strategic and thematic evaluations does require different evaluative skills and competencies. The MDB Evaluation Pyramid presented below in Figure 3 shows the different types of evaluation and changing resource needs as one ascends the pyramid. Implicit here also is the change in the type of expertise and competencies needed as evaluation aspires to the higher levels.
Consequently for 2015, the OIE set itself the objective of networking and developing working partnerships with regional and international evaluation entities and academic institutions. The rationale was twofold: (1) secure further support and guidance as well as (2) increase its outreach and coverage through joint work and international exposure. Another implicit aim was to benefit from partners’ contacts in the BMCs wherever possible so as to improve data collection and quality.
Figure 3: The MDB Evaluation Pyramid115
The OIE has therefore linked up with Carleton University in Canada and the University of the West Indies, Barbados campus. The OIE was also approached by the Development Bank of South Africa to exchange experiences about setting up an evaluation entity in a “small” development bank. However, its attempt to become a member of the Evaluation Cooperation Group was not successful for reasons beyond its control.
The OIE is to be commended in addressing the issue of staff competencies and professional development more generally. New developments in evaluation as well as new developments in the scope of OIE’s work may necessitate new competencies. For this reason, organisations such as the International Developmental Evaluation Association have recommended that the
115 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
194
competencies of evaluators and evaluation managers should be periodically reviewed. Several publications now exist on competency requirements and suggestions for the periodic review of staff competencies.116
It is not within this remit to compare and contrast OIE’s competencies with those recommended by international and national agencies. However, what we can say is that the OIE demonstrates great forethought in taking this on board.
Capacity building within CDB
The OIE’s objective also consists of continuing to develop measures for improving the monitoring and self-evaluation side of CDB’s work. OIE’s strategy here is to use the windows of opportunity on offer through some of the training sessions that are being organised by CDB as part of its shift towards MfDR e.g. by Corporate Planning Services and Technical Assistance. For 2016 it is also planned to have the OIE present at the annual staff meeting and Learning Forum.
The OIE also organises some ad hoc training with operations, for example to help understand new tools e.g. for drawing out lessons from self-evaluation reports and, more generally, in helping staff appreciate how evaluation can add value to the organisation’s work. Measures include providing advisory services on demand, and providing training alongside the introduction of new or revised tools.
Capacity building in the BMCs
This is an ambitious task and would require additional investment; from the bi-annual work plans to be effective. A modest attempt has been made in 2015; from what we understand, the OIE has joined together with the Carleton University and the University of the West Indies, using their networks in some of the BMCs, to try to develop this aspect.
To conclude, we cannot comment on the quality or reaction to such training, but can commend the OIE for making capacity building one of its priority objectives. From both the Policy and the documents we reviewed, we note that capacity building was always seen to be an important aspect of OIE’s work, but hitherto has received little strategic focus. But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.
Adequacy of the OIE’s human and financial resources to support its work
OIE’s Human Resources;
The OIE is has a staff of 5; the head, 1 senior evaluation officer and two evaluation managers, plus one administrative assistant. Three of the five were recruited from within the CDB. The limited capacity means that it is not feasible to cover all the types of evaluation activities outlined in the Evaluation Policy. Yet there is some indication from the Board that OIE should embark on impact evaluations at some future stage. An increasing demand for evaluation and for impact evaluations in particular, would run the risk of overstretching the OIE’s capacity to deliver credible and useful evaluations. Moreover, there are many other designated OIE activities that should be recognised as valuable work; the validations, building CDB and BMC evaluation capacity, providing supervision, advice, knowledge management and brokerage as well as managing evaluation contracts, The time needs of dealing with all of these may be underestimated in OIE’s budgets; all are important for assuring best value from evaluation. The Panel is concerned that a demand for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level evaluations may undermine the importance and time needs of other 116 E.g. IDEAS, (2012) Competencies for Development Evaluation Evaluators, Managers and Commissioners, the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice (2010) and the Swiss Evaluation Society’s Evaluation Managers Competencies Framework (2014)
195
essential tasks.
Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses.
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.
Self- and independent evaluationSelf-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes. Both types of evaluation are important as they are at the very heart of the evaluation function; they are said to be the building blocks for the more strategic evaluations that the OIE is now undertaking.
The Evaluation Coordination Group recommends that the self-evaluations be carried out by the relevant operations department and in turn, reviewed and validated by the organisation’s independent evaluation office. The CDB’s Evaluation Policy therefore talks of “validating all self-evaluations” as being one of OIE’s essential oversight tasks.
196
Within CDB, the self-evaluations should provide management with performance assessments and thereby serve an accountability function to the CDB and Board. To support the process, the OIE provides operations with manuals and checklists for guidance. Once a self-evaluation report is to hand, it is given over to the OIE for the validation of its technical quality and credibility.117
However, in the CDB case, there are well-documented issues that have affected the quality and timeliness of the self-evaluations on the one hand, and therefore the quality of the foundation on which to build the independent evaluations. Paucity of documentation within CDB, paucity of data collected and available in the Borrowing Member Countries (BMCs), time delays in producing completion reports and in turn, having them validated by the OIE - all such issues were systematically raised during interviews and in some of the independent evaluation reports. There appears to be little incentive to complete self-evaluations in a timelier manner.
Generally speaking, many of the monitoring data problems appear to be due to a lack of management oversight. For example, with the introduction of results-based management, the logic frame and monitoring and data needs are systematically being built into intervention design. However, the BMCs are not delivering the data as contractually agreed at the outset. Incentives to support any significant change towards building a results-based culture seem to be weak and sanctions seem to be rarely enforced when the supply of data is lacking or lengthy delays to the projects occur. Although we can appreciate the complexities of trying to enforce monitoring compliance, this means that often, project deadlines have had to be extended, data gaps are not being satisfactorily dealt with and in turn, there has been a void in the quality and quantity of available evidence for the CDB’s self-assessment of project performance. For some time, this lack of oversight has been tolerated. Part of the problem is the low priority accorded to completing the self-evaluation reports by operations, coupled with the absence of any focal point within senior management to drive the process and deal with the problems.
No record is kept of how the self-evaluation results are actually used. They do not appear on the CDB website, but we were told that the findings are integrated into the following project designs. Hence we are somewhat unclear as to the utility of these reports at present. The situation is exacerbated by a rather confused image of evaluation: some operations staff consider OIE’s input (through validations or independent evaluations) to be sometimes over-critical, regulatory and adding little value; it is a threat rather than an opportunity for learning. Yet at the same time, evaluation is recognized as an integral part of result-based management.
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.
Some managers however seem to start changing the status quo. For example a revised and simplified template for producing project completion reports is being considered, and mid-term project reviews are expected to be more stringent in looking at monitoring plans and practices and tying disbursements to performance. In some cases we also learned of incentives being introduced to encourage project managers to complete their reports in a timelier manner. But much remains to be done and, since the OIE is no longer responsible for monitoring and project evaluations, there is a void that needs to be filled. It is up to line managers to drive this work forward.
To conclude, it is fair to say that in view of a number of “frustrations” between the OIE and operations, which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring data, the added value that evaluation might offer to the operations area is ill recognized. Moreover, the link between self-evaluation as the
117 According to the Evaluation Policy, OIE should validate all PCRs and CCRs but due to the backlog of reports and the delay in completing them (sometimes years later) since October 2015, the OIE has secured OAC agreement to validate a maximum of 6 per year, which are selected in consultation with the OAC.
197
building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
198
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”118
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
118 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
199
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”119 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 120 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
119 CDB (2011) Evaluation Policy (p.2)120
200
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
201
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
202
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
203
mendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
204
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
205
APPENDICES
Appendix I - The External Review Mandate – Terms of Reference and Approach Paper
Appendix II -Review Approach, Data collection and Analysis, and Limitations
Appendix III – Overview of OIE Evaluation Practice
Appendix IV - List of Persons Interviewed
Appendix V - List of Documents Reviewed
Appendix VI- List of Topics used to guide interviews with members of CDB Board of Directors
Appendix VII - List of Topics used to guide interviews with CDB staff
206
Appendix III – Overview of OIE Evaluation Practice (prepared by the OIE in response to Reviewer’s request)
Caribbean Development Bank, Office of Independent Evaluation - OIE
Category Response
Percentage of projects subject to project (self-) evaluation
100% - Project Completion Reports (PCR)
Percentage of projects subject to validation by OIE
Approximately 40-50%
About 15 projects exit portfolio annually. Evaluation Policy calls for all PCR to be validated. However, OIE resources insufficient. Validation process reviewed in 2014. Now OAC (Board committee) selects a sample of 6-8 PCR for validation each year.
Percentage/number of projects subject to in-depth review by OIE
None – unless specifically requested by OAC
Due to limited resources, focus of OIE evaluation work programme is on PCR validations and high-level evaluations – including country strategy and programme evaluations (CSPE).
Number of high-level evaluations conducted by OIE (e.g. sector, thematic, geographic)
1-2 per year since 2011
Plan is 2-4 per year from 2016. This would include CSPE (1st planned for Q1 2016: Haiti)
Number of project impact evaluations conducted by OIE
None
OIE includes “impact questions” in high-level evaluations.
Number of project impact evaluations conducted by Bank staff or other non-OIE staff
OIE is not aware of any impact evaluation conducted by the Bank.
However, OIE provides technical support to the Basic Needs Trust Fund (BNTF) in its design of an M&E framework that entails impact evaluations.
Budget In USD mn: 0.78 in 2015; 0.82 in 2016. This is equivalent to about 2.5% of total CDB Administrative Budget.
75% of the budget is for Staff salaries (4 Professionals, 1 Support staff), leaving around USD 190,000 (in 2015) for other expenses, including consultants e.g. for external evaluations. Additional funding is accessed via the Special Development Fund (SDF). This varies according to type and scope of the evaluation, e.g. the ongoing SDF 6/7 Evaluation is SDF funded at USD 255,000.
Budget determined by Board, not separate from administrative budget.
SDF funding for evaluations is considered separately and subject to Bank internal approval process. SDF funding cannot be used to cover OIE expenses such as staff time or travel. Country eligibility for SDF funding is also a consideration. OIE expressed concerns about this funding track in respect to predictability, independence and eligibility limitations.
Head of OIE reports to Board, with administrative link to the President
Terms of appointment for Head
207
5 year term, renewable once. Appointed by the President with the agreement of the Board.
Right of Return for Head Not eligible for other staff positions.
Consultants as proportions of OIE budget
2015: 19% (USD 145,000)
Plus SDF funding. SDF funded evaluations are outsourced.
Last external evaluation (or peer review) of OIE
No external evaluation, though a review of the function was done in 2011, leading to the Evaluation Policy.
OIE External Review completed in April, 2016
Departments or special programmes supporting impact evaluation
None
208
Appendix IV – List of Persons Interviewed
Name Function relative to OIE Type interview
Mrs. Colleen Wainwright Member CDB Board of Directors (UK)
Face to face
Mrs. Cherianne Clarke Alternate MemberCDB Board of Directors (UK)
Face to face
Mrs. Jean McCardle Member CDB Board of Directors (Canada)
Face to face
Dr. Louis Woodroofe MemberCDB Board of Directors (Barbados)
Mr. A: de Brigard Former Member CDB Board of Directors
Skype interview
Mr. H. Illi Fromer Member CDB Board ofDirectors
Telephone interview
Mrs. Claudia Reyes Nieto Member CDB Board of Directors
Telephone interview
Mr. Bu Yu alternate DirectorCDB Board of Directors
Face to face
Mr. Michael Schroll(Barbados)
Head OIE
series of interviews viaSkype and face-to-face
Mr. Mark Clayton OIE Senior Evaluation Officer Focus GroupMrs. Egene Baccus Latchman OIE Evaluation OfficerMr. Everton Clinton OIE Evaluation OfficerMrs. Valerie Pilgrim OIE Evaluation Officer
Dr. Justin Ram CDB Director Economics Department
Face to face
Mr. Ian Durant CDB Deputy Director Economics Dept Face to faceDr. Wm Warren Smith CDB President
Joint interviewFace to face
Mrs. Yvette Lemonias-Seale CDB Vice President Corporate Services & Bank Secretariat
Mr. Denis Bergevin CDB Deputy DirectorInternal Audit
Face to face
Mr. Edward Greene CDB Division Chief, Technical Cooperation Division
Face to face
Mrs. Monica La Bennett CDB Deputy Director Corporate Planning Face to faceMrs. Patricia McKenzie CDB Vice President Operations Face to faceMs. Deidre Clarendon CDB Division Chief
Social Sector DivisionFace to face
Mrs. Cheryl Dixon CDB Co-ordinator, Environmental Sustainability Unit
Focus group
Mrs. Denise Noel- Debique CDB Gender Equality Advisor Mrs. Tessa Williams-Robertson CDB Head Renewable EnergyMrs. Klao Bell-Lewis CDB Head Corporate Communications Face to faceMr. Daniel Best CDB Director
Projects DepartmentFace to face
Mr. Carlyle Assue CDB Director Face to face
209
Finance Department
210
Appendix VI - Interview Guide: Members of CDB Board of Directors
Below are a list of themes that I should like to raise with you based on your experience and knowledge of the CDB’s independent evaluation function (Office of Independent
Evaluation).
In each case, I should be grateful if you could illustrate your responses with examples or help this Review by, wherever possible, sending me (or telling me where I can find) any
documents that could support your responses.
This guide is being sent to you in advance to help prepare our meeting. However, our interview will be conducted more in the style of a conversation. The following sub-questions will be used to GUIDE the interview. Please feel encouraged to raise any
additional issues that you feel we should take into account
On the governance and Independence of CDB’s evaluation functionWhat mechanisms are there in place to support its independence?
How satisfactory are the current arrangements in your opinion?
How is the balance between independence and the need for interaction with line management dealt with by the system? For example, what mechanisms exist to ensure that the OIE is kept up to date with decisions, policy / programme changes, other contextual changes etc that could have an affect on OIE evaluation studies / evaluation planning?
On the OIE’s Evaluation PolicyThe CDB’s Evaluation Policy was established in 2011. To what degree do you feel it is adequate? Still relevant?
What suggestions do you have for any improvements?
In your opinion, how adequate is the current quality assurance system for over viewing the evaluation function?
On the quality and credibility of evaluation studiesTo what degree do you believe the reports are fair and impartial?
Do you consider them to be of good quality? Are they credible?
Are you adequately consulted/involved on evaluations of interest to you?
On the relevance and usefulness of evaluations How well does the OIE engage with you / your committee during the preparation, implementation and reporting of an evaluation study to assure that it will be useful to the CDB?
How are the priorities set for the independent evaluations? What criteria are used? Are you satisfied with the current procedure?
211
When OIE evaluation studies are outsourced to external consultants, what criteria are used to make this decision?
How are the priorities for the OIE’s 3.year rolling work plan agreed? In your opinion, is the current plan adequate in terms of coverage and diversity?
In your opinion, do the evaluations address important and pressing programs and issues?
To what extent do you feel that the OIE’s evaluations integrate the cross-cutting theme such as gender, energy efficiency/renewable energy, climate change? What improvements might be made and how?
On the dissemination and uptake of evaluation findings and recommendationsTo what extent do you feel that evaluation findings are communicated to the CDB and its stakeholders in a
a) useful, b) constructive andc) timely manner?
Are evaluation recommendations useful? Realistic?
What mechanisms are in place to assure that evaluation results are taken into account in decision making and planning? What improvements do you feel could be made?
How have you used the findings from any evaluations? Examples?
To what degree do you feel that evaluation contributes to institutional learning? And what about to institutional accountability? Any examples?
What mechanisms are in place to ensure that knowledge from evaluation is accessible toCDB staff and other relevant stakeholders? Are the current arrangements satisfactory?
How satisfied are you with current arrangements? What expectations do you have for the future?
On resourcesHow is the OIE resourced financially and is this satisfactory?
What about the OIE staff, are all the important areas of expertise represented in the team?
On this Review of the Office of Independent EvaluationWhat are your expectations? What are you particularly hoping to learn from it?
Thank you very much for your cooperation and input
212
Appendix VII : Interview Pro-Forma – CDB Staff membersThis presents a list of the topics raised during interviews. It was used to guide the open-ended
discussion – this means that the sequence and exact wording of the questions may not necessarily have followed in this order or been asked in exactly this way.
Changeover to an Independent Evaluation Office? Expectations? Advantages and disadvantages??
Satisfaction with working relations between operations and the OIE from your perspective?
Process of dealing with the PCRs and CCRs? Advantages and limitations?
Quality and credibility of the validation process?
How are the self-evaluation reports used?
Credibility and Quality of OIE’s evaluation reports
Communication of self and OIE independent evaluations? To whom, in what way? Possible improvements?
actual or potential conflict of interestThe PanelIt must be s
The PanelisThe; this affects also of y, Work PracticesThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. therefore The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation
213
pyramid121 are brought out in the remaining sections of this Review, not least given the limited resources available.
But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities. sTheThes before completing the final version. However, p are only submitted to the CDB line and senior managers.Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the following the recommendations of professional good practices and standards on participative approaches; it has succeeded in , The Panelsstaff fromas well as There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. OIE doesarrange .sthe Panel also wishes tonewly appointed rsA was evidentthey expressed interest in In one case, interest was followed up in practice; can be improvedfostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that can offer can add value to understanding the strengths and weaknesses of such strategies. Tduring this transitional phase, Manual to guide and support the independent evaluation process.and operations staff sevaluation activities. ,oOIE’sjudginga . As such, theyare thatAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An is processed as follows;the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
SThisrItand Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
The rmade
- is still considered to be good practice to have the elaborated in the initial design documents the122 such as Developmental Evaluation (Patton, 2010123)
-)
said abovePCHowever, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s
121 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).122 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.123 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press
214
agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessonsthe Panelas done this ing on lessonsAlthough nothing has happened since, it is , sometimes indicate (Panel has already referred above to ’s lack of oversight in the use of evaluation.)
sThe Panelsevaluation work Moreover, the 2015 budget provides only US$2’000 for
communication – nothing of which is intended for outreach.Reviewerseither confusing or and
budgetedConsequently for 2015, theFigure 3: The MDB Evaluation Pyramid124
124 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
215
ys to be effective. A modest attempt has been made in 2015; OIE hased But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.man and financial resources to support its work
OIE’s Human Resources;
5eThree of the five were recruited from within the CDB. edfrom the Board that OIE
should embark on ee and for impact evaluations in particular,OIE’s ee Moreover, there
are many other designated OIE activities that should be recognised as valuable work; the
validations, building CDB and BMC evaluation capacity, providing supervision, advice,
knowledge management and brokerage as well as managing evaluation contracts, The
time needs of dealing with all of these may be underestimated in OIE’s budgets; all are
important for assuring best value from evaluation. The Panel is concerned that a demand
for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level
evaluations may undermine the importance and time needs of other essential
tasks.Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses.
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies
216
does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.Self-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes.types of evaluation y There appears to be little incentive to complete self-evaluations in a timelier manner.
.
; it is a threat rather than an opportunity for learning. Yis recognized as
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.a number of , which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring dataadded value that evaluation might offer to the operations area is ill recognized Moreover, the link between self-evaluation as the building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
217
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”125
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
125 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
218
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”126 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 127 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
126 CDB (2011) Evaluation Policy (p.2)127
219
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
220
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
221
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
222
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
Independence of the Office of Independent Evaluation (OIEIndependence is absolutely central to the integrity and trustworthiness of evaluation. It is an agreed requirement within the development agencies and in the evaluation community as a whole. In examining the issue of independence and good practice, reviewers are guided by the Evaluation Cooperation Group’s recommendations on good practices, the CDB’s Evaluation Policy and by the 2011 consultancy review of independence relative to the CDB’s evaluation and oversight division128. The appraisal is based on a comparison of the ECG’s recommendations on independence129 and the current OIE status.
OIE and Independence: Recommendations from the OECD Evaluation Cooperation Group (ECG)
The ECG’s considers the issue of independence according to three specific areas: organisational, or structural independence, behavioural, or functional independence and protection from outside interference, or operational independence.
Organizational independence, ensures that the evaluation unit and staff are protected against any influence or control by senior or line management, and have unrestricted access to all documents and information sources needed for conducting their evaluations. Also, that the scope of evaluations selected can cover all relevant aspects of their institution.
Behavioural independence, generally refers to the evaluation unit’s autonomy in selecting and conducting setting its work programme and in producing quality reports which can be delivered without management interference.
Protection from outside interference refers to the extent to which the evaluation function is autonomous in setting its priorities, and conducting its studies and processes and in reaching its judgments, and in managing its human and budget resources without management interference.
Conflict of interest safeguards refers to protection against staff conflict of interests be they current, immediate, future or prior professional and personal relationships and considerations or financial interests for which there should be provision in the institution’s human resource policies.
The OIE’s Independence in Practice
Organisational / structural independenceOn the whole, the Panel acknowledges and commends the efforts being made by the CDB to assure OIE’s organisational independence. The CDB’s Evaluation Policy provides for the OIE’s organisational independence from line management and the interview data suggests that there is also wide acceptance and acknowledgement of why the OIE should have such independent status. Table 1 below provides our overall assessment of this aspect of OIE’s independence when compared with ECG recommendations. 130
128 Osvaldo Feinstein & Patrick G. Grasso, Consultants, May 2011 Consultancy to Review the Independence of the Evaluation and Oversight Division of the Caribbean Development Bank129 ECG 2014 Evaluation Good Practice Standards, Template for Assessing the Independence of Evaluation Organizations, Annexe II.1 130 Based on ECG (2014) Template for Assessing the Independence of Evaluation Organizations, Evaluation Good
Practice Standards, Annexe II.1
223
Table 1: OIE organisational independence compared with ECG recommendations
Aspects Indicators CDB Evaluation Policy (EP) and Practice
The structure and role of evaluation unit
Whether the evaluation unit has a mandate statement that makes clear its scope of responsibility extends to all operations of the organization, and that its reporting line, staff, budget and functions are organizationally independent from the organization’s operational, policy, and strategy departments and related decision-making
Partially Complies The Policy is broad enough to cover the full range of MDB type of evaluations. However in practice this would not be possible without additional human and budget resources
The unit is accountable to, and reports evaluation results to, the head or deputy head of the organization or its governing Board
Whether there is a direct reporting relationship between the unit, and
a) the Management, and/or
b) Board or
c) relevant Board Committee, of the institution
Complies - OIE reports to the Board of Directors (BoD) through its Oversight Assurance Committee (OAC)
The unit is located organizationally outside the staff or line management function of the program, activity or entity being evaluated
The unit’s position in the organization relative to the program, activity or entity being evaluated
Complies - The OIE is located outside, and is therefore independent of CDB line management
The unit reports regularly to the larger organization’s audit committee or other oversight body
Reporting relationship and frequency of reporting to the oversight body
Complies - The OIE reports x 5 per year to the OAC . Board approval for an additional executive meeting between the Head of the OIE and the OAC at least once per year was given in October 2015
The unit is sufficiently removed from political pressures to be able to report findings without fear of repercussions
Extent to which the evaluation unit and its staff are not accountable to political authorities, and are insulated from participation in political activities
Complies
Unit staffers are protected by a personnel system in which compensation, training, tenure and advancement are based on merit
Extent to which a merit system covering compensation, training, tenure and advancement is in place and enforced
Partially Complies - with CDB human resource policy. However the skill needs of OIE staff ought to be regularly reviewed in light of its move towards higher-level evaluations. Appraisal of skill needs and hiring of relevant staff should be completely under the authority of the Head of Evaluation. This is not sufficiently clear in the Policy or other documents we reviewed.
224
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Unit has access to all needed information and information sources
Extent to which the evaluation unit has access to the organization’s
a) staff, records, and project sites;
b) co-financiers and other partners, clients; and
c) programs, activities, or entities it funds or sponsors
Complies –The available evidence suggests that there is no reason to doubt such access. But systematic and easily accessible documentation is lacking in the CDB; it is one of its weak points.. Delays in getting hold of the relevant documents can have consequences on the timeliness of evaluation studies
However, independence should not mean isolation: There appears to be a detachment between the OIE and CDB that is of concern to the Panel; on the one hand, between the OIE and operations staff, and (2) on the other, in terms of the structural arrangements between the OIE and senior management.
12) In agreeing for the OIE to concentrate on strategic and thematic, in-depth evaluations, responsibility for project monitoring and evaluation were given over to operations. The division is clear and respected. However, it has its drawbacks. With the OIE no longer systematically involved at the front-end of project design, the monitoring data needs are likely to be poorly defined. Weak monitoring data will contribute to weaker evaluations. (More on this point under the heading self and independent evaluations.)
In the reviewers’ opinion, it is a common misunderstanding to assume that providing evaluator advice on monitoring and evaluation data will comprise evaluator independence. On the contrary, evaluation input into project design is essential to assure that the logic, indicators and data needs are addressed so that at some future point in time an evaluation of the achievements can be empirically grounded.
This is not to say that the OIE no longer has any influence at the front-end design stage; it has merely shifted the point of focus. The OIE is now systematically providing such input more generally to the corporate planning teams for the tools and systems they are developing to support the MfDR framework. The monitoring data for projects and their implementation should be improved once the Project Performance Evaluation System (PPES) and the Portfolio Performance Management System (PPMS) are updated and operational.
13) In the second place, the OIE has limited formal access to the Advisory Management Team (AMT) weekly meetings where the President and senior management gather to exchange up-to-date information on the dynamics of CDB policy and practice. The OIE is not regularly invited in any capacity to these meetings or given a copy of the agenda or minutes; the OIE is occasionally invited to attend in order to discuss an evaluation report or management feedback. For the OIE, this means that it is unlikely to pick up on the ‘when’ and ‘what’ of key decisional issues or provide input into the discussion based on evaluative information. Its observer status at Loans Committee meetings, or as a participant informer at the OAC and BoD meetings and discussions do not necessarily provide the same insight as to the dynamics of management actions and/or decisions. .
To respond to this situation, the President has agreed to meet regularly with the Head of the OIE in order to keep him up to date with CDB strategic thinking. This is a welcomed change.
OIE Independence and Behavioural Issues The Panel has concerns about some behavioural issues. For example, through both the interviews and documentary review, we learned of considerable delays in processing both the
225
independent evaluation reports as well as OIE’s validation of the CDB’s self-evaluations. Delays are generally due to receiving feedback on the independent reports from first, the relevant operational department, then from the AMT, and then on providing the OIE with a management response that is initially drafted by operations staff before being reviewed by the AMT. (OIE reports cannot be submitted to the OAC without the relevant management response). This two-layer process for preparing submissions to the Board is inefficient and could potentially be a threat to evaluation’s independence in the future by delaying OIE’s timely reporting to the OAC.
OIE validations of the CDB self-evaluations are also submitted to the OAC, but it is in both sides’ interest to clear up any misunderstandings beforehand. Despite attempts to improve the timeframe for completing these validations, delays are more the norm than the exception. Table 2 below summarises our assessment of the behavioural aspects of independence.
Table 2: OIE and Behavioural Independence
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Ability and willingness to issue strong, high quality, and uncompromising reports
Extent to which the evaluation unit:
a) has issued high quality reports that invite public scrutiny (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk) of the lessons from the organization’s programs and activities;
b) proposes standards for performance that are in advance of those in current use by the organization; and
c) critiques the outcomes of the organization’s programs, activities and entities
Partially complies – paucity of data and documentation sometimes hinder the quality of reports. The OIE emphasizes the learning part of evaluation, and is cautious in its criticism recognising that management is going through a transitory stage and can still be overly defensive.
Ability to report candidly
Extent to which the organization’s mandate provides that the evaluation unit transmits its reports to the Management/Board after review and comment by relevant corporate units but without management-imposed restrictions on their scope and comments
Partially complies - as sometimes reporting to the Board is compromised by delays in the review/comment process between the OIE and the CDB. Any delay with the production of a Management Response will also mean that submitting a report to the Board in a timely manner is impaired since the two have to be submitted together.
Transparency in the reporting of evaluation findings
Extent to which the organization’s disclosure rules permit the evaluation unit to report significant findings to concerned stakeholders, both internal and external (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk).
Who determines evaluation unit’s disclosure policy and procedures: Board, relevant committee, or management.
Partially complies - The OIE’s conforms to the CDB’s disclosure policy. However, the dissemination of evaluation findings appears to be currently restricted to website publication and reports to the Board. A more targeted communication strategy to include other key stakeholders, e.g. project implementers in the BMCs should be developed and put in place.
Self-selection of items for work program
Procedures for selection of work program items are chosen, through systematic or purposive means, by the
Complies - The OIE also ensures that its work program is drawn up after consultation with both CDB Management
226
Aspects Indicators CDB Evaluation Policy (EP) and Practice
evaluation organization; consultation on work program with Management and Board
and Board to seek their input on relevant topics and themes.
Protection of administrative budget, and other budget sources, for evaluation function
Line item of administrative budget for evaluation determined in accordance with a clear policy parameter, and preserved at an indicated level or proportion; access to additional sources of funding with only formal review of content of submissions
Partially complies - The administrative budget for supporting OIE work is protected. Access to additional sources of funding is possible if well argued and justified. But the approval process is complex and inefficient. (See Figure 1 below)
OIE and Protection from External influence or interference
Our overall assessment is provided in Table 3 below. The OIE’s independence in the design, conduct and content of its evaluations does not appear to be subjected to any external interference. But securing funding from any sources outside the OIE’s administrative budget, i.e. from the Social Development Fund, is an unduly complex and long process. As such we consider that the current funding process can affect the OIE’s choice with regard to the type of evaluations it can undertake. (See Figures 1 and 2 below)
Table 3: OIE and its Independence from External influence or interference
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Proper design and execution of an evaluation
Extent to which the evaluation unit is able to determine the design, scope, timing and conduct of evaluations without Management interference
Complies – however within limits of restricted human and financial resources available
Evaluation study funding
Extent to which the evaluation unit is unimpeded by restrictions on funds or other resources that would adversely affect its ability to carry out its responsibilities
Partially Complies - OIE must work within the limits of the agreed administrative budget wherever possible. If additional resources are needed for studies it must seek alternative funds elsewhere. The budget limitations can have an affect on the type of evaluations undertaken and therefore its independence in terms of choice.
Judgments made by the evaluators
Extent to which the evaluator’s judgment as to the appropriate content of a report is not subject to overruling or influence by an external authority
Complies – the evidence available suggests that the Board and Management accept the evaluators’ independent interpretation and conclusions Management responses are agreed to be the accepted place to raise any difference of opinion.
Evaluation unit head hiring/firing, term of office, performance review and compensation
Mandate or equivalent document specifies procedures for the
a) hiring, firing,
b) term of office,
c) performance review, and d). compensation of the evaluation unit head that ensure independence from operational management
Complies – the Head of OIE is appointed by the CDB President in agreement with the OAC for a 5 year period which is renewable x 1. The Head could be removed from Office by the President or the Board but only with the agreement of both parties.
However the Head reports to the President for all administrative and personnel matters. Even though this was not recommended in the Osvaldo Feinstein & Patrick G. Grasso report on Independence in 2011, the BoD accepted CDB’s reasons for keeping this arrangement. (e.g.most OAC members are non residents and cannot
227
Aspects Indicators CDB Evaluation Policy (EP) and Practice
oversee day-to-day work)
. Extent to which the evaluation unit has control over:
a) staff hiring,
b) promotion, pay increases, and
c) firing, within a merit system
Partially complies - All OIE staff members are treated in the same way as other CDB staff. The Head has limited control over the hiring, firing or promotion of OIE staff.
Continued staff employment
Extent to which the evaluator’s continued employment is based only on reasons related to job performance, competency or the need for evaluator services
Partially complies - Whilst the EP is clear about procedures for hiring, firing and promotion, all of which must conform with CDB human resource policy, there is nothing mentioned about any difference of opinion between the CDB and the Head of the OIE with regard to continued staff employment subject to changes in the level of technical or interpersonal competencies needed to meet new demands.
Avoidance of Financial, Personal or Professional conflicts of interest
This particular aspect refers to the organisation’s Human Resources Policy; there must be provisions in place to protect against actual or potential conflict of interest. The Panel requested via the OIE, to have evidence from human resources on any such provisions but did not receive an answer. It must be assumes that this aspect of independence, past or present, does indeed form part of normal CDB Human Resource Policies
To conclude: The Panel is impressed with the measures CDB has taken to assure the organisational independence of the OIE. Its independent status is accepted and respected by senior and line management. The OIE’s budget is not independent from the overall CDB administrative budget; this affects its choice of evaluation types or approaches. Some of the behavioural issues affecting independence were also of concern, especially due to the delays in the exchange of documents, between the OIE and operations departments, which has a direct effect on timely reporting to the OAC. As for protection from outside interference, our concerns are largely to do with OIE’s independence over staffing issue; there are potential loopholes in current arrangements that could undermine OIE’s autonomy over its staff.
OIE’s Strategy, Work Practices and Work ProgrammeThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. Much of the period 2012 to 2015 has therefore been taken up with preparing OIE’s shift in focus from project-based evaluations to the high-level thematic and in-depth strategic studies. This has meant adopting a three-way approach; (1) for self-evaluations, reducing its time input to support the process and (2) for independent evaluations, taking stock of the gaps in coverage and expertise, and (3) networking to share experiences with centres of expertise and align OIE with international practices. In addition, amongst other duties, it has been supporting the development of MfDR tools and systems such as the Project Performance Assessment System by providing advice and input on programme logic and monitoring needs. The OIE plans to conduct 2-4 high-level studies per year from 2016. The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
228
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation pyramid131 are brought out in the remaining sections of this Review, not least given the limited resources available.
To conclude: The OIE has made a first step in proposing a strategy for establishing itself as an independent evaluation resource. But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities.
The Value / Usefulness of OIE’s Independent EvaluationsEvaluation is a powerful tool that can provide useful, evidence-based information to help inform and influence policy and practice. But useful evaluations depend not only on the evaluators’ skills, but on several other important factors as well; 1) on planning evaluations to be relevant to the priorities of the organisation’s work and for their results to be delivered in time to be useful; on the degree of 2) consultation and ultimately ownership by those who seek evaluative information; on the 3) tools used to support the evaluation process per se; and on the 4) credibility and quality of the evaluation products132.
1. Planning relevant and timely evaluationsThe OIE is now working on a 3 year rolling work plan that sets out the broad areas for enquiry. So far, there are no agreed criteria for making the selection of the specific topics for independent evaluation, although the priorities tend to reflect those of the CDB’s strategic plan. Nevertheless decision-making is rather arbitrary based on a process of dialogue between the OIE and the CDB and the OIE and the Board.
One of the OIE’s two objectives for 2015 therefore, was to define a work plan and agree priorities based on an approach that is “utilisation-focused”. This means that the studies are selected and planned to be relevant and useful to the organisation’s needs.
The OIE has achieved this objective with respect to its latest studies, which concerns the Social Development Fund (SDF) Multicycle 6&7 Evaluation, the Haiti Country Strategy evaluation and the evaluation of the CDB’s Policy Based Operations. Each of these three have been planned to deliver their results in time to provide the CDB Board of Directors with relevant information for negotiating the next round of funding. In spite of some delays due to a myriad of reasons, not least to the extra effort needed to secure essential data, the studies are expected to deliver on time.
The processes for agreeing OIE’s work plan and specific evaluations on the one hand, and, in securing alternative funding on the other, are shown in Figure 1 below. The Panel was surprised at learning how bureaucratic (the internal approval process), and inefficient (in view of the time it takes) the process seems to be. The concern here is that such a process could possibly pose a threat to assuring the Board of “timely studies.”
131 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).132 These aspects reflect the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
229
Figure 1: Selection of Evaluation Topics and Funding Source
Consultation with CDB Operations and OAC/Board for selection of
evaluation topic
Internal review of Approach Paper
Specific Evaluation Study Design and Budgeting
OIE Draft Terms of Reference / Approach
Paper
Detailed ToR or Final Approach Paper if sufficiently detailed.
Finalise Approach Paper and submit to OAC/Board
Final Approach Paper
OAC ApprovalOAC minutes
Paper
Funding Track
Final Approach Paper/ToR
3-year Work Programme and Budget (approved by Board)
Board approval necessary If above USD
150,000
Board notification only if USD 150,000 or
below
Board Approval
Board Paper
Annual OIE report and work plan
submission to OAC
OIE – Selection of consultants (if any) contracting
OIE Admin Budget or …
… SDF
Prepare TA Paper (content similar to Approach Paper but different
format.
TA Paper
Approval – Internal Loans Committee
OIE – Selection of consultants (if any)
contracting
230
2. Consultation and ownership“The credibility of evaluations depends to some degree on whether and how the organization’s approach
to evaluation fosters partnership and helps build ownership and capacity in developing countries.”
(ECG good practices)
The OIE engages with the OAC, CDB senior management and operations for agreeing its 3-year work plan and then for selecting the specific topics and themes. It also discusses the evaluation approach paper (design and implementation plan) with the CDB and OAC before completing the final version. However, preliminary and final drafts of the report are only submitted to the CDB line and senior managers for comment and factual errors. Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the results and their implications. Discussions with the OAC are more limited due to the overburdened agenda of OAC and Board meetings, as previously discussed.
In short, the OIE is to be commended for following the recommendations of professional good practices and standards on participative approaches; it has succeeded in having introduced a modus operandi that involves the key players in the selection of evaluation topics, the evaluation designs and their results. Figure 2 below provides an overview of the evaluation implementation and stakeholder engagement processes.
Figure 2: Evaluation Study Implementation and Feedback Loops
Arrangement AFully outsourced / external
consultants; oversight by OIE
Preparations:Detailed evaluation plan (incl tools,
timeline, etc.) and logistics
Production of Inception Report / Approach Paper
Arrangement BConducted by OIE
staff
Arrangement CJointly: external
consultants and OIE
Terms of Reference
Prepares Inception Report /
Approach Paper
Presentation/workshop:Interim findings and conclusions for immediate feedback and validation
Data Collection and Analysis
OIE
Summary and ppt for workshop presentation
and discussion with CDBSubmission of Draft Final
Report to OIE
Board notification only if USD 150,000 or
below
Draft Final Report
Review loops – OIE and CDB (potentially also BMC)
231
11.12.
Notes to Figure 2
11. The OIE informed the Panel that this is an abbreviated version as there are e.g. additional steps (secondary processes) when evaluations are procured (tendering or single source), when there are additional review loops and updates to OAC etc.
12. OAC may also decide to return the report to OIE, the Panel were informed, or demand from Management specific actions based on the report.
This process is engaging and appears to have secured senior management and OAC interest and buy-in as witnessed in the latest studies. But there is the downside too! The process takes much time and, in our view, is partly unnecessary. The Panel appreciates that staff from operations as well as the AMT may both want to confer on an appropriate management response, but this should not be the case for reviewing an independent report for factual errors. The two-phase approach seems somewhat inefficient and unnecessary in our opinion.
Contact between the OIE, the CDB and/or the OAC during the actual study implementation is most often restricted to the occasional progress report, particularly when studies run behind time. There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. The OIE does, however, arrange discussions for reflecting on emerging findings, but we are not sure of how systematic this feedback loop is.
Prepare for disclosure and dissemination
Final OIE approved report to CDB Senior Management for Management Response
Feedback to evaluation lead
Submission of Final Report to
OIE
Final Report
Final Report and Management Response submitted to
OAC/BoardFinal Report and
Mgt. Resp.
Management Response
OIE ApprovalFinal Report and Management Response considered by CDB
AMT
OAC/Board endorsed
232
More generally speaking, outside of an evaluation study, the OIE has limited dealings with operations. The OIE has an advisory role in providing them with help, particularly with providing training, guidelines and tools to support self-evaluations. We are nevertheless concerned about the seeming distance between these two and how this has affected the perceived value of evaluation. (For further on this point, please see the section below on “Self- and Independent Evaluations”)
But the Panel also wishes to stress that this is not the case for newly appointed senior managers. A much more open attitude to evaluation and appreciation of its potential value was evident; they expressed interest in drawing out important lessons on what works, how, for whom, and under what conditions. In one case, interest was followed up in practice; the OIE was recently invited by a senior manager to share evaluative knowledge and experience with his staff regarding policy based operations.
Certainly, we can say that overall, the key stakeholders within the CDB are adequately integrated into the evaluation process as to foster their buy-in and ownership. But more generally, we feel that the utility of independent evaluations can be improved by fostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that evaluation can offer can add value to understanding the strengths and weaknesses of such strategies. This however cannot be done overnight and takes a long time.
3. Tools to support the evaluation processSo far, during this transitional phase, the OIE has mainly focussed on improving the tools to support the operations areas’ self-evaluations. This has left the OIE with little time to produce the checklists or tools to support its own studies. There are plans to develop an OIE Manual to guide and support the independent evaluation process. Such plans should be encouraged, as these documents will form a very important part of training, particularly for newcomers to the OIE team.
In the meantime, the OIE and operations staff refers to the Performance Assessment System (PAS) Manuals for evaluation activities. The manuals are based on DAC criteria and ECG principles. Much emphasis is given to the rating system and how and what should be rated. However we find them lengthy, unwieldy and overcomplicated. Moreover, such manuals should be used for reference, but cannot and should not replace first-hand training in how to plan, conduct and manage the evaluation process.
Quality Assessment (QA) and Quality at Entry (QaE)
There was a transition period between 2012 and 2014 to establish the OIE. Work on the PAS, QaE, PCRs, ARPP, which had started earlier, was therefore completed after OIE came into existence, but it effectively had no formal ‘home’ in operations. The Panel was told that there had been some discussions about creating a Quality Assurance unit within CDB (OPS) but the current status is unclear.
The QaE Guidance Questionnaire was developed before and completed by the OIE. It was used to assess the documents that came across to the OIE for comments at the Review Stage. The results were then sent to the Portfolio Manager/Project Coordinator indicating any gaps/issues that needed to be addressed or clarified. QaE Guidance Questionnaires were developed for all the Bank’s lending products, CSP and to assess the quality of supervision.
After the QaE was launched bank wide, several operations officers saw the merit in using the QaE Guidance Questionnaire in the field and adopted it as a tool for their use during the appraisal mission in order to cross check and test their data collection and analysis.
OIE’s use of the QaE was discontinued in 2014 due to limited resources and a stronger focus on evaluations. It still sometimes comments on specific appraisals, but very selectively.
233
Both QaE and QaS (quality at supervision) are also addressed in the PAS Manuals. In addition the QaE and PAS have been incorporated in Volume 2 of the Operations Manual OPPM.
The Review Panel assessed the QaE forms. They are relatively standard, adapted to the specificities of the CDB. They contribute to judging a project’s expected quality in a relatively objective way. As such, they are are helpful, as a benchmark, in the ex-post assessment of projects.
The Panel considers that the lack of an established Quality Unit in the CDB (and independent from OIE) is a weakness that should be addressed in the near future.
4. Credibility and Quality of Evaluation ProductsAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An independent evaluation is processed as follows; the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
Since 2012, the OIE has produced a range of studies and approach papers. This review is based on those listed below as provided by the OIE, and cover the period from May 2012 to December 2015. It includes 3 evaluations (in blue), 4 Assessment studies (in brown) 14 validations of self-evaluations (in green) and 3 Approach Papers (in purple) for upcoming evaluations. These are listed below in Table 4.
Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
Board Meeting
Date Type / Topic
251 May 2012 Ex-Post Evaluation Report on Road Improvement and Maintenance Project, Nevis -St. Kitts and Nevis.
Validation of Project Completion Report on Sites and Services – Grenada. Assessment of Effectiveness of Implementation of Poverty Reduction
Strategy 2004-09.253 Oct. 2012 Assessment of Extent and Effectiveness of Mainstreaming Environment,
Climate Change, Disaster Management at CDB.254 Dec. 2012 Assessment of the Implementation Effectiveness of the Gender Equality
Policy and Operational Strategy of the Caribbean Development Bank. Validation of Project Completion Report on Enhancement of Technical and
Vocational Education and Training – Belize. Validation of Project Completion Report on Fourth Road (Northern Coastal
Highway Improvement Section 1 of Segment II) Project – Jamaica. Assessment of the Effectiveness of the Policy-based Lending Instrument.
256 May 2013 Validation of Project Completion Report on Expansion of Grantley Adams International Airport – Barbados.
234
Validation of Project Completion Report on Fifth Water Supply Project – Saint Lucia.
261 May 2014 Validation of Project Completion Report on Immediate Response Loan, Tropical Storm Gustav, Jamaica.
Validation of Project Completion Report on Social Investment Fund, Jamaica.
Validation of Project Completion Report on Disaster Mitigation and Restoration – Rockfall and Landslip, Grenada.
263 Oct. 2014 Validation of Project Completion Report on Basic Education Project – Antigua and Barbuda
263 Oct. 2014 Approach Paper for SDF 6 & 7 Multicycle Evaluation
264 Dec. 2014 Validation of Project Completion Report on Policy-Based Loan – Anguilla Validation of Project Completion Report on Immediate Response Loan -
Tropical Storm Arthur – Belize. Evaluation of Technical Assistance Interventions of the Caribbean
Development Bank Related To Tax Administration and Tax Reform in The Borrowing Member Countries 2005-2012.
265 March
2015
Approach Paper for the Evaluation of Policy Based Operations
266 May 2015 Validation of Project Completion Report on Upgrading of Ecotourism Sites – Dominica
The Evaluation of the Caribbean Development Bank’s Intervention in Technical and Vocational Education and Training (1990-2012)
267 July 2015 Validation of Project Completion Report on The Belize Social Investment Fund I Project − Belize
268 Oct.2015 Approach Paper Country Strategy and Programme Evaluation, Haiti
The review and analysis of these documents is based on the UNEG Quality Checklist for Evaluation Reports (http://www.uneval.org/document/detail/607) as well as on ECG guidance (Big Book on Good Practice Standards).
Approach Papers
Three Approach Papers (APs) were made available to the panel (see Table [ref] above). An AP describes the rationale for the evaluation, the background to the topic evaluated, the evaluation framework (criteria and questions) and approach. It also describes the team and provides an initial planning. Being the first main deliverable of OIE’s evaluation process, APs are the starting point and therefore a major determining element in the roll-out of each evaluation. Therefore APs “have to get it right”.
The APs examined are clearly written, well-structured and of reasonable length.133 We were surprised to find, however, that they do not make explicit the objectives of the evaluated intervention(s), e.g., through a clear objective tree, or through an explicit theory of change, intervention logic or logframe. Whilst one of the APs contains, in an appendix, a results framework for the evaluation, the results framework for the intervention (PBO) itself is lacking.
Inception reports
Only one Inception Report was given to the Panel for review (SDF 6&7). This gives an in-depth description of the evaluated programme and provides a clear Theory of Change. It is good
133 Opportunities remain of course to be more concise and to move parts to appendices, e.g., detailed descriptions of the evaluation team or part of the description of the evaluated intervention.
235
practice that this is established after a pilot field mission, which helps to amend the initial AP on the basis of field observations and sharpen the evaluation questions if needed.
However, it is still considered to be good practice to have the Theory of Change elaborated in the initial design documents . This would facilitate OIE evaluations after project completion. Establishing the Theory of Change of any intervention would be included in the QaE form more explicitly, to be developed between the Quality unit referred to above, and OIE.
Evaluations and Assessments
Three evaluations and four assessment reports completed during the review period were considered. Assessments are similar to evaluations but have a narrower scope; they focus on a limited set of aspects or judgment criteria, mainly effectiveness, i.e. achievement of objectives. Evaluations generally base their judgment on the internationally recognised DAC criteria as well as aspects of the CDB and BMC’s management of the intervention.
In general, these reports are of reasonable quality. In the main, they explain the evaluated object134 and provide evaluation objectives. The findings are organised around the evaluation criteria or questions detailed in the scope and objectives section of the report. They are based on evidence derived from data collection and analysis methods as described in the methodology section. The reports tend to dwell on the limitations that the evaluation encountered, but without becoming defensive. In one case (PBL Assessment) the report starts with a summary of the reviews on the topic done by other MDBs. This was a pleasant surprise and indeed a good practice that could well be adopted in future evaluations too.
However, the reports also show several significant weaknesses:
- Reports do not always provide clear (reconstructed) intervention logics or theories of change for the intervention(s) evaluated.135 Evaluation criteria and questions are defined at a fairly general level. They are translated into more precise “research questions” (in an “Evaluation Design Matrix”, for each project for each criterion). However, it is unclear how these questions relate to the intervention logic (as this is not made explicit). This may be done in inception reports (of which, as noted above, only one was available for review), but should be done also in the final reports.
- The reports do not describe the link from the evaluation questions to the answers, how the evaluation judgments are made and how these ultimately transform into ratings for each criterion and each project. In other words, the explanation provided in the evaluation frameworks is inadequate. The “evaluation design matrix” currently used does not provide sufficient insight into how ultimately an intervention’s performance is judged.136 Links between findings, conclusions and recommendations could be improved by making this more explicit. In other words, reports should include the story on how the evaluand is credibly linked to any observed outcomes and impacts, and should be clear on how causal claims are made.
- With the exception of the PBL Assessment, reports are lengthy and detailed. One reason for this is an over-emphasis on ratings. Their detailed discussion, project by project, criterion by criterion, occupies a very prominent position in the evaluation reports’ main body of text. Although ratings are traditionally an important element in evaluations of MDBs, too strong
134 Sometimes in great length: for instance with the SDF 6&7 multicycle evaluation report it is only at page 30 that we find the beginning of the report on findings…135 Again with the SDF 6&7 evaluation, it is said to be guided by a “Logic Model” which is not explained.136 Marlène: I moreover have the idea that the methodology (often described as “visits”) is based on interviews and little hard evidence. Any view on this?.JM: My “interview-based evaluations”!!
236
an emphasis can be tedious and may distract the reader from the real lessons to be drawn. The detailed discussion of ratings, and their evidence base, would be better placed in an Appendix, with a brief summary in the main report. This would help give the lessons and recommendations a more prominent position than is now the case. This would also help make the evaluation reports not only shorter but also more interesting to read; this could help add value to evaluation’s image within the organisation.
- The reviewers feel that the OIE evaluations tend to over-emphasise objective-based evaluation137 and the DAC criteria to the exclusions of considering other evaluation approaches such as Developmental Evaluation (Patton, 2010138); evaluation should be case specific and answer the actual information needs of managers and other decisions makers rather than always concentrating on final performance.
- Related to the previous point (and again with the exception of the PBL Assessment) executive summaries (approximately 8 pages) are too long. For the evaluation report to increase potential impact, they would need to be reduced to 2 to 3 pages and be more focused; again this could be done by dwelling less on the individual ratings of projects and more on key findings, lessons and conclusions. More generally, reports could be better adapted to the needs of the different audiences. Although not strictly limited to evaluations, The Health Evidence Network Reports139 are a model that could be adapted for evaluation reporting purposes; they are specifically geared towards addressing policy and decision-making.
- The “Recommendations to BMCs” are an interesting feature of the reports, (although we are unsure to what degree such recommendations could be effectively followed up by OIE or the Bank, but certainly could taken up with BMC Board members.
- Reports (e.g. the evaluation report on Technical Assistance) focus much on technical problems that were encountered during the evaluation. Although these are important issues, again to improve the report’s flow and “readability” this section would be better placed in the Appendix. What counts is the story of the intervention, not the story of the evaluation (see “Limitations” section in the TA report for instance)
OIE Validations of Project and Country Strategy Programme Completion Reports (referred to globally as PCRs hereafter)
As said above, the OIE has the mandate to validate the Project and Economic departments PCRs and CSPCRs. However, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
The validations tend to repeat the different items reported in the PCRs and then provide extensive comment on each. The PCVRs go into great depth and detail, which makes the documents rich and complete. This is their strength – but also their weakness. The depth and
137 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.138 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press139 See the reports available at the WHO’s Health Evidence Netowkr at http://www.euro.who.int/en/data-and-evidence/evidence-informed-policy-making/health-evidence-network-hen
237
level of detail, as well as the repetitions from the original PCRs, makes PCVRs (overly) lengthy (20-40 pages) and difficult to read. The OIE reported spending approximately 27.2% of its time on validating PCRs in 2015 compared with 44.4% on its core work, i.e. doing or managing the higher level evaluations. That is more than half of its evaluation work is being spent on the validation process. Finally, the PCVRs now seem to be, to a great extent, a standalone output of OIE. It is not always clear to us how they are being used as the “building blocks” for the OIE’s independent evaluations. Making this clearer in the independent evaluations would help show the link and therefore the value of the time being spent on the self-evaluation validations.
To conclude, the review finds that the OIE has taken steps to improve the perceived utility of evaluation in several ways. In the first instance, by planning its work to provide relevant and timely evidence geared towards helping the Board with its oversight and decision making tasks. The topics are selected through dialogue between the OIE and key CDB stakeholders and reflect priorities of the CDBs strategic plan. Secondly, by securing the interest and consequently the buy-in of the OAC and CDB senior management through engaging their input throughout the evaluation process. This is evidenced by the reported interest in the latest three studies, the Country strategy programme in Haiti, the evaluation of policy-based operations and the SDF 6& 7 multicycle assessment.
The OIE products are of an acceptable quality and could be even better if some of the shortcomings were addressed. However, the products themselves do not impair the utility of OIE’s work; this is undermined in several ways: (1) by the time delays in commenting on PCRs (OIE) and providing feedback to the independent evaluations (operations and management) (2) by the inefficient processes for agreeing topics and funding sources as well as providing OIE with management responses to its reports.
Putting Evaluation to Use: transparency, feedback and follow-upThere are several ways that evaluation can be, and is being used. As John Mayne has pointed out in his many publications on the issue,140 when we talk of evaluation use, we are mainly thinking about its Instrumental use—use made to directly improve programming and performance. But there is also conceptual use - use which often goes unnoticed or more precisely, unmeasured. This refers to the kind of use made to enhance knowledge about the type of intervention under study in a more general way. Or even Reflective use— this refers to using discussions or workshops to encourage and support reflection on the evaluation findings to see how they might contribute to future strategies.
In the case of the CDB there is some evidence to suggest that “use” is not only instrumental, but other types are also developing. For example, in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessons. (Reflective use)
Another important use, as recommended by the ECG, is that from time to time a synthesis of lessons is drawn from a number of evaluations and made available publically. In fact the Panel was impressed to hear that in the past, the evaluation unit had done this drawing on lessons from evaluations of the power sector. (Conceptual use) Although nothing has happened since, it is now on the “to do list” for 2016 (OIE’s 2016 Work Plan).
As for instrumental use, responsibility for using the knowledge generated through evaluation and for possibly drawing up an action plan of what should be done is up to CDB senior management and the relevant CDB department and division. Oversight on applying recommendations and picking up on the lessons drawn is the responsibility of the OAC.
Evidence on how evaluations have actually contributed to decisions or negotiations is lacking or confusing, Certainly the OIE is unaware of the extent to which its evaluations are put to use. On
140 See for example, his opening chapter to Enhancing Evaluation use: Insights from internal Evaluation Units, Läubli Loud, M. and Mayne, J. 2014, Sage Publications
238
the one hand, the OAC minutes sometimes indicate that lessons learned are integrated into the next phase. On the other hand, the reviewers were told that often in the past, the evaluation results were “too old” to be of use as the lessons had already been drawn and used way before the report was completed. Similarly, people’s gaps in memory on how well the evaluative information from previous studies may have been used may also account for the scarcity of evidence.
In response, the Panel questioned CDB staff and the OIE about a particular study, the Technical and Vocational Education and Training Assessment. The feedback was somewhat contradictory. On the one hand, the study was criticised as “confirming” news rather than bringing “new news”. However, on the other, we learned that In October 2015, the Board of Directors approved a proposal for the revision of CDB’s Education and Training Policy and Strategy. Work on this has already begun and an external consultant has been engaged to lead the process.
Although it is one of the OIE’s tasks to set up a database on results and lessons learned from evaluations, so far this has not been a priority. There is also currently no systematic tracking of lessons or recommendations arising from the evaluations, or on any progress in their uptake. (The Panel has already referred above to OAC’s lack of oversight in the use of evaluation.)
The OIE’s role in supporting CDB’s organisational learning is clearly specified in the Evaluation Policy, with many good suggestions for knowledge sharing activities such as “brown-bag lunches, workshops, pamphlets and short issues papers” (p. 19). So far, however, the OIE’s lead role on the knowledge sharing side appears to be quite limited. It has provided advisory input in Loan Committee discussions, and organises workshops together with the relative operations department for discussing the implications of evaluation studies. Ultimately, of course, the uptake of evaluation results and knowledge is in the hands of management. But the evaluation unit has an important role to play in terms of knowledge broker and knowledge manager. Both have tended to be underplayed in OIE’s work plan so far.
Transparency: The Communication Strategy
In recent times and with the approval of its new Disclosure Policy, the CDB has started to post its independent evaluation reports on its website. (There is nothing on the self-evaluations). The website also presents a good overview of the role and function of the OIE and evaluation within the CDB. This is a step in the right direction for sharing information. However, in our view, the CDB’s communication strategy is the weakest part of the evaluation system to date.
The Panel has already commended the OIE in its efforts to engage the CDB and the OAC in evaluation work. But reporting and communicating the lessons seem to be entirely targeted at the Board and the CDB. Moreover, the 2015 budget provides only US$2’000 for communication – nothing of which is intended for outreach.
Reviewers feel that actively engaging with the more indirect stakeholders, for example project implementers in the BMCs, NGOs or project beneficiaries is relatively weak141. There appears to be little reflection on drawing out significant messages for the broader group of stakeholders, or on how then to transmit them to the “right” people in the “right” way (knowledge brokerage).
To conclude, evidence on the uptake of evaluation is either confusing or sparse. It is unfortunate that so far no systematic record keeping system has been put into place to track lessons learned or the uptake of recommendations (or actions agreed from management responses). The OIE plays a weak role in brokering the knowledge generated through evaluations to the benefit of external partners and in managing such knowledge. Although the Evaluation Policy specifies the need for “distilling evaluation findings and lessons learned in appropriate formats for targeted audiences both within and outside the CDB” (p.19) such a targeted communication strategy has yet to be developed and budgeted.
141 A broader communication strategy is one of the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
239
Strengthening Evaluation Capacities and Networking From the onset in 2012, the OIE has stressed the importance of developing and strengthening evaluation capacities within the OIE, the CDB and, subject to available resources, in borrowing member countries. Building evaluation capacity in BMCs and the CDB is one of the OIE’s mandated tasks. It has been a priority that figures on the work plan from the beginning (Work Programme and Budget 2012-2104) The idea of developing an internship programme for graduates from the Caribbean region was one idea that was advanced to help build local evaluation resources. However, the capacity-building has primarily been focused on OIE and CDB staff to date. One of the OIE’s two objectives for 2015 therefore was to take up the challenge and “strengthen evaluation capacities and networking” to include reaching out to the BMCs.
Developing OIE staff capacities
The change from project level to strategic and thematic evaluations does require different evaluative skills and competencies. The MDB Evaluation Pyramid presented below in Figure 3 shows the different types of evaluation and changing resource needs as one ascends the pyramid. Implicit here also is the change in the type of expertise and competencies needed as evaluation aspires to the higher levels.
Consequently for 2015, the OIE set itself the objective of networking and developing working partnerships with regional and international evaluation entities and academic institutions. The rationale was twofold: (1) secure further support and guidance as well as (2) increase its outreach and coverage through joint work and international exposure. Another implicit aim was to benefit from partners’ contacts in the BMCs wherever possible so as to improve data collection and quality.
Figure 3: The MDB Evaluation Pyramid142
142 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
240
The OIE has therefore linked up with Carleton University in Canada and the University of the West Indies, Barbados campus. The OIE was also approached by the Development Bank of South Africa to exchange experiences about setting up an evaluation entity in a “small” development bank. However, its attempt to become a member of the Evaluation Cooperation Group was not successful for reasons beyond its control.
The OIE is to be commended in addressing the issue of staff competencies and professional development more generally. New developments in evaluation as well as new developments in the scope of OIE’s work may necessitate new competencies. For this reason, organisations such as the International Developmental Evaluation Association have recommended that the competencies of evaluators and evaluation managers should be periodically reviewed. Several publications now exist on competency requirements and suggestions for the periodic review of staff competencies.143
It is not within this remit to compare and contrast OIE’s competencies with those recommended by international and national agencies. However, what we can say is that the OIE demonstrates great forethought in taking this on board.
Capacity building within CDB
The OIE’s objective also consists of continuing to develop measures for improving the monitoring and self-evaluation side of CDB’s work. OIE’s strategy here is to use the windows of opportunity on offer through some of the training sessions that are being organised by CDB as part of its shift towards MfDR e.g. by Corporate Planning Services and Technical Assistance. For 2016 it is also planned to have the OIE present at the annual staff meeting and Learning Forum.
The OIE also organises some ad hoc training with operations, for example to help understand new tools e.g. for drawing out lessons from self-evaluation reports and, more generally, in helping staff appreciate how evaluation can add value to the organisation’s work. Measures include providing advisory services on demand, and providing training alongside the introduction of new or revised tools.
Capacity building in the BMCs
This is an ambitious task and would require additional investment; from the bi-annual work plans to be effective. A modest attempt has been made in 2015; from what we understand, the OIE has joined together with the Carleton University and the University of the West Indies, using their networks in some of the BMCs, to try to develop this aspect.
To conclude, we cannot comment on the quality or reaction to such training, but can commend the OIE for making capacity building one of its priority objectives. From both the Policy and the documents we reviewed, we note that capacity building was always seen to be an important aspect of OIE’s work, but hitherto has received little strategic focus. But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.
Adequacy of the OIE’s human and financial resources to support its work
OIE’s Human Resources;
The OIE is has a staff of 5; the head, 1 senior evaluation officer and two evaluation managers, plus one administrative assistant. Three of the five were recruited from within the CDB. The
143 E.g. IDEAS, (2012) Competencies for Development Evaluation Evaluators, Managers and Commissioners, the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice (2010) and the Swiss Evaluation Society’s Evaluation Managers Competencies Framework (2014)
241
limited capacity means that it is not feasible to cover all the types of evaluation activities outlined in the Evaluation Policy. Yet there is some indication from the Board that OIE should embark on impact evaluations at some future stage. An increasing demand for evaluation and for impact evaluations in particular, would run the risk of overstretching the OIE’s capacity to deliver credible and useful evaluations. Moreover, there are many other designated OIE activities that should be recognised as valuable work; the validations, building CDB and BMC evaluation capacity, providing supervision, advice, knowledge management and brokerage as well as managing evaluation contracts, The time needs of dealing with all of these may be underestimated in OIE’s budgets; all are important for assuring best value from evaluation. The Panel is concerned that a demand for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level evaluations may undermine the importance and time needs of other essential tasks.
Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses.
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.
242
Self- and independent evaluationSelf-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes. Both types of evaluation are important as they are at the very heart of the evaluation function; they are said to be the building blocks for the more strategic evaluations that the OIE is now undertaking.
The Evaluation Coordination Group recommends that the self-evaluations be carried out by the relevant operations department and in turn, reviewed and validated by the organisation’s independent evaluation office. The CDB’s Evaluation Policy therefore talks of “validating all self-evaluations” as being one of OIE’s essential oversight tasks.
Within CDB, the self-evaluations should provide management with performance assessments and thereby serve an accountability function to the CDB and Board. To support the process, the OIE provides operations with manuals and checklists for guidance. Once a self-evaluation report is to hand, it is given over to the OIE for the validation of its technical quality and credibility.144
However, in the CDB case, there are well-documented issues that have affected the quality and timeliness of the self-evaluations on the one hand, and therefore the quality of the foundation on which to build the independent evaluations. Paucity of documentation within CDB, paucity of data collected and available in the Borrowing Member Countries (BMCs), time delays in producing completion reports and in turn, having them validated by the OIE - all such issues were systematically raised during interviews and in some of the independent evaluation reports. There appears to be little incentive to complete self-evaluations in a timelier manner.
Generally speaking, many of the monitoring data problems appear to be due to a lack of management oversight. For example, with the introduction of results-based management, the logic frame and monitoring and data needs are systematically being built into intervention design. However, the BMCs are not delivering the data as contractually agreed at the outset. Incentives to support any significant change towards building a results-based culture seem to be weak and sanctions seem to be rarely enforced when the supply of data is lacking or lengthy delays to the projects occur. Although we can appreciate the complexities of trying to enforce monitoring compliance, this means that often, project deadlines have had to be extended, data gaps are not being satisfactorily dealt with and in turn, there has been a void in the quality and quantity of available evidence for the CDB’s self-assessment of project performance. For some time, this lack of oversight has been tolerated. Part of the problem is the low priority accorded to completing the self-evaluation reports by operations, coupled with the absence of any focal point within senior management to drive the process and deal with the problems.
No record is kept of how the self-evaluation results are actually used. They do not appear on the CDB website, but we were told that the findings are integrated into the following project designs. Hence we are somewhat unclear as to the utility of these reports at present. The situation is exacerbated by a rather confused image of evaluation: some operations staff consider OIE’s input (through validations or independent evaluations) to be sometimes over-critical, regulatory and adding little value; it is a threat rather than an opportunity for learning. Yet at the same time, evaluation is recognized as an integral part of result-based management.
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.
Some managers however seem to start changing the status quo. For example a revised and simplified template for producing project completion reports is being considered, and mid-term
144 According to the Evaluation Policy, OIE should validate all PCRs and CCRs but due to the backlog of reports and the delay in completing them (sometimes years later) since October 2015, the OIE has secured OAC agreement to validate a maximum of 6 per year, which are selected in consultation with the OAC.
243
project reviews are expected to be more stringent in looking at monitoring plans and practices and tying disbursements to performance. In some cases we also learned of incentives being introduced to encourage project managers to complete their reports in a timelier manner. But much remains to be done and, since the OIE is no longer responsible for monitoring and project evaluations, there is a void that needs to be filled. It is up to line managers to drive this work forward.
To conclude, it is fair to say that in view of a number of “frustrations” between the OIE and operations, which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring data, the added value that evaluation might offer to the operations area is ill recognized. Moreover, the link between self-evaluation as the building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
244
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”145
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
145 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
245
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”146 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 147 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
146 CDB (2011) Evaluation Policy (p.2)147
246
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
247
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
248
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
249
mendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
250
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
251
APPENDICES
Appendix I - The External Review Mandate – Terms of Reference and Approach Paper
Appendix II -Review Approach, Data collection and Analysis, and Limitations
Appendix III – Overview of OIE Evaluation Practice
Appendix IV - List of Persons Interviewed
Appendix V - List of Documents Reviewed
Appendix VI- List of Topics used to guide interviews with members of CDB Board of Directors
Appendix VII - List of Topics used to guide interviews with CDB staff
252
Appendix III – Overview of OIE Evaluation Practice (prepared by the OIE in response to Reviewer’s request)
Caribbean Development Bank, Office of Independent Evaluation - OIE
Category Response
Percentage of projects subject to project (self-) evaluation
100% - Project Completion Reports (PCR)
Percentage of projects subject to validation by OIE
Approximately 40-50%
About 15 projects exit portfolio annually. Evaluation Policy calls for all PCR to be validated. However, OIE resources insufficient. Validation process reviewed in 2014. Now OAC (Board committee) selects a sample of 6-8 PCR for validation each year.
Percentage/number of projects subject to in-depth review by OIE
None – unless specifically requested by OAC
Due to limited resources, focus of OIE evaluation work programme is on PCR validations and high-level evaluations – including country strategy and programme evaluations (CSPE).
Number of high-level evaluations conducted by OIE (e.g. sector, thematic, geographic)
1-2 per year since 2011
Plan is 2-4 per year from 2016. This would include CSPE (1st planned for Q1 2016: Haiti)
Number of project impact evaluations conducted by OIE
None
OIE includes “impact questions” in high-level evaluations.
Number of project impact evaluations conducted by Bank staff or other non-OIE staff
OIE is not aware of any impact evaluation conducted by the Bank.
However, OIE provides technical support to the Basic Needs Trust Fund (BNTF) in its design of an M&E framework that entails impact evaluations.
Budget In USD mn: 0.78 in 2015; 0.82 in 2016. This is equivalent to about 2.5% of total CDB Administrative Budget.
75% of the budget is for Staff salaries (4 Professionals, 1 Support staff), leaving around USD 190,000 (in 2015) for other expenses, including consultants e.g. for external evaluations. Additional funding is accessed via the Special Development Fund (SDF). This varies according to type and scope of the evaluation, e.g. the ongoing SDF 6/7 Evaluation is SDF funded at USD 255,000.
Budget determined by Board, not separate from administrative budget.
SDF funding for evaluations is considered separately and subject to Bank internal approval process. SDF funding cannot be used to cover OIE expenses such as staff time or travel. Country eligibility for SDF funding is also a consideration. OIE expressed concerns about this funding track in respect to predictability, independence and eligibility limitations.
Head of OIE reports to Board, with administrative link to the President
Terms of appointment for Head
253
5 year term, renewable once. Appointed by the President with the agreement of the Board.
Right of Return for Head Not eligible for other staff positions.
Consultants as proportions of OIE budget
2015: 19% (USD 145,000)
Plus SDF funding. SDF funded evaluations are outsourced.
Last external evaluation (or peer review) of OIE
No external evaluation, though a review of the function was done in 2011, leading to the Evaluation Policy.
OIE External Review completed in April, 2016
Departments or special programmes supporting impact evaluation
None
254
Appendix IV – List of Persons Interviewed
Name Function relative to OIE Type interview
Mrs. Colleen Wainwright Member CDB Board of Directors (UK)
Face to face
Mrs. Cherianne Clarke Alternate MemberCDB Board of Directors (UK)
Face to face
Mrs. Jean McCardle Member CDB Board of Directors (Canada)
Face to face
Dr. Louis Woodroofe MemberCDB Board of Directors (Barbados)
Mr. A: de Brigard Former Member CDB Board of Directors
Skype interview
Mr. H. Illi Fromer Member CDB Board ofDirectors
Telephone interview
Mrs. Claudia Reyes Nieto Member CDB Board of Directors
Telephone interview
Mr. Bu Yu alternate DirectorCDB Board of Directors
Face to face
Mr. Michael Schroll(Barbados)
Head OIE
series of interviews viaSkype and face-to-face
Mr. Mark Clayton OIE Senior Evaluation Officer Focus GroupMrs. Egene Baccus Latchman OIE Evaluation OfficerMr. Everton Clinton OIE Evaluation OfficerMrs. Valerie Pilgrim OIE Evaluation Officer
Dr. Justin Ram CDB Director Economics Department
Face to face
Mr. Ian Durant CDB Deputy Director Economics Dept Face to faceDr. Wm Warren Smith CDB President
Joint interviewFace to face
Mrs. Yvette Lemonias-Seale CDB Vice President Corporate Services & Bank Secretariat
Mr. Denis Bergevin CDB Deputy DirectorInternal Audit
Face to face
Mr. Edward Greene CDB Division Chief, Technical Cooperation Division
Face to face
Mrs. Monica La Bennett CDB Deputy Director Corporate Planning Face to faceMrs. Patricia McKenzie CDB Vice President Operations Face to faceMs. Deidre Clarendon CDB Division Chief
Social Sector DivisionFace to face
Mrs. Cheryl Dixon CDB Co-ordinator, Environmental Sustainability Unit
Focus group
Mrs. Denise Noel- Debique CDB Gender Equality Advisor Mrs. Tessa Williams-Robertson CDB Head Renewable EnergyMrs. Klao Bell-Lewis CDB Head Corporate Communications Face to faceMr. Daniel Best CDB Director
Projects DepartmentFace to face
Mr. Carlyle Assue CDB Director Face to face
255
Finance Department
256
Appendix VI - Interview Guide: Members of CDB Board of Directors
Below are a list of themes that I should like to raise with you based on your experience and knowledge of the CDB’s independent evaluation function (Office of Independent
Evaluation).
In each case, I should be grateful if you could illustrate your responses with examples or help this Review by, wherever possible, sending me (or telling me where I can find) any
documents that could support your responses.
This guide is being sent to you in advance to help prepare our meeting. However, our interview will be conducted more in the style of a conversation. The following sub-questions will be used to GUIDE the interview. Please feel encouraged to raise any
additional issues that you feel we should take into account
On the governance and Independence of CDB’s evaluation functionWhat mechanisms are there in place to support its independence?
How satisfactory are the current arrangements in your opinion?
How is the balance between independence and the need for interaction with line management dealt with by the system? For example, what mechanisms exist to ensure that the OIE is kept up to date with decisions, policy / programme changes, other contextual changes etc that could have an affect on OIE evaluation studies / evaluation planning?
On the OIE’s Evaluation PolicyThe CDB’s Evaluation Policy was established in 2011. To what degree do you feel it is adequate? Still relevant?
What suggestions do you have for any improvements?
In your opinion, how adequate is the current quality assurance system for over viewing the evaluation function?
On the quality and credibility of evaluation studiesTo what degree do you believe the reports are fair and impartial?
Do you consider them to be of good quality? Are they credible?
Are you adequately consulted/involved on evaluations of interest to you?
On the relevance and usefulness of evaluations How well does the OIE engage with you / your committee during the preparation, implementation and reporting of an evaluation study to assure that it will be useful to the CDB?
How are the priorities set for the independent evaluations? What criteria are used? Are you satisfied with the current procedure?
257
When OIE evaluation studies are outsourced to external consultants, what criteria are used to make this decision?
How are the priorities for the OIE’s 3.year rolling work plan agreed? In your opinion, is the current plan adequate in terms of coverage and diversity?
In your opinion, do the evaluations address important and pressing programs and issues?
To what extent do you feel that the OIE’s evaluations integrate the cross-cutting theme such as gender, energy efficiency/renewable energy, climate change? What improvements might be made and how?
On the dissemination and uptake of evaluation findings and recommendationsTo what extent do you feel that evaluation findings are communicated to the CDB and its stakeholders in a
a) useful, b) constructive andc) timely manner?
Are evaluation recommendations useful? Realistic?
What mechanisms are in place to assure that evaluation results are taken into account in decision making and planning? What improvements do you feel could be made?
How have you used the findings from any evaluations? Examples?
To what degree do you feel that evaluation contributes to institutional learning? And what about to institutional accountability? Any examples?
What mechanisms are in place to ensure that knowledge from evaluation is accessible toCDB staff and other relevant stakeholders? Are the current arrangements satisfactory?
How satisfied are you with current arrangements? What expectations do you have for the future?
On resourcesHow is the OIE resourced financially and is this satisfactory?
What about the OIE staff, are all the important areas of expertise represented in the team?
On this Review of the Office of Independent EvaluationWhat are your expectations? What are you particularly hoping to learn from it?
Thank you very much for your cooperation and input
258
Appendix VII : Interview Pro-Forma – CDB Staff membersThis presents a list of the topics raised during interviews. It was used to guide the open-ended
discussion – this means that the sequence and exact wording of the questions may not necessarily have followed in this order or been asked in exactly this way.
Changeover to an Independent Evaluation Office? Expectations? Advantages and disadvantages??
Satisfaction with working relations between operations and the OIE from your perspective?
Process of dealing with the PCRs and CCRs? Advantages and limitations?
Quality and credibility of the validation process?
How are the self-evaluation reports used?
Credibility and Quality of OIE’s evaluation reports
Communication of self and OIE independent evaluations? To whom, in what way? Possible improvements?
Independence of the Office of Independent Evaluation (OIEIndependence is absolutely central to the integrity and trustworthiness of evaluation. It is an agreed requirement within the development agencies and in the evaluation community as a whole. In examining the issue of independence and good practice, reviewers are guided by the Evaluation Cooperation Group’s recommendations on good practices, the CDB’s Evaluation Policy and by the 2011 consultancy review of independence relative to the CDB’s evaluation and oversight division148. The appraisal is based on a comparison of the ECG’s recommendations on independence149 and the current OIE status.
OIE and Independence: Recommendations from the OECD Evaluation Cooperation Group (ECG)
The ECG’s considers the issue of independence according to three specific areas: organisational, or structural independence, behavioural, or functional independence and protection from outside interference, or operational independence.
148 Osvaldo Feinstein & Patrick G. Grasso, Consultants, May 2011 Consultancy to Review the Independence of the Evaluation and Oversight Division of the Caribbean Development Bank149 ECG 2014 Evaluation Good Practice Standards, Template for Assessing the Independence of Evaluation Organizations, Annexe II.1
259
Organizational independence, ensures that the evaluation unit and staff are protected against any influence or control by senior or line management, and have unrestricted access to all documents and information sources needed for conducting their evaluations. Also, that the scope of evaluations selected can cover all relevant aspects of their institution.
Behavioural independence, generally refers to the evaluation unit’s autonomy in selecting and conducting setting its work programme and in producing quality reports which can be delivered without management interference.
Protection from outside interference refers to the extent to which the evaluation function is autonomous in setting its priorities, and conducting its studies and processes and in reaching its judgments, and in managing its human and budget resources without management interference.
Conflict of interest safeguards refers to protection against staff conflict of interests be they current, immediate, future or prior professional and personal relationships and considerations or financial interests for which there should be provision in the institution’s human resource policies.
The OIE’s Independence in Practice
Organisational / structural independenceOn the whole, the Panel acknowledges and commends the efforts being made by the CDB to assure OIE’s organisational independence. The CDB’s Evaluation Policy provides for the OIE’s organisational independence from line management and the interview data suggests that there is also wide acceptance and acknowledgement of why the OIE should have such independent status. Table 1 below provides our overall assessment of this aspect of OIE’s independence when compared with ECG recommendations. 150
Table 1: OIE organisational independence compared with ECG recommendations
Aspects Indicators CDB Evaluation Policy (EP) and Practice
The structure and role of evaluation unit
Whether the evaluation unit has a mandate statement that makes clear its scope of responsibility extends to all operations of the organization, and that its reporting line, staff, budget and functions are organizationally independent from the organization’s operational, policy, and strategy departments and related decision-making
Partially Complies The Policy is broad enough to cover the full range of MDB type of evaluations. However in practice this would not be possible without additional human and budget resources
The unit is accountable to, and reports evaluation results to, the head or deputy head of the organization or its governing Board
Whether there is a direct reporting relationship between the unit, and
a) the Management, and/or
b) Board or
c) relevant Board Committee, of the institution
Complies - OIE reports to the Board of Directors (BoD) through its Oversight Assurance Committee (OAC)
The unit is located organizationally outside the staff or line management function of
The unit’s position in the organization relative to the program, activity or entity being evaluated
Complies - The OIE is located outside, and is therefore independent of CDB line management
150 Based on ECG (2014) Template for Assessing the Independence of Evaluation Organizations, Evaluation Good
Practice Standards, Annexe II.1
260
Aspects Indicators CDB Evaluation Policy (EP) and Practice
the program, activity or entity being evaluated
The unit reports regularly to the larger organization’s audit committee or other oversight body
Reporting relationship and frequency of reporting to the oversight body
Complies - The OIE reports x 5 per year to the OAC . Board approval for an additional executive meeting between the Head of the OIE and the OAC at least once per year was given in October 2015
The unit is sufficiently removed from political pressures to be able to report findings without fear of repercussions
Extent to which the evaluation unit and its staff are not accountable to political authorities, and are insulated from participation in political activities
Complies
Unit staffers are protected by a personnel system in which compensation, training, tenure and advancement are based on merit
Extent to which a merit system covering compensation, training, tenure and advancement is in place and enforced
Partially Complies - with CDB human resource policy. However the skill needs of OIE staff ought to be regularly reviewed in light of its move towards higher-level evaluations. Appraisal of skill needs and hiring of relevant staff should be completely under the authority of the Head of Evaluation. This is not sufficiently clear in the Policy or other documents we reviewed.
Unit has access to all needed information and information sources
Extent to which the evaluation unit has access to the organization’s
a) staff, records, and project sites;
b) co-financiers and other partners, clients; and
c) programs, activities, or entities it funds or sponsors
Complies –The available evidence suggests that there is no reason to doubt such access. But systematic and easily accessible documentation is lacking in the CDB; it is one of its weak points.. Delays in getting hold of the relevant documents can have consequences on the timeliness of evaluation studies
However, independence should not mean isolation: There appears to be a detachment between the OIE and CDB that is of concern to the Panel; on the one hand, between the OIE and operations staff, and (2) on the other, in terms of the structural arrangements between the OIE and senior management.
14) In agreeing for the OIE to concentrate on strategic and thematic, in-depth evaluations, responsibility for project monitoring and evaluation were given over to operations. The division is clear and respected. However, it has its drawbacks. With the OIE no longer systematically involved at the front-end of project design, the monitoring data needs are likely to be poorly defined. Weak monitoring data will contribute to weaker evaluations. (More on this point under the heading self and independent evaluations.)
In the reviewers’ opinion, it is a common misunderstanding to assume that providing evaluator advice on monitoring and evaluation data will comprise evaluator independence. On the contrary, evaluation input into project design is essential to assure that the logic, indicators and data needs are addressed so that at some future point in time an evaluation of the achievements can be empirically grounded.
261
This is not to say that the OIE no longer has any influence at the front-end design stage; it has merely shifted the point of focus. The OIE is now systematically providing such input more generally to the corporate planning teams for the tools and systems they are developing to support the MfDR framework. The monitoring data for projects and their implementation should be improved once the Project Performance Evaluation System (PPES) and the Portfolio Performance Management System (PPMS) are updated and operational.
15) In the second place, the OIE has limited formal access to the Advisory Management Team (AMT) weekly meetings where the President and senior management gather to exchange up-to-date information on the dynamics of CDB policy and practice. The OIE is not regularly invited in any capacity to these meetings or given a copy of the agenda or minutes; the OIE is occasionally invited to attend in order to discuss an evaluation report or management feedback. For the OIE, this means that it is unlikely to pick up on the ‘when’ and ‘what’ of key decisional issues or provide input into the discussion based on evaluative information. Its observer status at Loans Committee meetings, or as a participant informer at the OAC and BoD meetings and discussions do not necessarily provide the same insight as to the dynamics of management actions and/or decisions. .
To respond to this situation, the President has agreed to meet regularly with the Head of the OIE in order to keep him up to date with CDB strategic thinking. This is a welcomed change.
OIE Independence and Behavioural Issues The Panel has concerns about some behavioural issues. For example, through both the interviews and documentary review, we learned of considerable delays in processing both the independent evaluation reports as well as OIE’s validation of the CDB’s self-evaluations. Delays are generally due to receiving feedback on the independent reports from first, the relevant operational department, then from the AMT, and then on providing the OIE with a management response that is initially drafted by operations staff before being reviewed by the AMT. (OIE reports cannot be submitted to the OAC without the relevant management response). This two-layer process for preparing submissions to the Board is inefficient and could potentially be a threat to evaluation’s independence in the future by delaying OIE’s timely reporting to the OAC.
OIE validations of the CDB self-evaluations are also submitted to the OAC, but it is in both sides’ interest to clear up any misunderstandings beforehand. Despite attempts to improve the timeframe for completing these validations, delays are more the norm than the exception. Table 2 below summarises our assessment of the behavioural aspects of independence.
Table 2: OIE and Behavioural Independence
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Ability and willingness to issue strong, high quality, and uncompromising reports
Extent to which the evaluation unit:
a) has issued high quality reports that invite public scrutiny (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk) of the lessons from the organization’s programs and activities;
b) proposes standards for performance that are in advance of those in current use by the organization; and
c) critiques the outcomes of the
Partially complies – paucity of data and documentation sometimes hinder the quality of reports. The OIE emphasizes the learning part of evaluation, and is cautious in its criticism recognising that management is going through a transitory stage and can still be overly defensive.
262
Aspects Indicators CDB Evaluation Policy (EP) and Practice
organization’s programs, activities and entities
Ability to report candidly
Extent to which the organization’s mandate provides that the evaluation unit transmits its reports to the Management/Board after review and comment by relevant corporate units but without management-imposed restrictions on their scope and comments
Partially complies - as sometimes reporting to the Board is compromised by delays in the review/comment process between the OIE and the CDB. Any delay with the production of a Management Response will also mean that submitting a report to the Board in a timely manner is impaired since the two have to be submitted together.
Transparency in the reporting of evaluation findings
Extent to which the organization’s disclosure rules permit the evaluation unit to report significant findings to concerned stakeholders, both internal and external (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk).
Who determines evaluation unit’s disclosure policy and procedures: Board, relevant committee, or management.
Partially complies - The OIE’s conforms to the CDB’s disclosure policy. However, the dissemination of evaluation findings appears to be currently restricted to website publication and reports to the Board. A more targeted communication strategy to include other key stakeholders, e.g. project implementers in the BMCs should be developed and put in place.
Self-selection of items for work program
Procedures for selection of work program items are chosen, through systematic or purposive means, by the evaluation organization; consultation on work program with Management and Board
Complies - The OIE also ensures that its work program is drawn up after consultation with both CDB Management and Board to seek their input on relevant topics and themes.
Protection of administrative budget, and other budget sources, for evaluation function
Line item of administrative budget for evaluation determined in accordance with a clear policy parameter, and preserved at an indicated level or proportion; access to additional sources of funding with only formal review of content of submissions
Partially complies - The administrative budget for supporting OIE work is protected. Access to additional sources of funding is possible if well argued and justified. But the approval process is complex and inefficient. (See Figure 1 below)
OIE and Protection from External influence or interference
Our overall assessment is provided in Table 3 below. The OIE’s independence in the design, conduct and content of its evaluations does not appear to be subjected to any external interference. But securing funding from any sources outside the OIE’s administrative budget, i.e. from the Social Development Fund, is an unduly complex and long process. As such we consider that the current funding process can affect the OIE’s choice with regard to the type of evaluations it can undertake. (See Figures 1 and 2 below)
Table 3: OIE and its Independence from External influence or interference
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Proper design and execution of an evaluation
Extent to which the evaluation unit is able to determine the design, scope, timing and conduct of evaluations without Management
Complies – however within limits of restricted human and financial resources available
263
Aspects Indicators CDB Evaluation Policy (EP) and Practice
interference
Evaluation study funding
Extent to which the evaluation unit is unimpeded by restrictions on funds or other resources that would adversely affect its ability to carry out its responsibilities
Partially Complies - OIE must work within the limits of the agreed administrative budget wherever possible. If additional resources are needed for studies it must seek alternative funds elsewhere. The budget limitations can have an affect on the type of evaluations undertaken and therefore its independence in terms of choice.
Judgments made by the evaluators
Extent to which the evaluator’s judgment as to the appropriate content of a report is not subject to overruling or influence by an external authority
Complies – the evidence available suggests that the Board and Management accept the evaluators’ independent interpretation and conclusions Management responses are agreed to be the accepted place to raise any difference of opinion.
Evaluation unit head hiring/firing, term of office, performance review and compensation
Mandate or equivalent document specifies procedures for the
a) hiring, firing,
b) term of office,
c) performance review, and d). compensation of the evaluation unit head that ensure independence from operational management
Complies – the Head of OIE is appointed by the CDB President in agreement with the OAC for a 5 year period which is renewable x 1. The Head could be removed from Office by the President or the Board but only with the agreement of both parties.
However the Head reports to the President for all administrative and personnel matters. Even though this was not recommended in the Osvaldo Feinstein & Patrick G. Grasso report on Independence in 2011, the BoD accepted CDB’s reasons for keeping this arrangement. (e.g.most OAC members are non residents and cannot oversee day-to-day work)
. Extent to which the evaluation unit has control over:
a) staff hiring,
b) promotion, pay increases, and
c) firing, within a merit system
Partially complies - All OIE staff members are treated in the same way as other CDB staff. The Head has limited control over the hiring, firing or promotion of OIE staff.
Continued staff employment
Extent to which the evaluator’s continued employment is based only on reasons related to job performance, competency or the need for evaluator services
Partially complies - Whilst the EP is clear about procedures for hiring, firing and promotion, all of which must conform with CDB human resource policy, there is nothing mentioned about any difference of opinion between the CDB and the Head of the OIE with regard to continued staff employment subject to changes in the level of technical or interpersonal competencies needed to meet new demands.
Avoidance of Financial, Personal or Professional conflicts of interest
This particular aspect refers to the organisation’s Human Resources Policy; there must be provisions in place to protect against actual or potential conflict of interest. The Panel requested via the OIE, to have evidence from human resources on any such provisions but did not receive an answer. It must be assumes that this aspect of independence, past or present, does indeed form part of normal CDB Human Resource Policies
To conclude: The Panel is impressed with the measures CDB has taken to assure the organisational independence of the OIE. Its independent status is accepted and respected by
264
senior and line management. The OIE’s budget is not independent from the overall CDB administrative budget; this affects its choice of evaluation types or approaches. Some of the behavioural issues affecting independence were also of concern, especially due to the delays in the exchange of documents, between the OIE and operations departments, which has a direct effect on timely reporting to the OAC. As for protection from outside interference, our concerns are largely to do with OIE’s independence over staffing issue; there are potential loopholes in current arrangements that could undermine OIE’s autonomy over its staff.
OIE’s Strategy, Work Practices and Work ProgrammeThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. Much of the period 2012 to 2015 has therefore been taken up with preparing OIE’s shift in focus from project-based evaluations to the high-level thematic and in-depth strategic studies. This has meant adopting a three-way approach; (1) for self-evaluations, reducing its time input to support the process and (2) for independent evaluations, taking stock of the gaps in coverage and expertise, and (3) networking to share experiences with centres of expertise and align OIE with international practices. In addition, amongst other duties, it has been supporting the development of MfDR tools and systems such as the Project Performance Assessment System by providing advice and input on programme logic and monitoring needs. The OIE plans to conduct 2-4 high-level studies per year from 2016. The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation pyramid151 are brought out in the remaining sections of this Review, not least given the limited resources available.
To conclude: The OIE has made a first step in proposing a strategy for establishing itself as an independent evaluation resource. But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities.
The Value / Usefulness of OIE’s Independent EvaluationsEvaluation is a powerful tool that can provide useful, evidence-based information to help inform and influence policy and practice. But useful evaluations depend not only on the evaluators’ skills, but on several other important factors as well; 1) on planning evaluations to be relevant to the priorities of the organisation’s work and for their results to be delivered in time to be useful; on the degree of 2) consultation and ultimately ownership by those who seek evaluative information; on the 3) tools used to support the evaluation process per se; and on the 4) credibility and quality of the evaluation products152.
151 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).152 These aspects reflect the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
265
1. Planning relevant and timely evaluationsThe OIE is now working on a 3 year rolling work plan that sets out the broad areas for enquiry. So far, there are no agreed criteria for making the selection of the specific topics for independent evaluation, although the priorities tend to reflect those of the CDB’s strategic plan. Nevertheless decision-making is rather arbitrary based on a process of dialogue between the OIE and the CDB and the OIE and the Board.
One of the OIE’s two objectives for 2015 therefore, was to define a work plan and agree priorities based on an approach that is “utilisation-focused”. This means that the studies are selected and planned to be relevant and useful to the organisation’s needs.
The OIE has achieved this objective with respect to its latest studies, which concerns the Social Development Fund (SDF) Multicycle 6&7 Evaluation, the Haiti Country Strategy evaluation and the evaluation of the CDB’s Policy Based Operations. Each of these three have been planned to deliver their results in time to provide the CDB Board of Directors with relevant information for negotiating the next round of funding. In spite of some delays due to a myriad of reasons, not least to the extra effort needed to secure essential data, the studies are expected to deliver on time.
The processes for agreeing OIE’s work plan and specific evaluations on the one hand, and, in securing alternative funding on the other, are shown in Figure 1 below. The Panel was surprised at learning how bureaucratic (the internal approval process), and inefficient (in view of the time it takes) the process seems to be. The concern here is that such a process could possibly pose a threat to assuring the Board of “timely studies.”
Figure 1: Selection of Evaluation Topics and Funding Source
Consultation with CDB Operations and OAC/Board for selection of
evaluation topic
Internal review of Approach Paper
Specific Evaluation Study Design and Budgeting
OIE Draft Terms of Reference / Approach
Paper
Finalise Approach Paper and submit to OAC/Board
Final Approach Paper
OAC ApprovalOAC minutes
Paper
3-year Work Programme and Budget (approved by Board)
Board approval necessary If above USD
150,000
Board notification only if USD 150,000 or
below
Board Paper
Annual OIE report and work plan
submission to OAC
266
2. Consultation and ownership“The credibility of evaluations depends to some degree on whether and how the organization’s approach
to evaluation fosters partnership and helps build ownership and capacity in developing countries.”
(ECG good practices)
The OIE engages with the OAC, CDB senior management and operations for agreeing its 3-year work plan and then for selecting the specific topics and themes. It also discusses the evaluation approach paper (design and implementation plan) with the CDB and OAC before completing the final version. However, preliminary and final drafts of the report are only submitted to the CDB line and senior managers for comment and factual errors. Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the results and their implications. Discussions with the OAC are more limited due to the overburdened agenda of OAC and Board meetings, as previously discussed.
In short, the OIE is to be commended for following the recommendations of professional good practices and standards on participative approaches; it has succeeded in having introduced a modus operandi that involves the key players in the selection of evaluation topics, the evaluation designs and their results. Figure 2 below provides an overview of the evaluation implementation and stakeholder engagement processes.
Figure 2: Evaluation Study Implementation and Feedback Loops
Detailed ToR or Final Approach Paper if sufficiently detailed.
Funding Track
Final Approach Paper/ToR Board
Approval
OIE – Selection of consultants (if any) contracting
OIE Admin Budget or …
… SDF
Prepare TA Paper (content similar to Approach Paper but different
format.
TA Paper
Approval – Internal Loans Committee
Arrangement AFully outsourced / external
consultants; oversight by OIE
Arrangement BConducted by OIE
staff
Arrangement CJointly: external
consultants and OIE
Terms of Reference
OIE – Selection of consultants (if any)
contracting
267
13.14.
Preparations:Detailed evaluation plan (incl tools,
timeline, etc.) and logistics
Production of Inception Report / Approach Paper
Prepares Inception Report /
Approach Paper
Prepare for disclosure and dissemination
Presentation/workshop:Interim findings and conclusions for immediate feedback and validation
Data Collection and Analysis
OIE
Summary and ppt for workshop presentation
and discussion with CDBSubmission of Draft Final
Report to OIE
Final OIE approved report to CDB Senior Management for Management Response
Board notification only if USD 150,000 or
below
Draft Final Report
Review loops – OIE and CDB (potentially also BMC)
Feedback to evaluation lead
Submission of Final Report to
OIE
Final Report
Final Report and Management Response submitted to
OAC/BoardFinal Report and
Mgt. Resp.
Management Response
OIE ApprovalFinal Report and Management Response considered by CDB
AMT
OAC/Board endorsed
268
Notes to Figure 2
13. The OIE informed the Panel that this is an abbreviated version as there are e.g. additional steps (secondary processes) when evaluations are procured (tendering or single source), when there are additional review loops and updates to OAC etc.
14. OAC may also decide to return the report to OIE, the Panel were informed, or demand from Management specific actions based on the report.
This process is engaging and appears to have secured senior management and OAC interest and buy-in as witnessed in the latest studies. But there is the downside too! The process takes much time and, in our view, is partly unnecessary. The Panel appreciates that staff from operations as well as the AMT may both want to confer on an appropriate management response, but this should not be the case for reviewing an independent report for factual errors. The two-phase approach seems somewhat inefficient and unnecessary in our opinion.
Contact between the OIE, the CDB and/or the OAC during the actual study implementation is most often restricted to the occasional progress report, particularly when studies run behind time. There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. The OIE does, however, arrange discussions for reflecting on emerging findings, but we are not sure of how systematic this feedback loop is.
More generally speaking, outside of an evaluation study, the OIE has limited dealings with operations. The OIE has an advisory role in providing them with help, particularly with providing training, guidelines and tools to support self-evaluations. We are nevertheless concerned about the seeming distance between these two and how this has affected the perceived value of evaluation. (For further on this point, please see the section below on “Self- and Independent Evaluations”)
But the Panel also wishes to stress that this is not the case for newly appointed senior managers. A much more open attitude to evaluation and appreciation of its potential value was evident; they expressed interest in drawing out important lessons on what works, how, for whom, and under what conditions. In one case, interest was followed up in practice; the OIE was recently invited by a senior manager to share evaluative knowledge and experience with his staff regarding policy based operations.
Certainly, we can say that overall, the key stakeholders within the CDB are adequately integrated into the evaluation process as to foster their buy-in and ownership. But more generally, we feel that the utility of independent evaluations can be improved by fostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that evaluation can offer can add value to understanding the strengths and weaknesses of such strategies. This however cannot be done overnight and takes a long time.
3. Tools to support the evaluation processSo far, during this transitional phase, the OIE has mainly focussed on improving the tools to support the operations areas’ self-evaluations. This has left the OIE with little time to produce the checklists or tools to support its own studies. There are plans to develop an OIE Manual to guide and support the independent evaluation process. Such plans should be encouraged, as these documents will form a very important part of training, particularly for newcomers to the OIE team.
In the meantime, the OIE and operations staff refers to the Performance Assessment System (PAS) Manuals for evaluation activities. The manuals are based on DAC criteria and ECG principles. Much emphasis is given to the rating system and how and what should be rated.
269
However we find them lengthy, unwieldy and overcomplicated. Moreover, such manuals should be used for reference, but cannot and should not replace first-hand training in how to plan, conduct and manage the evaluation process.
Quality Assessment (QA) and Quality at Entry (QaE)
There was a transition period between 2012 and 2014 to establish the OIE. Work on the PAS, QaE, PCRs, ARPP, which had started earlier, was therefore completed after OIE came into existence, but it effectively had no formal ‘home’ in operations. The Panel was told that there had been some discussions about creating a Quality Assurance unit within CDB (OPS) but the current status is unclear.
The QaE Guidance Questionnaire was developed before and completed by the OIE. It was used to assess the documents that came across to the OIE for comments at the Review Stage. The results were then sent to the Portfolio Manager/Project Coordinator indicating any gaps/issues that needed to be addressed or clarified. QaE Guidance Questionnaires were developed for all the Bank’s lending products, CSP and to assess the quality of supervision.
After the QaE was launched bank wide, several operations officers saw the merit in using the QaE Guidance Questionnaire in the field and adopted it as a tool for their use during the appraisal mission in order to cross check and test their data collection and analysis.
OIE’s use of the QaE was discontinued in 2014 due to limited resources and a stronger focus on evaluations. It still sometimes comments on specific appraisals, but very selectively.
Both QaE and QaS (quality at supervision) are also addressed in the PAS Manuals. In addition the QaE and PAS have been incorporated in Volume 2 of the Operations Manual OPPM.
The Review Panel assessed the QaE forms. They are relatively standard, adapted to the specificities of the CDB. They contribute to judging a project’s expected quality in a relatively objective way. As such, they are are helpful, as a benchmark, in the ex-post assessment of projects.
The Panel considers that the lack of an established Quality Unit in the CDB (and independent from OIE) is a weakness that should be addressed in the near future.
4. Credibility and Quality of Evaluation ProductsAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An independent evaluation is processed as follows; the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
Since 2012, the OIE has produced a range of studies and approach papers. This review is based on those listed below as provided by the OIE, and cover the period from May 2012 to December 2015. It includes 3 evaluations (in blue), 4 Assessment studies (in brown) 14 validations of self-evaluations (in green) and 3 Approach Papers (in purple) for upcoming evaluations. These are listed below in Table 4.
270
Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
Board Meeting
Date Type / Topic
251 May 2012 Ex-Post Evaluation Report on Road Improvement and Maintenance Project, Nevis -St. Kitts and Nevis.
Validation of Project Completion Report on Sites and Services – Grenada. Assessment of Effectiveness of Implementation of Poverty Reduction
Strategy 2004-09.253 Oct. 2012 Assessment of Extent and Effectiveness of Mainstreaming Environment,
Climate Change, Disaster Management at CDB.254 Dec. 2012 Assessment of the Implementation Effectiveness of the Gender Equality
Policy and Operational Strategy of the Caribbean Development Bank. Validation of Project Completion Report on Enhancement of Technical and
Vocational Education and Training – Belize. Validation of Project Completion Report on Fourth Road (Northern Coastal
Highway Improvement Section 1 of Segment II) Project – Jamaica. Assessment of the Effectiveness of the Policy-based Lending Instrument.
256 May 2013 Validation of Project Completion Report on Expansion of Grantley Adams International Airport – Barbados.
Validation of Project Completion Report on Fifth Water Supply Project – Saint Lucia.
261 May 2014 Validation of Project Completion Report on Immediate Response Loan, Tropical Storm Gustav, Jamaica.
Validation of Project Completion Report on Social Investment Fund, Jamaica.
Validation of Project Completion Report on Disaster Mitigation and Restoration – Rockfall and Landslip, Grenada.
263 Oct. 2014 Validation of Project Completion Report on Basic Education Project – Antigua and Barbuda
263 Oct. 2014 Approach Paper for SDF 6 & 7 Multicycle Evaluation
264 Dec. 2014 Validation of Project Completion Report on Policy-Based Loan – Anguilla Validation of Project Completion Report on Immediate Response Loan -
Tropical Storm Arthur – Belize. Evaluation of Technical Assistance Interventions of the Caribbean
Development Bank Related To Tax Administration and Tax Reform in The Borrowing Member Countries 2005-2012.
265 March
2015
Approach Paper for the Evaluation of Policy Based Operations
266 May 2015 Validation of Project Completion Report on Upgrading of Ecotourism Sites – Dominica
The Evaluation of the Caribbean Development Bank’s Intervention in Technical and Vocational Education and Training (1990-2012)
267 July 2015 Validation of Project Completion Report on The Belize Social Investment Fund I Project − Belize
268 Oct.2015 Approach Paper Country Strategy and Programme Evaluation, Haiti
The review and analysis of these documents is based on the UNEG Quality Checklist for Evaluation Reports (http://www.uneval.org/document/detail/607) as well as on ECG guidance (Big Book on Good Practice Standards).
271
Approach Papers
Three Approach Papers (APs) were made available to the panel (see Table [ref] above). An AP describes the rationale for the evaluation, the background to the topic evaluated, the evaluation framework (criteria and questions) and approach. It also describes the team and provides an initial planning. Being the first main deliverable of OIE’s evaluation process, APs are the starting point and therefore a major determining element in the roll-out of each evaluation. Therefore APs “have to get it right”.
The APs examined are clearly written, well-structured and of reasonable length.153 We were surprised to find, however, that they do not make explicit the objectives of the evaluated intervention(s), e.g., through a clear objective tree, or through an explicit theory of change, intervention logic or logframe. Whilst one of the APs contains, in an appendix, a results framework for the evaluation, the results framework for the intervention (PBO) itself is lacking.
Inception reports
Only one Inception Report was given to the Panel for review (SDF 6&7). This gives an in-depth description of the evaluated programme and provides a clear Theory of Change. It is good practice that this is established after a pilot field mission, which helps to amend the initial AP on the basis of field observations and sharpen the evaluation questions if needed.
However, it is still considered to be good practice to have the Theory of Change elaborated in the initial design documents . This would facilitate OIE evaluations after project completion. Establishing the Theory of Change of any intervention would be included in the QaE form more explicitly, to be developed between the Quality unit referred to above, and OIE.
Evaluations and Assessments
Three evaluations and four assessment reports completed during the review period were considered. Assessments are similar to evaluations but have a narrower scope; they focus on a limited set of aspects or judgment criteria, mainly effectiveness, i.e. achievement of objectives. Evaluations generally base their judgment on the internationally recognised DAC criteria as well as aspects of the CDB and BMC’s management of the intervention.
In general, these reports are of reasonable quality. In the main, they explain the evaluated object154 and provide evaluation objectives. The findings are organised around the evaluation criteria or questions detailed in the scope and objectives section of the report. They are based on evidence derived from data collection and analysis methods as described in the methodology section. The reports tend to dwell on the limitations that the evaluation encountered, but without becoming defensive. In one case (PBL Assessment) the report starts with a summary of the reviews on the topic done by other MDBs. This was a pleasant surprise and indeed a good practice that could well be adopted in future evaluations too.
However, the reports also show several significant weaknesses:
- Reports do not always provide clear (reconstructed) intervention logics or theories of change for the intervention(s) evaluated.155 Evaluation criteria and questions are defined at a fairly general level. They are translated into more precise “research questions” (in an “Evaluation Design Matrix”, for each project for each criterion). However, it is unclear how these questions relate to the intervention logic (as this is not made explicit). This may be
153 Opportunities remain of course to be more concise and to move parts to appendices, e.g., detailed descriptions of the evaluation team or part of the description of the evaluated intervention.154 Sometimes in great length: for instance with the SDF 6&7 multicycle evaluation report it is only at page 30 that we find the beginning of the report on findings…155 Again with the SDF 6&7 evaluation, it is said to be guided by a “Logic Model” which is not explained.
272
done in inception reports (of which, as noted above, only one was available for review), but should be done also in the final reports.
- The reports do not describe the link from the evaluation questions to the answers, how the evaluation judgments are made and how these ultimately transform into ratings for each criterion and each project. In other words, the explanation provided in the evaluation frameworks is inadequate. The “evaluation design matrix” currently used does not provide sufficient insight into how ultimately an intervention’s performance is judged.156 Links between findings, conclusions and recommendations could be improved by making this more explicit. In other words, reports should include the story on how the evaluand is credibly linked to any observed outcomes and impacts, and should be clear on how causal claims are made.
- With the exception of the PBL Assessment, reports are lengthy and detailed. One reason for this is an over-emphasis on ratings. Their detailed discussion, project by project, criterion by criterion, occupies a very prominent position in the evaluation reports’ main body of text. Although ratings are traditionally an important element in evaluations of MDBs, too strong an emphasis can be tedious and may distract the reader from the real lessons to be drawn. The detailed discussion of ratings, and their evidence base, would be better placed in an Appendix, with a brief summary in the main report. This would help give the lessons and recommendations a more prominent position than is now the case. This would also help make the evaluation reports not only shorter but also more interesting to read; this could help add value to evaluation’s image within the organisation.
- The reviewers feel that the OIE evaluations tend to over-emphasise objective-based evaluation157 and the DAC criteria to the exclusions of considering other evaluation approaches such as Developmental Evaluation (Patton, 2010158); evaluation should be case specific and answer the actual information needs of managers and other decisions makers rather than always concentrating on final performance.
- Related to the previous point (and again with the exception of the PBL Assessment) executive summaries (approximately 8 pages) are too long. For the evaluation report to increase potential impact, they would need to be reduced to 2 to 3 pages and be more focused; again this could be done by dwelling less on the individual ratings of projects and more on key findings, lessons and conclusions. More generally, reports could be better adapted to the needs of the different audiences. Although not strictly limited to evaluations, The Health Evidence Network Reports159 are a model that could be adapted for evaluation reporting purposes; they are specifically geared towards addressing policy and decision-making.
- The “Recommendations to BMCs” are an interesting feature of the reports, (although we are unsure to what degree such recommendations could be effectively followed up by OIE or the Bank, but certainly could taken up with BMC Board members.
156 Marlène: I moreover have the idea that the methodology (often described as “visits”) is based on interviews and little hard evidence. Any view on this?.JM: My “interview-based evaluations”!!157 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.158 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press159 See the reports available at the WHO’s Health Evidence Netowkr at http://www.euro.who.int/en/data-and-evidence/evidence-informed-policy-making/health-evidence-network-hen
273
- Reports (e.g. the evaluation report on Technical Assistance) focus much on technical problems that were encountered during the evaluation. Although these are important issues, again to improve the report’s flow and “readability” this section would be better placed in the Appendix. What counts is the story of the intervention, not the story of the evaluation (see “Limitations” section in the TA report for instance)
OIE Validations of Project and Country Strategy Programme Completion Reports (referred to globally as PCRs hereafter)
As said above, the OIE has the mandate to validate the Project and Economic departments PCRs and CSPCRs. However, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
The validations tend to repeat the different items reported in the PCRs and then provide extensive comment on each. The PCVRs go into great depth and detail, which makes the documents rich and complete. This is their strength – but also their weakness. The depth and level of detail, as well as the repetitions from the original PCRs, makes PCVRs (overly) lengthy (20-40 pages) and difficult to read. The OIE reported spending approximately 27.2% of its time on validating PCRs in 2015 compared with 44.4% on its core work, i.e. doing or managing the higher level evaluations. That is more than half of its evaluation work is being spent on the validation process. Finally, the PCVRs now seem to be, to a great extent, a standalone output of OIE. It is not always clear to us how they are being used as the “building blocks” for the OIE’s independent evaluations. Making this clearer in the independent evaluations would help show the link and therefore the value of the time being spent on the self-evaluation validations.
To conclude, the review finds that the OIE has taken steps to improve the perceived utility of evaluation in several ways. In the first instance, by planning its work to provide relevant and timely evidence geared towards helping the Board with its oversight and decision making tasks. The topics are selected through dialogue between the OIE and key CDB stakeholders and reflect priorities of the CDBs strategic plan. Secondly, by securing the interest and consequently the buy-in of the OAC and CDB senior management through engaging their input throughout the evaluation process. This is evidenced by the reported interest in the latest three studies, the Country strategy programme in Haiti, the evaluation of policy-based operations and the SDF 6& 7 multicycle assessment.
The OIE products are of an acceptable quality and could be even better if some of the shortcomings were addressed. However, the products themselves do not impair the utility of OIE’s work; this is undermined in several ways: (1) by the time delays in commenting on PCRs (OIE) and providing feedback to the independent evaluations (operations and management) (2) by the inefficient processes for agreeing topics and funding sources as well as providing OIE with management responses to its reports.
Putting Evaluation to Use: transparency, feedback and follow-upThere are several ways that evaluation can be, and is being used. As John Mayne has pointed out in his many publications on the issue,160 when we talk of evaluation use, we are mainly thinking about its Instrumental use—use made to directly improve programming and performance. But there is also conceptual use - use which often goes unnoticed or more precisely, unmeasured. This refers to the kind of use made to enhance knowledge about the type of intervention under study in a more general way. Or even Reflective use— this refers to using discussions or
160 See for example, his opening chapter to Enhancing Evaluation use: Insights from internal Evaluation Units, Läubli Loud, M. and Mayne, J. 2014, Sage Publications
274
workshops to encourage and support reflection on the evaluation findings to see how they might contribute to future strategies.
In the case of the CDB there is some evidence to suggest that “use” is not only instrumental, but other types are also developing. For example, in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessons. (Reflective use)
Another important use, as recommended by the ECG, is that from time to time a synthesis of lessons is drawn from a number of evaluations and made available publically. In fact the Panel was impressed to hear that in the past, the evaluation unit had done this drawing on lessons from evaluations of the power sector. (Conceptual use) Although nothing has happened since, it is now on the “to do list” for 2016 (OIE’s 2016 Work Plan).
As for instrumental use, responsibility for using the knowledge generated through evaluation and for possibly drawing up an action plan of what should be done is up to CDB senior management and the relevant CDB department and division. Oversight on applying recommendations and picking up on the lessons drawn is the responsibility of the OAC.
Evidence on how evaluations have actually contributed to decisions or negotiations is lacking or confusing, Certainly the OIE is unaware of the extent to which its evaluations are put to use. On the one hand, the OAC minutes sometimes indicate that lessons learned are integrated into the next phase. On the other hand, the reviewers were told that often in the past, the evaluation results were “too old” to be of use as the lessons had already been drawn and used way before the report was completed. Similarly, people’s gaps in memory on how well the evaluative information from previous studies may have been used may also account for the scarcity of evidence.
In response, the Panel questioned CDB staff and the OIE about a particular study, the Technical and Vocational Education and Training Assessment. The feedback was somewhat contradictory. On the one hand, the study was criticised as “confirming” news rather than bringing “new news”. However, on the other, we learned that In October 2015, the Board of Directors approved a proposal for the revision of CDB’s Education and Training Policy and Strategy. Work on this has already begun and an external consultant has been engaged to lead the process.
Although it is one of the OIE’s tasks to set up a database on results and lessons learned from evaluations, so far this has not been a priority. There is also currently no systematic tracking of lessons or recommendations arising from the evaluations, or on any progress in their uptake. (The Panel has already referred above to OAC’s lack of oversight in the use of evaluation.)
The OIE’s role in supporting CDB’s organisational learning is clearly specified in the Evaluation Policy, with many good suggestions for knowledge sharing activities such as “brown-bag lunches, workshops, pamphlets and short issues papers” (p. 19). So far, however, the OIE’s lead role on the knowledge sharing side appears to be quite limited. It has provided advisory input in Loan Committee discussions, and organises workshops together with the relative operations department for discussing the implications of evaluation studies. Ultimately, of course, the uptake of evaluation results and knowledge is in the hands of management. But the evaluation unit has an important role to play in terms of knowledge broker and knowledge manager. Both have tended to be underplayed in OIE’s work plan so far.
Transparency: The Communication Strategy
In recent times and with the approval of its new Disclosure Policy, the CDB has started to post its independent evaluation reports on its website. (There is nothing on the self-evaluations). The website also presents a good overview of the role and function of the OIE and evaluation within the CDB. This is a step in the right direction for sharing information. However, in our view, the CDB’s communication strategy is the weakest part of the evaluation system to date.
275
The Panel has already commended the OIE in its efforts to engage the CDB and the OAC in evaluation work. But reporting and communicating the lessons seem to be entirely targeted at the Board and the CDB. Moreover, the 2015 budget provides only US$2’000 for communication – nothing of which is intended for outreach.
Reviewers feel that actively engaging with the more indirect stakeholders, for example project implementers in the BMCs, NGOs or project beneficiaries is relatively weak161. There appears to be little reflection on drawing out significant messages for the broader group of stakeholders, or on how then to transmit them to the “right” people in the “right” way (knowledge brokerage).
To conclude, evidence on the uptake of evaluation is either confusing or sparse. It is unfortunate that so far no systematic record keeping system has been put into place to track lessons learned or the uptake of recommendations (or actions agreed from management responses). The OIE plays a weak role in brokering the knowledge generated through evaluations to the benefit of external partners and in managing such knowledge. Although the Evaluation Policy specifies the need for “distilling evaluation findings and lessons learned in appropriate formats for targeted audiences both within and outside the CDB” (p.19) such a targeted communication strategy has yet to be developed and budgeted.
Strengthening Evaluation Capacities and Networking From the onset in 2012, the OIE has stressed the importance of developing and strengthening evaluation capacities within the OIE, the CDB and, subject to available resources, in borrowing member countries. Building evaluation capacity in BMCs and the CDB is one of the OIE’s mandated tasks. It has been a priority that figures on the work plan from the beginning (Work Programme and Budget 2012-2104) The idea of developing an internship programme for graduates from the Caribbean region was one idea that was advanced to help build local evaluation resources. However, the capacity-building has primarily been focused on OIE and CDB staff to date. One of the OIE’s two objectives for 2015 therefore was to take up the challenge and “strengthen evaluation capacities and networking” to include reaching out to the BMCs.
Developing OIE staff capacities
The change from project level to strategic and thematic evaluations does require different evaluative skills and competencies. The MDB Evaluation Pyramid presented below in Figure 3 shows the different types of evaluation and changing resource needs as one ascends the pyramid. Implicit here also is the change in the type of expertise and competencies needed as evaluation aspires to the higher levels.
Consequently for 2015, the OIE set itself the objective of networking and developing working partnerships with regional and international evaluation entities and academic institutions. The rationale was twofold: (1) secure further support and guidance as well as (2) increase its outreach and coverage through joint work and international exposure. Another implicit aim was to benefit from partners’ contacts in the BMCs wherever possible so as to improve data collection and quality.
Figure 3: The MDB Evaluation Pyramid162
161 A broader communication strategy is one of the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.162 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
276
The OIE has therefore linked up with Carleton University in Canada and the University of the West Indies, Barbados campus. The OIE was also approached by the Development Bank of South Africa to exchange experiences about setting up an evaluation entity in a “small” development bank. However, its attempt to become a member of the Evaluation Cooperation Group was not successful for reasons beyond its control.
The OIE is to be commended in addressing the issue of staff competencies and professional development more generally. New developments in evaluation as well as new developments in the scope of OIE’s work may necessitate new competencies. For this reason, organisations such as the International Developmental Evaluation Association have recommended that the competencies of evaluators and evaluation managers should be periodically reviewed. Several publications now exist on competency requirements and suggestions for the periodic review of staff competencies.163
It is not within this remit to compare and contrast OIE’s competencies with those recommended by international and national agencies. However, what we can say is that the OIE demonstrates great forethought in taking this on board.
Capacity building within CDB
The OIE’s objective also consists of continuing to develop measures for improving the monitoring and self-evaluation side of CDB’s work. OIE’s strategy here is to use the windows of opportunity on offer through some of the training sessions that are being organised by CDB as part of its shift towards MfDR e.g. by Corporate Planning Services and Technical Assistance. For 2016 it is also planned to have the OIE present at the annual staff meeting and Learning Forum.
The OIE also organises some ad hoc training with operations, for example to help understand new tools e.g. for drawing out lessons from self-evaluation reports and, more generally, in
163 E.g. IDEAS, (2012) Competencies for Development Evaluation Evaluators, Managers and Commissioners, the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice (2010) and the Swiss Evaluation Society’s Evaluation Managers Competencies Framework (2014)
277
helping staff appreciate how evaluation can add value to the organisation’s work. Measures include providing advisory services on demand, and providing training alongside the introduction of new or revised tools.
Capacity building in the BMCs
This is an ambitious task and would require additional investment; from the bi-annual work plans to be effective. A modest attempt has been made in 2015; from what we understand, the OIE has joined together with the Carleton University and the University of the West Indies, using their networks in some of the BMCs, to try to develop this aspect.
To conclude, we cannot comment on the quality or reaction to such training, but can commend the OIE for making capacity building one of its priority objectives. From both the Policy and the documents we reviewed, we note that capacity building was always seen to be an important aspect of OIE’s work, but hitherto has received little strategic focus. But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.
Adequacy of the OIE’s human and financial resources to support its work
OIE’s Human Resources;
The OIE is has a staff of 5; the head, 1 senior evaluation officer and two evaluation managers, plus one administrative assistant. Three of the five were recruited from within the CDB. The limited capacity means that it is not feasible to cover all the types of evaluation activities outlined in the Evaluation Policy. Yet there is some indication from the Board that OIE should embark on impact evaluations at some future stage. An increasing demand for evaluation and for impact evaluations in particular, would run the risk of overstretching the OIE’s capacity to deliver credible and useful evaluations. Moreover, there are many other designated OIE activities that should be recognised as valuable work; the validations, building CDB and BMC evaluation capacity, providing supervision, advice, knowledge management and brokerage as well as managing evaluation contracts, The time needs of dealing with all of these may be underestimated in OIE’s budgets; all are important for assuring best value from evaluation. The Panel is concerned that a demand for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level evaluations may undermine the importance and time needs of other essential tasks.
Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses.
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the
278
backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.
Self- and independent evaluationSelf-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes. Both types of evaluation are important as they are at the very heart of the evaluation function; they are said to be the building blocks for the more strategic evaluations that the OIE is now undertaking.
The Evaluation Coordination Group recommends that the self-evaluations be carried out by the relevant operations department and in turn, reviewed and validated by the organisation’s independent evaluation office. The CDB’s Evaluation Policy therefore talks of “validating all self-evaluations” as being one of OIE’s essential oversight tasks.
Within CDB, the self-evaluations should provide management with performance assessments and thereby serve an accountability function to the CDB and Board. To support the process, the OIE provides operations with manuals and checklists for guidance. Once a self-evaluation report is to hand, it is given over to the OIE for the validation of its technical quality and credibility.164
However, in the CDB case, there are well-documented issues that have affected the quality and timeliness of the self-evaluations on the one hand, and therefore the quality of the foundation on which to build the independent evaluations. Paucity of documentation within CDB, paucity of data collected and available in the Borrowing Member Countries (BMCs), time delays in producing completion reports and in turn, having them validated by the OIE - all such issues were systematically raised during interviews and in some of the independent evaluation reports. There appears to be little incentive to complete self-evaluations in a timelier manner.
Generally speaking, many of the monitoring data problems appear to be due to a lack of management oversight. For example, with the introduction of results-based management, the logic frame and monitoring and data needs are systematically being built into intervention design. However, the BMCs are not delivering the data as contractually agreed at the outset.
164 According to the Evaluation Policy, OIE should validate all PCRs and CCRs but due to the backlog of reports and the delay in completing them (sometimes years later) since October 2015, the OIE has secured OAC agreement to validate a maximum of 6 per year, which are selected in consultation with the OAC.
279
Incentives to support any significant change towards building a results-based culture seem to be weak and sanctions seem to be rarely enforced when the supply of data is lacking or lengthy delays to the projects occur. Although we can appreciate the complexities of trying to enforce monitoring compliance, this means that often, project deadlines have had to be extended, data gaps are not being satisfactorily dealt with and in turn, there has been a void in the quality and quantity of available evidence for the CDB’s self-assessment of project performance. For some time, this lack of oversight has been tolerated. Part of the problem is the low priority accorded to completing the self-evaluation reports by operations, coupled with the absence of any focal point within senior management to drive the process and deal with the problems.
No record is kept of how the self-evaluation results are actually used. They do not appear on the CDB website, but we were told that the findings are integrated into the following project designs. Hence we are somewhat unclear as to the utility of these reports at present. The situation is exacerbated by a rather confused image of evaluation: some operations staff consider OIE’s input (through validations or independent evaluations) to be sometimes over-critical, regulatory and adding little value; it is a threat rather than an opportunity for learning. Yet at the same time, evaluation is recognized as an integral part of result-based management.
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.
Some managers however seem to start changing the status quo. For example a revised and simplified template for producing project completion reports is being considered, and mid-term project reviews are expected to be more stringent in looking at monitoring plans and practices and tying disbursements to performance. In some cases we also learned of incentives being introduced to encourage project managers to complete their reports in a timelier manner. But much remains to be done and, since the OIE is no longer responsible for monitoring and project evaluations, there is a void that needs to be filled. It is up to line managers to drive this work forward.
To conclude, it is fair to say that in view of a number of “frustrations” between the OIE and operations, which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring data, the added value that evaluation might offer to the operations area is ill recognized. Moreover, the link between self-evaluation as the building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
280
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”165
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
165 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
281
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”166 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 167 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
166 CDB (2011) Evaluation Policy (p.2)167
282
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
283
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
284
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
285
mendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
286
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
287
APPENDICES
Appendix I - The External Review Mandate – Terms of Reference and Approach Paper
Appendix II -Review Approach, Data collection and Analysis, and Limitations
Appendix III – Overview of OIE Evaluation Practice
Appendix IV - List of Persons Interviewed
Appendix V - List of Documents Reviewed
Appendix VI- List of Topics used to guide interviews with members of CDB Board of Directors
Appendix VII - List of Topics used to guide interviews with CDB staff
288
Appendix III – Overview of OIE Evaluation Practice (prepared by the OIE in response to Reviewer’s request)
Caribbean Development Bank, Office of Independent Evaluation - OIE
Category Response
Percentage of projects subject to project (self-) evaluation
100% - Project Completion Reports (PCR)
Percentage of projects subject to validation by OIE
Approximately 40-50%
About 15 projects exit portfolio annually. Evaluation Policy calls for all PCR to be validated. However, OIE resources insufficient. Validation process reviewed in 2014. Now OAC (Board committee) selects a sample of 6-8 PCR for validation each year.
Percentage/number of projects subject to in-depth review by OIE
None – unless specifically requested by OAC
Due to limited resources, focus of OIE evaluation work programme is on PCR validations and high-level evaluations – including country strategy and programme evaluations (CSPE).
Number of high-level evaluations conducted by OIE (e.g. sector, thematic, geographic)
1-2 per year since 2011
Plan is 2-4 per year from 2016. This would include CSPE (1st planned for Q1 2016: Haiti)
Number of project impact evaluations conducted by OIE
None
OIE includes “impact questions” in high-level evaluations.
Number of project impact evaluations conducted by Bank staff or other non-OIE staff
OIE is not aware of any impact evaluation conducted by the Bank.
However, OIE provides technical support to the Basic Needs Trust Fund (BNTF) in its design of an M&E framework that entails impact evaluations.
Budget In USD mn: 0.78 in 2015; 0.82 in 2016. This is equivalent to about 2.5% of total CDB Administrative Budget.
75% of the budget is for Staff salaries (4 Professionals, 1 Support staff), leaving around USD 190,000 (in 2015) for other expenses, including consultants e.g. for external evaluations. Additional funding is accessed via the Special Development Fund (SDF). This varies according to type and scope of the evaluation, e.g. the ongoing SDF 6/7 Evaluation is SDF funded at USD 255,000.
Budget determined by Board, not separate from administrative budget.
SDF funding for evaluations is considered separately and subject to Bank internal approval process. SDF funding cannot be used to cover OIE expenses such as staff time or travel. Country eligibility for SDF funding is also a consideration. OIE expressed concerns about this funding track in respect to predictability, independence and eligibility limitations.
Head of OIE reports to Board, with administrative link to the President
Terms of appointment for Head
289
5 year term, renewable once. Appointed by the President with the agreement of the Board.
Right of Return for Head Not eligible for other staff positions.
Consultants as proportions of OIE budget
2015: 19% (USD 145,000)
Plus SDF funding. SDF funded evaluations are outsourced.
Last external evaluation (or peer review) of OIE
No external evaluation, though a review of the function was done in 2011, leading to the Evaluation Policy.
OIE External Review completed in April, 2016
Departments or special programmes supporting impact evaluation
None
290
Appendix IV – List of Persons Interviewed
Name Function relative to OIE Type interview
Mrs. Colleen Wainwright Member CDB Board of Directors (UK)
Face to face
Mrs. Cherianne Clarke Alternate MemberCDB Board of Directors (UK)
Face to face
Mrs. Jean McCardle Member CDB Board of Directors (Canada)
Face to face
Dr. Louis Woodroofe MemberCDB Board of Directors (Barbados)
Mr. A: de Brigard Former Member CDB Board of Directors
Skype interview
Mr. H. Illi Fromer Member CDB Board ofDirectors
Telephone interview
Mrs. Claudia Reyes Nieto Member CDB Board of Directors
Telephone interview
Mr. Bu Yu alternate DirectorCDB Board of Directors
Face to face
Mr. Michael Schroll(Barbados)
Head OIE
series of interviews viaSkype and face-to-face
Mr. Mark Clayton OIE Senior Evaluation Officer Focus GroupMrs. Egene Baccus Latchman OIE Evaluation OfficerMr. Everton Clinton OIE Evaluation OfficerMrs. Valerie Pilgrim OIE Evaluation Officer
Dr. Justin Ram CDB Director Economics Department
Face to face
Mr. Ian Durant CDB Deputy Director Economics Dept Face to faceDr. Wm Warren Smith CDB President
Joint interviewFace to face
Mrs. Yvette Lemonias-Seale CDB Vice President Corporate Services & Bank Secretariat
Mr. Denis Bergevin CDB Deputy DirectorInternal Audit
Face to face
Mr. Edward Greene CDB Division Chief, Technical Cooperation Division
Face to face
Mrs. Monica La Bennett CDB Deputy Director Corporate Planning Face to faceMrs. Patricia McKenzie CDB Vice President Operations Face to faceMs. Deidre Clarendon CDB Division Chief
Social Sector DivisionFace to face
Mrs. Cheryl Dixon CDB Co-ordinator, Environmental Sustainability Unit
Focus group
Mrs. Denise Noel- Debique CDB Gender Equality Advisor Mrs. Tessa Williams-Robertson CDB Head Renewable EnergyMrs. Klao Bell-Lewis CDB Head Corporate Communications Face to faceMr. Daniel Best CDB Director
Projects DepartmentFace to face
Mr. Carlyle Assue CDB Director Face to face
291
Finance Department
292
Appendix VI - Interview Guide: Members of CDB Board of Directors
Below are a list of themes that I should like to raise with you based on your experience and knowledge of the CDB’s independent evaluation function (Office of Independent
Evaluation).
In each case, I should be grateful if you could illustrate your responses with examples or help this Review by, wherever possible, sending me (or telling me where I can find) any
documents that could support your responses.
This guide is being sent to you in advance to help prepare our meeting. However, our interview will be conducted more in the style of a conversation. The following sub-questions will be used to GUIDE the interview. Please feel encouraged to raise any
additional issues that you feel we should take into account
On the governance and Independence of CDB’s evaluation functionWhat mechanisms are there in place to support its independence?
How satisfactory are the current arrangements in your opinion?
How is the balance between independence and the need for interaction with line management dealt with by the system? For example, what mechanisms exist to ensure that the OIE is kept up to date with decisions, policy / programme changes, other contextual changes etc that could have an affect on OIE evaluation studies / evaluation planning?
On the OIE’s Evaluation PolicyThe CDB’s Evaluation Policy was established in 2011. To what degree do you feel it is adequate? Still relevant?
What suggestions do you have for any improvements?
In your opinion, how adequate is the current quality assurance system for over viewing the evaluation function?
On the quality and credibility of evaluation studiesTo what degree do you believe the reports are fair and impartial?
Do you consider them to be of good quality? Are they credible?
Are you adequately consulted/involved on evaluations of interest to you?
On the relevance and usefulness of evaluations How well does the OIE engage with you / your committee during the preparation, implementation and reporting of an evaluation study to assure that it will be useful to the CDB?
How are the priorities set for the independent evaluations? What criteria are used? Are you satisfied with the current procedure?
293
When OIE evaluation studies are outsourced to external consultants, what criteria are used to make this decision?
How are the priorities for the OIE’s 3.year rolling work plan agreed? In your opinion, is the current plan adequate in terms of coverage and diversity?
In your opinion, do the evaluations address important and pressing programs and issues?
To what extent do you feel that the OIE’s evaluations integrate the cross-cutting theme such as gender, energy efficiency/renewable energy, climate change? What improvements might be made and how?
On the dissemination and uptake of evaluation findings and recommendationsTo what extent do you feel that evaluation findings are communicated to the CDB and its stakeholders in a
a) useful, b) constructive andc) timely manner?
Are evaluation recommendations useful? Realistic?
What mechanisms are in place to assure that evaluation results are taken into account in decision making and planning? What improvements do you feel could be made?
How have you used the findings from any evaluations? Examples?
To what degree do you feel that evaluation contributes to institutional learning? And what about to institutional accountability? Any examples?
What mechanisms are in place to ensure that knowledge from evaluation is accessible toCDB staff and other relevant stakeholders? Are the current arrangements satisfactory?
How satisfied are you with current arrangements? What expectations do you have for the future?
On resourcesHow is the OIE resourced financially and is this satisfactory?
What about the OIE staff, are all the important areas of expertise represented in the team?
On this Review of the Office of Independent EvaluationWhat are your expectations? What are you particularly hoping to learn from it?
Thank you very much for your cooperation and input
294
Appendix VII : Interview Pro-Forma – CDB Staff membersThis presents a list of the topics raised during interviews. It was used to guide the open-ended
discussion – this means that the sequence and exact wording of the questions may not necessarily have followed in this order or been asked in exactly this way.
Changeover to an Independent Evaluation Office? Expectations? Advantages and disadvantages??
Satisfaction with working relations between operations and the OIE from your perspective?
Process of dealing with the PCRs and CCRs? Advantages and limitations?
Quality and credibility of the validation process?
How are the self-evaluation reports used?
Credibility and Quality of OIE’s evaluation reports
Communication of self and OIE independent evaluations? To whom, in what way? Possible improvements?
External Review of the Office of Independent Evaluation
Caribbean Development Bank
April, 2016
295
Principal ReviewerMarlène Läubli Loud,
Review Panel MembersJohn Mayne
Bastiaan de Laat
296
Primary audiencesMembers of the Oversight Assurance Committee
Members of the Board of Directors
Independent Evaluation Office
Caribbean Development Bank Management and Staff
Review Panel Members
Marlène Läubli Loud (DPhil) is currently an independent consultant and trainer in public sector evaluation. She has worked with a range of organizations, small and big including the European Commission, the World Health Organisation, the United Nations Evaluation Group, the UK Employment Department, UK Health Promotion Agency (now merged and become NICE), and the English Nursing Board. She was head of the Research and Evaluation Unit at the Swiss Federal Office of Public Health for nearly twenty years where she gained much experience in evaluation management, and especially in the ways and means for improving the use and utility of evaluation in organisations. She continues to have a keen theoretical and practical interest in this area. Prior to this, she was an independent evaluator in the UK, specializing in the evaluation of developmental programmes in health and general education. She was also a research fellow at the Department of Education, University of Surrey and in the Social Science Faculty, University of Oxford, UK.
John Mayne (PhD) is an independent advisor on public sector performance. He has been working with a number of organizations and jurisdictions, including several agencies of the UN, the Challenge Program on Water and Food, the European Union, the Scottish Government, the United Nations Secretariat, the International Development Research Centre, the Asian Development Bank and several Canadian federal departments on results management, evaluation and accountability issues. Until 2004, he was at the Office of the Auditor General where he led efforts at developing practices for effective managing for results and performance reporting in the government of Canada, as well as leading the Office’s audit efforts in accountability and governance. Prior to 1995, John was with the Canadian Treasury Board Secretariat and Office of the Comptroller General. He has authored numerous articles and reports, and edited five books in the areas of program evaluation, public administration and performance monitoring. In 1989 and in 1995, he was awarded the Canadian Evaluation Society Award for Contribution to Evaluation in Canada. In 2006, he became a Canadian Evaluation Society Fellow.
Bastiaan de Laat (PhD) is Evaluation Expert and Team
297
Leader at the European Investment Bank (EIB) where over the past two of years he has been in charge of major evaluations in important areas such as Climate Action, SME support and Technical Assistance. He has a longstanding experience in evaluation as well as in foresight. Founder-director of the French subsidiary of the Technopolis Group (1998-2006) he led many evaluations for and provided policy advice to a great variety of local, national and international public bodies. He trained several hundreds of European Commission staff and national government officials in evaluation and designed monitoring and evaluation systems for various public organisations. Before joining the EIB he worked as Evaluator at the Council of Europe Development Bank. He has developed tools and performed programme, policy and regulatory evaluations, both ex ante and ex post, in a variety of fields. He has also made several academic contributions, most recently with articles on evaluation use and on the "Tricky Triangle", on the relationships between evaluator, evaluation commissioner and evaluand. In his private capacity, Bastiaan served as Secretary General of the European Evaluation Society and was recently elected Vice-President.
Acknowledgements The Review exercise could not have been possible without the support and commitment of the OAC, the OIE and the CDB. The exploratory discussions with the OIE staff, members of the Board of Directors as well as with the CDB management and staff provided great insight and were a valuable contribution to the Review.
We are indebted to the Head of OIE, Michael Schroll, and his team for their cooperation, insight, and readiness to provide us with any information requests. We are also grateful to them for their useful comments regarding the first draft of our report, and for their suggested improvements.
We are especially appreciative of OIE’s administrative assist, Denise xxxx, for her help in coordinating the interviews during our 10-day field study in Barbados and in providing us with all documents we requested.
298
AcronymsLIST OF ABBREVIATIONS
AMT Advisory Management TeamAPECAPAR
Audit and Post Evaluation CommitteeApproach PaperAppraisal Report
BOD Board of DirectorsBMCs Borrowing Member CountriesBNTF Basic Needs Trust FundCDB Caribbean Development BankCSPDAC
Country Strategy PaperDevelopment Assistance Committee
DFI Development Financial InstitutionED Economics DepartmentFI Financial InstitutionIRLMDB
Immediate Response LoanMultilateral Development Bank
mn millionM&E Monitoring and EvaluationMfDR Managing for Development ResultsOAC Oversight and Assurance CommitteeOIE Office of Independent EvaluationPAS Performance Assessment SystemPBG Policy-Based GrantPBL Policy-Based Loan PCR Project Completion Report PCVR Project Completion Validation ReportPPES Project Performance Evaluation SystemPPMS Portfolio Performance Management SystemSDF Special Development FundTA Technical Assistance WB World Bank
299
Preface Evaluation work at the Caribbean Development Bank (CDB) has been ongoing since the early 1990s, although initially it was mainly focused on the ex-post evaluation of projects. However, in 2011, the CDB reviewed its evaluation system to bring it up to date with the good practices of international development organisations. In December that year, it produced its comprehensive Evaluation Policy (December 2011) setting out the aim and objectives and guiding principles for CDB’s evaluation system.
The Policy provides for the establishment of the Office of Independent Evaluation (OIE). Its main objective is to provide “CDB’s Board of Directors, President, Advisory Management Team, CDB staff and other stakeholders and partners with timely, credible and evidence-based information on the relevance and performance of CDB’s projects, programs, policies and other development activities.” (Evaluation Policy, 2011, p. 1).
To oversee and assess good practice, the Evaluation Cooperation Group (ECG) for Multilateral Development Banks (MDBs) recommends that the MDBs’ evaluation system and independent evaluation units be the subject of a review on a regular basis. The aim here is to help the institutions adopt recognised evaluation standards and practices so that its policies may benefit from evidence-based assessments.
In mid-2014, a new Head of the OIE was appointed and, following an initial learning period, he called for a peer review of the evaluation system. Even though the OIE had only been in existence since 2012, it was considered timely to take stock of what had been done so far in order to tease out the priorities for the next 3-4 years.
It was originally anticipated that such an assessment could be done by the ECG as part of the OIE’s application for ECG membership. This did not prove possible, since the CDB’s operation is considered too small for such membership. A review was therefore commissioned to independent experts in evaluation who are knowledgeable and experienced in the management of internal evaluation units.
Main AimThe Review’s main aim is to provide the CDB’s Board of Directors with an independent assessment of the OIE and CDB’s evaluation system. The intention is to highlight the factors that help or hinder the OIE’s independence and performance in order to identify where improvements could be made. This report will be presented together with a Management Response to the CDB’s Oversight Assurance Committee and its Board of Directors at its meeting in May 2016. It is anticipated that an action plan will be drawn up on the basis of the Board’s decision on how to address the recommendations put forward.
Report StructureThe Review starts with some general background information about the CDB and the setting up of an independent evaluation function. It also sets out the reasons for an external review and why this was requested at this particular point in time. Part One also outlines the Review, which, is presented in more detail in Appendix II. Part Two provides the Review’s findings and conclusions according to each of the criteria presented. The Panels general conclusions and recommendations are the subject of Part Three.
The Panel is grateful for the complete freedom it was given to form its own opinions and to reach conclusions based on its analysis. The findings, conclusions and recommendations
presented in this paper are those of the Peer Review Panel members. The views of the CDB, are provided separately in the Management Response that accompanies this Report.
300
Executive Summary
301
Part One: Introduction and BackgroundIn order to understand the development of the Office of Independent Evaluation’s (OIE) work, a brief description of the CDB’s current reforms is needed. First, there has been a change over the last decade from the nature of the programmes the bank supports; for example it has become increasingly engaged in funding policy-based operations and social development issues. Similarly, there have been changes in the whole of the development field, which is grappling to deal with complex issues such as gender or climate change. To meet today’s challenges and ensure that its work practices reflect the international standards of Multilateral Development Banks (MDBs), the CDB has introduced a number of measures aimed at improving its effectiveness and efficiency. For example, in line with international standards for Management for Development Results (MfDR), it has introduced a Results Based Management Framework for organising and assessing its performance.
In 2011, the CDB commissioned an external consultancy to undertake an assessment of its evaluation function in order to develop a policy that took account of good practices within the international development community.168 The CDB’s Evaluation Policy (referred to hereafter as the Policy) is a direct response to that review; it reflects the standards and good practices of the Evaluation Cooperation Group (ECG) of the Multilateral Development Banks (MDB) as well as the evaluation principles and standards of many professional associations. Similarly, the Bank showed its commitment to having evaluation as a core function by establishing an independent evaluation unit that reports to the Board of Directors.It is responsible for assessing the Bank’s activities and interventions, but especially for drawing out the key lessons and recommendations for improving the Bank’s performance.
As such, the monitoring tasks formally under the responsibility of the Evaluation and Oversight Division (EOV) were handed over to the Bank’s Operations and Economic Divisions. The OIE, in its advisory capacity, is expected to provide the sareaoperations area with the necessary support manuals, tools and guidance; the OIE then validates the credibility and rigour of the self-evaluations.
In addition to the OIE, the Bank also set up other independent functions: internal audit, risk assessment and management, integrity, compliance and accountability. The mainstreaming of three cross-cutting themes (gender, energy and climate issues) into CDB’s work has also been initiated. At the same time, there are limited funds available as the CDB is working within a Board-sanctioned policy based on the principle of a zero real growth, which is in line with the budget policy of other MDBs.
In short, the bank has taken many important steps towards updating CDB’s management practices in line with other MDBs. However, the introduction of many innovations in parallel requires coordination and a shift in working practices and thinking. There is also the need to engage in different types of evaluation; evaluations that take into account cross-cutting themes and different levels of complexity. As such, whilst this review is particularly focused on the CDB’s Office of Independent Evaluation (OIE), its work and utility depend to a large degree on the development of other management practices and the degree to which evaluation is able to link to their work.
A full description of the Review’s mandate, approach, process and methods are provided in Appendices I and II. It was designed to address the following four key questions as set out in the appended Terms of Reference and Approach Paper:
168 Osvaldo Feinstein & Patrick G. Grasso, Consultants, May 2011 Consultancy to Review the Independence of the Evaluation and Oversight Division of the Caribbean Development Bank
302
To what degree is the Office of Independent Evaluation independent at the strategic, functional and operational levels? Which measures help or hinder such independence?
To what extent is the OIE achieving its 2 strategic objectives? (which are (1) the timely delivery of good quality evaluations and PCR Reviews and (2) strengthening capacity building, networking and communication) How useful are the OIE’s procedures and products towards this end?
How adequate are the financial and human resources of the OIE for carrying out its tasks and achieving its objectives?
How effective is OIE in relating with its internal partners to develop evaluation capacity?
This Review of the OIE is based on the recommended criteria of the Evaluation Cooperation Group for Multilateral Development Banks; governance and independence, credibility, use and transparency.
The data used for analysing and interpreting the findings relied on exploratory, semi structured interviews with OIE staff as well as with CDB senior and middle managers and members of its Board of Directors. Whilst much of the interview data was collected during a 10-day intensive, on-site visit to the Bank, the majority of the Board members were interviewed through Skype. The interview data was complemented by a review of a range of key documents including the Bank’s Evaluation Policy, various kinds of reports on, or about evaluation, the complete set of minutes of meetings between the OIE and the Oversight Assurance Commission169 and the subsequent chairman’s report to the Board for the study period 2012 to 2015, OIE staff biographies as well as a number of other organisations’ evaluation principles, good practices and standards. A full list can be found in the Appendices (Appendix V). Not least, the Reviewers have also drawn on their own knowledge and experience of evaluation management to complement data analysis and interpretation.
Scope and Limitations
The Panel was asked to concentrate on the 4-year period since the establishment of the OIE, January 2012 to December 2015, but more particularly on the changes introduced since the new Head of the OIE was appointed (June 2014 to December 2015).
It has mainly focussed on the strategic role of the OIE within the CDB as well as its functional and operational roles and responsibilities.
It was planned as a Review and not a fully-fledged evaluation; this was due to the limited time and resources available for the exercise as well as the fact that a ”light” review is in keeping with the spirit of the OIE’s Terms of Reference. The Review could not undertake any in-depth analysis of documents or consult with country level stakeholders or other external sources of expertise. Moreover, of the 29 people identified for interview, despite several reminders (by email or telephone) , the Panel were unable to either contact or secure the agreement of 5 of the 14 Board members, and 1 CDB senior manager. In light of this experience, as well as the time invested in securing the interviews “at a distance”, the planned on-line survey to follow-up on face-to-face interview data was abandoned.
We regret that in the time available, full justice could not be done to all the material provided to the Panel by the OIE. Nevertheless, the documentary review and interviews focussed on addressing the key questions, and we are therefore confident that the main issues raised in the Terms of Reference have been addressed in this report.169 The Audit and Post Evaluation Committee, now the Oversight Assurance Committee, is a Board Committee responsible for the oversight of evaluation and other key management functions.
303
304
Part Two: What the Review Found In the first place, the Panel should like to commend the CDB for its efforts to establish an independent evaluation function.. Similarly, in spite of some of the challenges raised in this Review, the current Head of OIE and his team are to be commended for their efforts in advancing evaluation in the right direction of the UNEG Norms and Standards and ECG guidelines on Good Practices. The Panel presents its findings and analysis in a spirit of constructive criticism, highlighting the strengths of the current situation as well as several challenges that need to be addressed in order to bring out the full value of evaluation to the CDB.
In this part of the report, we present our findings and conclusions relative to the following criteria used to assess and respond to the four TOR questions:
the evaluation policy governance independence the OIE strategy, practices and work programme usefulness of evaluation, evaluation use communicating evaluation results (transparency) adequacy of resources, and finally the working relationship between self and independent evaluation
The Evaluation PolicyThe CDB Board agreed an Evaluation Policy (the Policy) in December 2011. It sets out the guiding principles and provisions for the OIE. It also aims at guaranteeing the independent functioning of the Office of Independent Evaluation (OIE) by having it report to the Board of Directors through the Oversight Assurance Committee, OAC. However, the President retains oversight on administrative matters for management of day-to-day activities such as travel approval..
Generally speaking, the Policy reflects many of the ECG’s recommendations on evaluation independence and good practices. Similarly, the evaluation criteria for judging outcomes are the five developed by the DAC, that is relevance, effectiveness, efficiency, impact and sustainability. In general, the Policy is intended to maximize the strategic value, timeliness and the learning aspect of evaluation.
Yet in reality, the Policy provides a framework for what could be achieved under optimal conditions. It is overambitious in terms of what could be done with the current level of resources. For example undertaking the validation of all Project and Country Completion Reports as well as engaging in the full range of evaluation types undertaken within the MDBs have proven to be simply not feasible at this stage. (More on this later in the report.)
Many important tasks outlined in the evaluation policy have not been done so far by either the OAC or the OIE. For instance, the OAC has yet to produce an annual report on OIE’s performance and the OIE has yet to establish a database of evaluation lessons, recommendations, actions and management responses.
To conclude: The Evaluation Policy is a mission statement of what could be achieved in time with sufficient financial and human resourcing. It reflects the internationally recognized evaluation principles and standards, but is probably somewhat ambitious for the OIE to fully put into practice for a number of years
305
Governance IssuesOversight of the OIE is entrusted to a Committee of the Board of Directors (originally called the Audit and Post-Evaluation Committee, APEC, and now the Oversight Assurance Committee (OAC).The OIE reports to the Board through the OAC. There are 5 members, of which only 2 are located in Barbados.
The OAC meets 5 times per year, the day before Board meetings. It has oversight responsibility for external and internal audit, independent evaluation, risk management and integrity, compliance and accountability in relation to CDB’s work.
The OAC Chairperson prepares a very brief resume of the day before’s meeting to present to the Board for approval. The report generally covers progress, shortcomings and risks but is only a small part of the Board meeting so that generally there is little discussion; evaluation is only one of many items on the agenda. (We were told that the report to the Board averages approximately 10 minutes). Some of our interviewees could not recall any discussion about evaluation during Board meetings or remember reference being made to any evaluation report.
Given the breadth of its oversight responsibilities, there is now provision for the OAC to hire in consultants to provide it with technical expertise as needs be. Resources would have to be provided out of the CDB’s administrative budget. Another novelty is the provision to meet with the head of the OIE in an executive session at least once per year. So far, our understanding is that neither of these opportunities has yet been taken up by the OAC.
The major problem for the OAC is the volume of paperwork and length of individual documents received in parallel from the CDB and its independent offices, generally very shortly ahead of its meetings. Both Board and OAC members expressed their deep concern about the need for the more timely delivery of reports and background papers.The OAC members fear they are unable to do justice to their oversight responsibilities. Hence, based on the Panel’s review of the minutes and comments from the OIE, the meetings appear to be more formalisticbut with little in-depth discussion or systematic follow up on the recommendations, agreed actions or the lessons drawn, The “follow up on actions agreed” does not appear to be a systematic item on each OAC meeting’s agenda..170 Similarly any attempt to identify key messages for various stakeholders other than the CDB is not mentioned in the minutes or reports to the Board.
In response, the OIE has greatly improved the presentation of technical reports by summarising the main points in its “Brief Reports” (e.g. the Tax Administration and Tax Reform and Technical and Vocational Education and Training evaluation). This is commendable and certainly a step in the right direction although the Panel considers that they should have a sharper focus on the strategic issues (which are the end of the brief rather than the beginning), be condensed and be made more “reader friendly”.
The Panel was also surprised to find that despite expressions of support for rigorous evaluation and its importance to the CDB, the OAC do not appear to be taking any firm position with regard to the paucity of available data. OAC has been made aware of the data problems in the BMCs (e.g. lack of rigorous monitoring and statistical data and the consequent effect on the rigour of OIE’s evaluations) as well as the delays in the submission of self-evaluations and their validations, yet there appears to be no OAC attempt to deal with such problems e.g. exerting any pressure on the CDB or on the BMCs through their representatives on the Board.
To conclude: The OAC has an expressed interest in advancing the role of evaluation as a strategic tool for CDB management. However, it is not performing its oversight function with sufficient firmness to bring about any change to the problems raised through evaluations, especially with regard to data issues and reporting delays. More generally there is a lack of any systematic
170 At the APEC meeting in May 2012, it was agreed that the OIE would prepare a Management Action Record to highlight the follow up actions taken to the recommendations of all evaluation reports, every two years, with the first report presented to APEC at the March 2013 Board Meeting. There is no record of this having ever been done or of the APEC / OAC’s following up on such request.
306
report on “follow up of actions agreed” which could be particularly useful for tracking changes as a consequence of an evaluation and management’s response. The OIE could do better justice to its oversight responsibly if it were to receive all background documents systematically at least two weeks before its meetings. Moreover, the volume and length of documents received at any one time is considered to be overwhelming. The number and/or importance of agenda items competing for attention at any one session is an additional handicap.
the Panelss
the PanelssUnit has access to all needed information and information sources
Extent to which the evaluation unit has access to the organization’s
a) staff, records, and project sites;
b) co-financiers and other partners, clients; and
c) programs, activities, or entities it funds or sponsors
Complies –The available evidence suggests that there is no reason to doubt such access. But systematic and easily accessible documentation is lacking in the CDB; it is one of its weak points.. Delays in getting hold of the relevant documents can have consequences on the timeliness of evaluation studies
16) There appears to be a that is of concern to the Panels (More on this point under the heading self and independent evaluations.)The OIE is not regularly invited in any capacity to these meetings or given a copy of the agenda or minutes; the OIE is occasionally invited to attend in order to discuss an evaluation report or management feedback. Its observer status at meetingsor as a do not necessarily provide the same insight as to the dynamics of management actions and/or decisions.
. Extent to which the evaluation unit has control over:
a) staff hiring,
b) promotion, pay increases, and
c) firing, within a merit system
Partially complies - All OIE staff members are treated in the same way as other CDB staff. The Head has limited control over the hiring, firing or promotion of OIE staff.
actual or potential conflict of interestThe PanelIt must be s
The PanelisThe; this affects also of y, Work PracticesThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. therefore The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation pyramid171 are brought out in the remaining sections of this Review, not least given the limited resources available.
But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities. sTheThes before completing the final
171 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
307
version. However, p are only submitted to the CDB line and senior managers.Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the following the recommendations of professional good practices and standards on participative approaches; it has succeeded in , The Panelsstaff fromas well as There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. OIE doesarrange .sthe Panel also wishes tonewly appointed rsA was evidentthey expressed interest in In one case, interest was followed up in practice; can be improvedfostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that can offer can add value to understanding the strengths and weaknesses of such strategies. Tduring this transitional phase, Manual to guide and support the independent evaluation process.and operations staff sevaluation activities. ,oOIE’sjudginga . As such, theyare thatAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An is processed as follows;the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
SThisrItand Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
The rmade
- is still considered to be good practice to have the elaborated in the initial design documents the172 such as Developmental Evaluation (Patton, 2010173)
-)
said abovePCHowever, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessonsthe Panelas done this ing
172 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.173 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press
308
on lessonsAlthough nothing has happened since, it is , sometimes indicate (Panel has already referred above to ’s lack of oversight in the use of evaluation.)
sThe Panelsevaluation work Moreover, the 2015 budget provides only US$2’000 for
communication – nothing of which is intended for outreach.Reviewerseither confusing or and
budgetedConsequently for 2015, theFigure 3: The MDB Evaluation Pyramid174
174 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
309
ys to be effective. A modest attempt has been made in 2015; OIE hased But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.man and financial resources to support its work
OIE’s Human Resources;
5eThree of the five were recruited from within the CDB. edfrom the Board that OIE
should embark on ee and for impact evaluations in particular,OIE’s ee Moreover, there
are many other designated OIE activities that should be recognised as valuable work; the
validations, building CDB and BMC evaluation capacity, providing supervision, advice,
knowledge management and brokerage as well as managing evaluation contracts, The
time needs of dealing with all of these may be underestimated in OIE’s budgets; all are
important for assuring best value from evaluation. The Panel is concerned that a demand
for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level
evaluations may undermine the importance and time needs of other essential
tasks.Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses.
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies
310
does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.Self-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes.types of evaluation y There appears to be little incentive to complete self-evaluations in a timelier manner.
.
; it is a threat rather than an opportunity for learning. Yis recognized as
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.a number of , which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring dataadded value that evaluation might offer to the operations area is ill recognized Moreover, the link between self-evaluation as the building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
311
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”175
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
175 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
312
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”176 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 177 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
176 CDB (2011) Evaluation Policy (p.2)177
313
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
314
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
315
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
316
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
Independence of the Office of Independent Evaluation (OIEIndependence is absolutely central to the integrity and trustworthiness of evaluation. It is an agreed requirement within the development agencies and in the evaluation community as a whole. In examining the issue of independence and good practice, reviewers are guided by the Evaluation Cooperation Group’s recommendations on good practices, the CDB’s Evaluation Policy and by the 2011 consultancy review of independence relative to the CDB’s evaluation and oversight division178. The appraisal is based on a comparison of the ECG’s recommendations on independence179 and the current OIE status.
OIE and Independence: Recommendations from the OECD Evaluation Cooperation Group (ECG)
The ECG’s considers the issue of independence according to three specific areas: organisational, or structural independence, behavioural, or functional independence and protection from outside interference, or operational independence.
Organizational independence, ensures that the evaluation unit and staff are protected against any influence or control by senior or line management, and have unrestricted access to all documents and information sources needed for conducting their evaluations. Also, that the scope of evaluations selected can cover all relevant aspects of their institution.
Behavioural independence, generally refers to the evaluation unit’s autonomy in selecting and conducting setting its work programme and in producing quality reports which can be delivered without management interference.
Protection from outside interference refers to the extent to which the evaluation function is autonomous in setting its priorities, and conducting its studies and processes and in reaching its judgments, and in managing its human and budget resources without management interference.
Conflict of interest safeguards refers to protection against staff conflict of interests be they current, immediate, future or prior professional and personal relationships and considerations or financial interests for which there should be provision in the institution’s human resource policies.
The OIE’s Independence in Practice
Organisational / structural independenceOn the whole, the Panel acknowledges and commends the efforts being made by the CDB to assure OIE’s organisational independence. The CDB’s Evaluation Policy provides for the OIE’s organisational independence from line management and the interview data suggests that there is also wide acceptance and acknowledgement of why the OIE should have such independent status. Table 1 below provides our overall assessment of this aspect of OIE’s independence when compared with ECG recommendations. 180
178 Osvaldo Feinstein & Patrick G. Grasso, Consultants, May 2011 Consultancy to Review the Independence of the Evaluation and Oversight Division of the Caribbean Development Bank179 ECG 2014 Evaluation Good Practice Standards, Template for Assessing the Independence of Evaluation Organizations, Annexe II.1 180 Based on ECG (2014) Template for Assessing the Independence of Evaluation Organizations, Evaluation Good
Practice Standards, Annexe II.1
317
Table 1: OIE organisational independence compared with ECG recommendations
Aspects Indicators CDB Evaluation Policy (EP) and Practice
The structure and role of evaluation unit
Whether the evaluation unit has a mandate statement that makes clear its scope of responsibility extends to all operations of the organization, and that its reporting line, staff, budget and functions are organizationally independent from the organization’s operational, policy, and strategy departments and related decision-making
Partially Complies The Policy is broad enough to cover the full range of MDB type of evaluations. However in practice this would not be possible without additional human and budget resources
The unit is accountable to, and reports evaluation results to, the head or deputy head of the organization or its governing Board
Whether there is a direct reporting relationship between the unit, and
a) the Management, and/or
b) Board or
c) relevant Board Committee, of the institution
Complies - OIE reports to the Board of Directors (BoD) through its Oversight Assurance Committee (OAC)
The unit is located organizationally outside the staff or line management function of the program, activity or entity being evaluated
The unit’s position in the organization relative to the program, activity or entity being evaluated
Complies - The OIE is located outside, and is therefore independent of CDB line management
The unit reports regularly to the larger organization’s audit committee or other oversight body
Reporting relationship and frequency of reporting to the oversight body
Complies - The OIE reports x 5 per year to the OAC . Board approval for an additional executive meeting between the Head of the OIE and the OAC at least once per year was given in October 2015
The unit is sufficiently removed from political pressures to be able to report findings without fear of repercussions
Extent to which the evaluation unit and its staff are not accountable to political authorities, and are insulated from participation in political activities
Complies
Unit staffers are protected by a personnel system in which compensation, training, tenure and advancement are based on merit
Extent to which a merit system covering compensation, training, tenure and advancement is in place and enforced
Partially Complies - with CDB human resource policy. However the skill needs of OIE staff ought to be regularly reviewed in light of its move towards higher-level evaluations. Appraisal of skill needs and hiring of relevant staff should be completely under the authority of the Head of Evaluation. This is not sufficiently clear in the Policy or other documents we reviewed.
Unit has access to all needed information and information sources
Extent to which the evaluation unit has access to the organization’s
a) staff, records, and project sites;
b) co-financiers and other partners, clients; and
Complies –The available evidence suggests that there is no reason to doubt such access. But systematic and easily accessible documentation is lacking in the CDB; it is one of its weak points.. Delays in getting hold of the relevant documents can have consequences on the timeliness of
318
Aspects Indicators CDB Evaluation Policy (EP) and Practice
c) programs, activities, or entities it funds or sponsors
evaluation studies
However, independence should not mean isolation: There appears to be a detachment between the OIE and CDB that is of concern to the Panel; on the one hand, between the OIE and operations staff, and (2) on the other, in terms of the structural arrangements between the OIE and senior management.
17) In agreeing for the OIE to concentrate on strategic and thematic, in-depth evaluations, responsibility for project monitoring and evaluation were given over to operations. The division is clear and respected. However, it has its drawbacks. With the OIE no longer systematically involved at the front-end of project design, the monitoring data needs are likely to be poorly defined. Weak monitoring data will contribute to weaker evaluations. (More on this point under the heading self and independent evaluations.)
In the reviewers’ opinion, it is a common misunderstanding to assume that providing evaluator advice on monitoring and evaluation data will comprise evaluator independence. On the contrary, evaluation input into project design is essential to assure that the logic, indicators and data needs are addressed so that at some future point in time an evaluation of the achievements can be empirically grounded.
This is not to say that the OIE no longer has any influence at the front-end design stage; it has merely shifted the point of focus. The OIE is now systematically providing such input more generally to the corporate planning teams for the tools and systems they are developing to support the MfDR framework. The monitoring data for projects and their implementation should be improved once the Project Performance Evaluation System (PPES) and the Portfolio Performance Management System (PPMS) are updated and operational.
18) In the second place, the OIE has limited formal access to the Advisory Management Team (AMT) weekly meetings where the President and senior management gather to exchange up-to-date information on the dynamics of CDB policy and practice. The OIE is not regularly invited in any capacity to these meetings or given a copy of the agenda or minutes; the OIE is occasionally invited to attend in order to discuss an evaluation report or management feedback. For the OIE, this means that it is unlikely to pick up on the ‘when’ and ‘what’ of key decisional issues or provide input into the discussion based on evaluative information. Its observer status at Loans Committee meetings, or as a participant informer at the OAC and BoD meetings and discussions do not necessarily provide the same insight as to the dynamics of management actions and/or decisions. .
To respond to this situation, the President has agreed to meet regularly with the Head of the OIE in order to keep him up to date with CDB strategic thinking. This is a welcomed change.
OIE Independence and Behavioural Issues The Panel has concerns about some behavioural issues. For example, through both the interviews and documentary review, we learned of considerable delays in processing both the independent evaluation reports as well as OIE’s validation of the CDB’s self-evaluations. Delays are generally due to receiving feedback on the independent reports from first, the relevant operational department, then from the AMT, and then on providing the OIE with a management response that is initially drafted by operations staff before being reviewed by the AMT. (OIE reports cannot be submitted to the OAC without the relevant management response). This two-layer process for preparing submissions to the Board is inefficient and could potentially be a
319
threat to evaluation’s independence in the future by delaying OIE’s timely reporting to the OAC.
OIE validations of the CDB self-evaluations are also submitted to the OAC, but it is in both sides’ interest to clear up any misunderstandings beforehand. Despite attempts to improve the timeframe for completing these validations, delays are more the norm than the exception. Table 2 below summarises our assessment of the behavioural aspects of independence.
Table 2: OIE and Behavioural Independence
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Ability and willingness to issue strong, high quality, and uncompromising reports
Extent to which the evaluation unit:
a) has issued high quality reports that invite public scrutiny (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk) of the lessons from the organization’s programs and activities;
b) proposes standards for performance that are in advance of those in current use by the organization; and
c) critiques the outcomes of the organization’s programs, activities and entities
Partially complies – paucity of data and documentation sometimes hinder the quality of reports. The OIE emphasizes the learning part of evaluation, and is cautious in its criticism recognising that management is going through a transitory stage and can still be overly defensive.
Ability to report candidly
Extent to which the organization’s mandate provides that the evaluation unit transmits its reports to the Management/Board after review and comment by relevant corporate units but without management-imposed restrictions on their scope and comments
Partially complies - as sometimes reporting to the Board is compromised by delays in the review/comment process between the OIE and the CDB. Any delay with the production of a Management Response will also mean that submitting a report to the Board in a timely manner is impaired since the two have to be submitted together.
Transparency in the reporting of evaluation findings
Extent to which the organization’s disclosure rules permit the evaluation unit to report significant findings to concerned stakeholders, both internal and external (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk).
Who determines evaluation unit’s disclosure policy and procedures: Board, relevant committee, or management.
Partially complies - The OIE’s conforms to the CDB’s disclosure policy. However, the dissemination of evaluation findings appears to be currently restricted to website publication and reports to the Board. A more targeted communication strategy to include other key stakeholders, e.g. project implementers in the BMCs should be developed and put in place.
Self-selection of items for work program
Procedures for selection of work program items are chosen, through systematic or purposive means, by the evaluation organization; consultation on work program with Management and Board
Complies - The OIE also ensures that its work program is drawn up after consultation with both CDB Management and Board to seek their input on relevant topics and themes.
Protection of administrative budget, and other budget
Line item of administrative budget for evaluation determined in accordance with a clear policy parameter, and
Partially complies - The administrative budget for supporting OIE work is protected. Access to additional sources of
320
Aspects Indicators CDB Evaluation Policy (EP) and Practice
sources, for evaluation function
preserved at an indicated level or proportion; access to additional sources of funding with only formal review of content of submissions
funding is possible if well argued and justified. But the approval process is complex and inefficient. (See Figure 1 below)
OIE and Protection from External influence or interference
Our overall assessment is provided in Table 3 below. The OIE’s independence in the design, conduct and content of its evaluations does not appear to be subjected to any external interference. But securing funding from any sources outside the OIE’s administrative budget, i.e. from the Social Development Fund, is an unduly complex and long process. As such we consider that the current funding process can affect the OIE’s choice with regard to the type of evaluations it can undertake. (See Figures 1 and 2 below)
Table 3: OIE and its Independence from External influence or interference
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Proper design and execution of an evaluation
Extent to which the evaluation unit is able to determine the design, scope, timing and conduct of evaluations without Management interference
Complies – however within limits of restricted human and financial resources available
Evaluation study funding
Extent to which the evaluation unit is unimpeded by restrictions on funds or other resources that would adversely affect its ability to carry out its responsibilities
Partially Complies - OIE must work within the limits of the agreed administrative budget wherever possible. If additional resources are needed for studies it must seek alternative funds elsewhere. The budget limitations can have an affect on the type of evaluations undertaken and therefore its independence in terms of choice.
Judgments made by the evaluators
Extent to which the evaluator’s judgment as to the appropriate content of a report is not subject to overruling or influence by an external authority
Complies – the evidence available suggests that the Board and Management accept the evaluators’ independent interpretation and conclusions Management responses are agreed to be the accepted place to raise any difference of opinion.
Evaluation unit head hiring/firing, term of office, performance review and compensation
Mandate or equivalent document specifies procedures for the
a) hiring, firing,
b) term of office,
c) performance review, and d). compensation of the evaluation unit head that ensure independence from operational management
Complies – the Head of OIE is appointed by the CDB President in agreement with the OAC for a 5 year period which is renewable x 1. The Head could be removed from Office by the President or the Board but only with the agreement of both parties.
However the Head reports to the President for all administrative and personnel matters. Even though this was not recommended in the Osvaldo Feinstein & Patrick G. Grasso report on Independence in 2011, the BoD accepted CDB’s reasons for keeping this arrangement. (e.g.most OAC members are non residents and cannot oversee day-to-day work)
. Extent to which the evaluation unit has control over:
a) staff hiring,
Partially complies - All OIE staff members are treated in the same way as other CDB staff. The Head has limited control over the hiring, firing or promotion of OIE staff.
321
Aspects Indicators CDB Evaluation Policy (EP) and Practice
b) promotion, pay increases, and
c) firing, within a merit system
Continued staff employment
Extent to which the evaluator’s continued employment is based only on reasons related to job performance, competency or the need for evaluator services
Partially complies - Whilst the EP is clear about procedures for hiring, firing and promotion, all of which must conform with CDB human resource policy, there is nothing mentioned about any difference of opinion between the CDB and the Head of the OIE with regard to continued staff employment subject to changes in the level of technical or interpersonal competencies needed to meet new demands.
Avoidance of Financial, Personal or Professional conflicts of interest
This particular aspect refers to the organisation’s Human Resources Policy; there must be provisions in place to protect against actual or potential conflict of interest. The Panel requested via the OIE, to have evidence from human resources on any such provisions but did not receive an answer. It must be assumes that this aspect of independence, past or present, does indeed form part of normal CDB Human Resource Policies
To conclude: The Panel is impressed with the measures CDB has taken to assure the organisational independence of the OIE. Its independent status is accepted and respected by senior and line management. The OIE’s budget is not independent from the overall CDB administrative budget; this affects its choice of evaluation types or approaches. Some of the behavioural issues affecting independence were also of concern, especially due to the delays in the exchange of documents, between the OIE and operations departments, which has a direct effect on timely reporting to the OAC. As for protection from outside interference, our concerns are largely to do with OIE’s independence over staffing issue; there are potential loopholes in current arrangements that could undermine OIE’s autonomy over its staff.
OIE’s Strategy, Work Practices and Work ProgrammeThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. Much of the period 2012 to 2015 has therefore been taken up with preparing OIE’s shift in focus from project-based evaluations to the high-level thematic and in-depth strategic studies. This has meant adopting a three-way approach; (1) for self-evaluations, reducing its time input to support the process and (2) for independent evaluations, taking stock of the gaps in coverage and expertise, and (3) networking to share experiences with centres of expertise and align OIE with international practices. In addition, amongst other duties, it has been supporting the development of MfDR tools and systems such as the Project Performance Assessment System by providing advice and input on programme logic and monitoring needs. The OIE plans to conduct 2-4 high-level studies per year from 2016. The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the
322
time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation pyramid181 are brought out in the remaining sections of this Review, not least given the limited resources available.
To conclude: The OIE has made a first step in proposing a strategy for establishing itself as an independent evaluation resource. But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities.
The Value / Usefulness of OIE’s Independent EvaluationsEvaluation is a powerful tool that can provide useful, evidence-based information to help inform and influence policy and practice. But useful evaluations depend not only on the evaluators’ skills, but on several other important factors as well; 1) on planning evaluations to be relevant to the priorities of the organisation’s work and for their results to be delivered in time to be useful; on the degree of 2) consultation and ultimately ownership by those who seek evaluative information; on the 3) tools used to support the evaluation process per se; and on the 4) credibility and quality of the evaluation products182.
1. Planning relevant and timely evaluationsThe OIE is now working on a 3 year rolling work plan that sets out the broad areas for enquiry. So far, there are no agreed criteria for making the selection of the specific topics for independent evaluation, although the priorities tend to reflect those of the CDB’s strategic plan. Nevertheless decision-making is rather arbitrary based on a process of dialogue between the OIE and the CDB and the OIE and the Board.
One of the OIE’s two objectives for 2015 therefore, was to define a work plan and agree priorities based on an approach that is “utilisation-focused”. This means that the studies are selected and planned to be relevant and useful to the organisation’s needs.
The OIE has achieved this objective with respect to its latest studies, which concerns the Social Development Fund (SDF) Multicycle 6&7 Evaluation, the Haiti Country Strategy evaluation and the evaluation of the CDB’s Policy Based Operations. Each of these three have been planned to deliver their results in time to provide the CDB Board of Directors with relevant information for negotiating the next round of funding. In spite of some delays due to a myriad of reasons, not least to the extra effort needed to secure essential data, the studies are expected to deliver on time.
The processes for agreeing OIE’s work plan and specific evaluations on the one hand, and, in securing alternative funding on the other, are shown in Figure 1 below. The Panel was surprised at learning how bureaucratic (the internal approval process), and inefficient (in view of the time it takes) the process seems to be. The concern here is that such a process could possibly pose a threat to assuring the Board of “timely studies.”
Figure 1: Selection of Evaluation Topics and Funding Source
181 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).182 These aspects reflect the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
Consultation with CDB Operations and OAC/Board for selection of
evaluation topic
3-year Work Programme and Budget (approved by Board)
Annual OIE report and work plan
submission to OAC
323
2. Consultation and ownership“The credibility of evaluations depends to some degree on whether and how the organization’s approach
to evaluation fosters partnership and helps build ownership and capacity in developing countries.”
(ECG good practices)
Internal review of Approach Paper
Specific Evaluation Study Design and Budgeting
OIE Draft Terms of Reference / Approach
Paper
Detailed ToR or Final Approach Paper if sufficiently detailed.
Finalise Approach Paper and submit to OAC/Board
Final Approach Paper
OAC ApprovalOAC minutes
Paper
Funding Track
Final Approach Paper/ToR
Board approval necessary If above USD
150,000
Board notification only if USD 150,000 or
below
Board Approval
Board Paper
OIE – Selection of consultants (if any) contracting
OIE Admin Budget or …
… SDF
Prepare TA Paper (content similar to Approach Paper but different
format.
TA Paper
Approval – Internal Loans Committee
OIE – Selection of consultants (if any)
contracting
324
The OIE engages with the OAC, CDB senior management and operations for agreeing its 3-year work plan and then for selecting the specific topics and themes. It also discusses the evaluation approach paper (design and implementation plan) with the CDB and OAC before completing the final version. However, preliminary and final drafts of the report are only submitted to the CDB line and senior managers for comment and factual errors. Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the results and their implications. Discussions with the OAC are more limited due to the overburdened agenda of OAC and Board meetings, as previously discussed.
In short, the OIE is to be commended for following the recommendations of professional good practices and standards on participative approaches; it has succeeded in having introduced a modus operandi that involves the key players in the selection of evaluation topics, the evaluation designs and their results. Figure 2 below provides an overview of the evaluation implementation and stakeholder engagement processes.
Figure 2: Evaluation Study Implementation and Feedback Loops
Arrangement AFully outsourced / external
consultants; oversight by OIE
Preparations:Detailed evaluation plan (incl tools,
timeline, etc.) and logistics
Production of Inception Report / Approach Paper
Arrangement BConducted by OIE
staff
Arrangement CJointly: external
consultants and OIE
Terms of Reference
Prepares Inception Report /
Approach Paper
Presentation/workshop:Interim findings and conclusions for immediate feedback and validation
Data Collection and Analysis
OIE
Summary and ppt for workshop presentation
and discussion with CDBSubmission of Draft Final
Report to OIE
Final OIE approved report to CDB Senior Management for Management Response
Board notification only if USD 150,000 or
below
Draft Final Report
Review loops – OIE and CDB (potentially also BMC)
Feedback to evaluation lead
Submission of Final Report to
OIE
325
15.16.
Notes to Figure 2
15. The OIE informed the Panel that this is an abbreviated version as there are e.g. additional steps (secondary processes) when evaluations are procured (tendering or single source), when there are additional review loops and updates to OAC etc.
16. OAC may also decide to return the report to OIE, the Panel were informed, or demand from Management specific actions based on the report.
This process is engaging and appears to have secured senior management and OAC interest and buy-in as witnessed in the latest studies. But there is the downside too! The process takes much time and, in our view, is partly unnecessary. The Panel appreciates that staff from operations as well as the AMT may both want to confer on an appropriate management response, but this should not be the case for reviewing an independent report for factual errors. The two-phase approach seems somewhat inefficient and unnecessary in our opinion.
Contact between the OIE, the CDB and/or the OAC during the actual study implementation is most often restricted to the occasional progress report, particularly when studies run behind time. There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. The OIE does, however, arrange discussions for reflecting on emerging findings, but we are not sure of how systematic this feedback loop is.
More generally speaking, outside of an evaluation study, the OIE has limited dealings with operations. The OIE has an advisory role in providing them with help, particularly with providing training, guidelines and tools to support self-evaluations. We are nevertheless concerned about the seeming distance between these two and how this has affected the perceived value of evaluation. (For further on this point, please see the section below on “Self- and Independent Evaluations”)
But the Panel also wishes to stress that this is not the case for newly appointed senior managers. A much more open attitude to evaluation and appreciation of its potential value was evident;
Prepare for disclosure and dissemination
Final Report
Final Report and Management Response submitted to
OAC/BoardFinal Report and
Mgt. Resp.
Management Response
OIE ApprovalFinal Report and Management Response considered by CDB
AMT
OAC/Board endorsed
326
they expressed interest in drawing out important lessons on what works, how, for whom, and under what conditions. In one case, interest was followed up in practice; the OIE was recently invited by a senior manager to share evaluative knowledge and experience with his staff regarding policy based operations.
Certainly, we can say that overall, the key stakeholders within the CDB are adequately integrated into the evaluation process as to foster their buy-in and ownership. But more generally, we feel that the utility of independent evaluations can be improved by fostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that evaluation can offer can add value to understanding the strengths and weaknesses of such strategies. This however cannot be done overnight and takes a long time.
3. Tools to support the evaluation processSo far, during this transitional phase, the OIE has mainly focussed on improving the tools to support the operations areas’ self-evaluations. This has left the OIE with little time to produce the checklists or tools to support its own studies. There are plans to develop an OIE Manual to guide and support the independent evaluation process. Such plans should be encouraged, as these documents will form a very important part of training, particularly for newcomers to the OIE team.
In the meantime, the OIE and operations staff refers to the Performance Assessment System (PAS) Manuals for evaluation activities. The manuals are based on DAC criteria and ECG principles. Much emphasis is given to the rating system and how and what should be rated. However we find them lengthy, unwieldy and overcomplicated. Moreover, such manuals should be used for reference, but cannot and should not replace first-hand training in how to plan, conduct and manage the evaluation process.
Quality Assessment (QA) and Quality at Entry (QaE)
There was a transition period between 2012 and 2014 to establish the OIE. Work on the PAS, QaE, PCRs, ARPP, which had started earlier, was therefore completed after OIE came into existence, but it effectively had no formal ‘home’ in operations. The Panel was told that there had been some discussions about creating a Quality Assurance unit within CDB (OPS) but the current status is unclear.
The QaE Guidance Questionnaire was developed before and completed by the OIE. It was used to assess the documents that came across to the OIE for comments at the Review Stage. The results were then sent to the Portfolio Manager/Project Coordinator indicating any gaps/issues that needed to be addressed or clarified. QaE Guidance Questionnaires were developed for all the Bank’s lending products, CSP and to assess the quality of supervision.
After the QaE was launched bank wide, several operations officers saw the merit in using the QaE Guidance Questionnaire in the field and adopted it as a tool for their use during the appraisal mission in order to cross check and test their data collection and analysis.
OIE’s use of the QaE was discontinued in 2014 due to limited resources and a stronger focus on evaluations. It still sometimes comments on specific appraisals, but very selectively.
Both QaE and QaS (quality at supervision) are also addressed in the PAS Manuals. In addition the QaE and PAS have been incorporated in Volume 2 of the Operations Manual OPPM.
The Review Panel assessed the QaE forms. They are relatively standard, adapted to the specificities of the CDB. They contribute to judging a project’s expected quality in a relatively objective way. As such, they are are helpful, as a benchmark, in the ex-post assessment of projects.
The Panel considers that the lack of an established Quality Unit in the CDB (and independent from OIE) is a weakness that should be addressed in the near future.
327
4. Credibility and Quality of Evaluation ProductsAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An independent evaluation is processed as follows; the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
Since 2012, the OIE has produced a range of studies and approach papers. This review is based on those listed below as provided by the OIE, and cover the period from May 2012 to December 2015. It includes 3 evaluations (in blue), 4 Assessment studies (in brown) 14 validations of self-evaluations (in green) and 3 Approach Papers (in purple) for upcoming evaluations. These are listed below in Table 4.
Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
Board Meeting
Date Type / Topic
251 May 2012 Ex-Post Evaluation Report on Road Improvement and Maintenance Project, Nevis -St. Kitts and Nevis.
Validation of Project Completion Report on Sites and Services – Grenada. Assessment of Effectiveness of Implementation of Poverty Reduction
Strategy 2004-09.253 Oct. 2012 Assessment of Extent and Effectiveness of Mainstreaming Environment,
Climate Change, Disaster Management at CDB.254 Dec. 2012 Assessment of the Implementation Effectiveness of the Gender Equality
Policy and Operational Strategy of the Caribbean Development Bank. Validation of Project Completion Report on Enhancement of Technical and
Vocational Education and Training – Belize. Validation of Project Completion Report on Fourth Road (Northern Coastal
Highway Improvement Section 1 of Segment II) Project – Jamaica. Assessment of the Effectiveness of the Policy-based Lending Instrument.
256 May 2013 Validation of Project Completion Report on Expansion of Grantley Adams International Airport – Barbados.
Validation of Project Completion Report on Fifth Water Supply Project – Saint Lucia.
261 May 2014 Validation of Project Completion Report on Immediate Response Loan, Tropical Storm Gustav, Jamaica.
Validation of Project Completion Report on Social Investment Fund, Jamaica.
Validation of Project Completion Report on Disaster Mitigation and Restoration – Rockfall and Landslip, Grenada.
263 Oct. 2014 Validation of Project Completion Report on Basic Education Project – Antigua and Barbuda
263 Oct. 2014 Approach Paper for SDF 6 & 7 Multicycle Evaluation
264 Dec. 2014 Validation of Project Completion Report on Policy-Based Loan – Anguilla
328
Validation of Project Completion Report on Immediate Response Loan - Tropical Storm Arthur – Belize.
Evaluation of Technical Assistance Interventions of the Caribbean Development Bank Related To Tax Administration and Tax Reform in The Borrowing Member Countries 2005-2012.
265 March
2015
Approach Paper for the Evaluation of Policy Based Operations
266 May 2015 Validation of Project Completion Report on Upgrading of Ecotourism Sites – Dominica
The Evaluation of the Caribbean Development Bank’s Intervention in Technical and Vocational Education and Training (1990-2012)
267 July 2015 Validation of Project Completion Report on The Belize Social Investment Fund I Project − Belize
268 Oct.2015 Approach Paper Country Strategy and Programme Evaluation, Haiti
The review and analysis of these documents is based on the UNEG Quality Checklist for Evaluation Reports (http://www.uneval.org/document/detail/607) as well as on ECG guidance (Big Book on Good Practice Standards).
Approach Papers
Three Approach Papers (APs) were made available to the panel (see Table [ref] above). An AP describes the rationale for the evaluation, the background to the topic evaluated, the evaluation framework (criteria and questions) and approach. It also describes the team and provides an initial planning. Being the first main deliverable of OIE’s evaluation process, APs are the starting point and therefore a major determining element in the roll-out of each evaluation. Therefore APs “have to get it right”.
The APs examined are clearly written, well-structured and of reasonable length.183 We were surprised to find, however, that they do not make explicit the objectives of the evaluated intervention(s), e.g., through a clear objective tree, or through an explicit theory of change, intervention logic or logframe. Whilst one of the APs contains, in an appendix, a results framework for the evaluation, the results framework for the intervention (PBO) itself is lacking.
Inception reports
Only one Inception Report was given to the Panel for review (SDF 6&7). This gives an in-depth description of the evaluated programme and provides a clear Theory of Change. It is good practice that this is established after a pilot field mission, which helps to amend the initial AP on the basis of field observations and sharpen the evaluation questions if needed.
However, it is still considered to be good practice to have the Theory of Change elaborated in the initial design documents . This would facilitate OIE evaluations after project completion. Establishing the Theory of Change of any intervention would be included in the QaE form more explicitly, to be developed between the Quality unit referred to above, and OIE.
Evaluations and Assessments
Three evaluations and four assessment reports completed during the review period were considered. Assessments are similar to evaluations but have a narrower scope; they focus on a limited set of aspects or judgment criteria, mainly effectiveness, i.e. achievement of objectives.
183 Opportunities remain of course to be more concise and to move parts to appendices, e.g., detailed descriptions of the evaluation team or part of the description of the evaluated intervention.
329
Evaluations generally base their judgment on the internationally recognised DAC criteria as well as aspects of the CDB and BMC’s management of the intervention.
In general, these reports are of reasonable quality. In the main, they explain the evaluated object184 and provide evaluation objectives. The findings are organised around the evaluation criteria or questions detailed in the scope and objectives section of the report. They are based on evidence derived from data collection and analysis methods as described in the methodology section. The reports tend to dwell on the limitations that the evaluation encountered, but without becoming defensive. In one case (PBL Assessment) the report starts with a summary of the reviews on the topic done by other MDBs. This was a pleasant surprise and indeed a good practice that could well be adopted in future evaluations too.
However, the reports also show several significant weaknesses:
- Reports do not always provide clear (reconstructed) intervention logics or theories of change for the intervention(s) evaluated.185 Evaluation criteria and questions are defined at a fairly general level. They are translated into more precise “research questions” (in an “Evaluation Design Matrix”, for each project for each criterion). However, it is unclear how these questions relate to the intervention logic (as this is not made explicit). This may be done in inception reports (of which, as noted above, only one was available for review), but should be done also in the final reports.
- The reports do not describe the link from the evaluation questions to the answers, how the evaluation judgments are made and how these ultimately transform into ratings for each criterion and each project. In other words, the explanation provided in the evaluation frameworks is inadequate. The “evaluation design matrix” currently used does not provide sufficient insight into how ultimately an intervention’s performance is judged.186 Links between findings, conclusions and recommendations could be improved by making this more explicit. In other words, reports should include the story on how the evaluand is credibly linked to any observed outcomes and impacts, and should be clear on how causal claims are made.
- With the exception of the PBL Assessment, reports are lengthy and detailed. One reason for this is an over-emphasis on ratings. Their detailed discussion, project by project, criterion by criterion, occupies a very prominent position in the evaluation reports’ main body of text. Although ratings are traditionally an important element in evaluations of MDBs, too strong an emphasis can be tedious and may distract the reader from the real lessons to be drawn. The detailed discussion of ratings, and their evidence base, would be better placed in an Appendix, with a brief summary in the main report. This would help give the lessons and recommendations a more prominent position than is now the case. This would also help make the evaluation reports not only shorter but also more interesting to read; this could help add value to evaluation’s image within the organisation.
- The reviewers feel that the OIE evaluations tend to over-emphasise objective-based evaluation187 and the DAC criteria to the exclusions of considering other evaluation
184 Sometimes in great length: for instance with the SDF 6&7 multicycle evaluation report it is only at page 30 that we find the beginning of the report on findings…185 Again with the SDF 6&7 evaluation, it is said to be guided by a “Logic Model” which is not explained.186 Marlène: I moreover have the idea that the methodology (often described as “visits”) is based on interviews and little hard evidence. Any view on this?.JM: My “interview-based evaluations”!!187 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.
330
approaches such as Developmental Evaluation (Patton, 2010188); evaluation should be case specific and answer the actual information needs of managers and other decisions makers rather than always concentrating on final performance.
- Related to the previous point (and again with the exception of the PBL Assessment) executive summaries (approximately 8 pages) are too long. For the evaluation report to increase potential impact, they would need to be reduced to 2 to 3 pages and be more focused; again this could be done by dwelling less on the individual ratings of projects and more on key findings, lessons and conclusions. More generally, reports could be better adapted to the needs of the different audiences. Although not strictly limited to evaluations, The Health Evidence Network Reports189 are a model that could be adapted for evaluation reporting purposes; they are specifically geared towards addressing policy and decision-making.
- The “Recommendations to BMCs” are an interesting feature of the reports, (although we are unsure to what degree such recommendations could be effectively followed up by OIE or the Bank, but certainly could taken up with BMC Board members.
- Reports (e.g. the evaluation report on Technical Assistance) focus much on technical problems that were encountered during the evaluation. Although these are important issues, again to improve the report’s flow and “readability” this section would be better placed in the Appendix. What counts is the story of the intervention, not the story of the evaluation (see “Limitations” section in the TA report for instance)
OIE Validations of Project and Country Strategy Programme Completion Reports (referred to globally as PCRs hereafter)
As said above, the OIE has the mandate to validate the Project and Economic departments PCRs and CSPCRs. However, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
The validations tend to repeat the different items reported in the PCRs and then provide extensive comment on each. The PCVRs go into great depth and detail, which makes the documents rich and complete. This is their strength – but also their weakness. The depth and level of detail, as well as the repetitions from the original PCRs, makes PCVRs (overly) lengthy (20-40 pages) and difficult to read. The OIE reported spending approximately 27.2% of its time on validating PCRs in 2015 compared with 44.4% on its core work, i.e. doing or managing the higher level evaluations. That is more than half of its evaluation work is being spent on the validation process. Finally, the PCVRs now seem to be, to a great extent, a standalone output of OIE. It is not always clear to us how they are being used as the “building blocks” for the OIE’s independent evaluations. Making this clearer in the independent evaluations would help show the link and therefore the value of the time being spent on the self-evaluation validations.
To conclude, the review finds that the OIE has taken steps to improve the perceived utility of evaluation in several ways. In the first instance, by planning its work to provide relevant and timely evidence geared towards helping the Board with its oversight and decision making tasks. The topics are selected through dialogue between the OIE and key CDB stakeholders and reflect
188 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press189 See the reports available at the WHO’s Health Evidence Netowkr at http://www.euro.who.int/en/data-and-evidence/evidence-informed-policy-making/health-evidence-network-hen
331
priorities of the CDBs strategic plan. Secondly, by securing the interest and consequently the buy-in of the OAC and CDB senior management through engaging their input throughout the evaluation process. This is evidenced by the reported interest in the latest three studies, the Country strategy programme in Haiti, the evaluation of policy-based operations and the SDF 6& 7 multicycle assessment.
The OIE products are of an acceptable quality and could be even better if some of the shortcomings were addressed. However, the products themselves do not impair the utility of OIE’s work; this is undermined in several ways: (1) by the time delays in commenting on PCRs (OIE) and providing feedback to the independent evaluations (operations and management) (2) by the inefficient processes for agreeing topics and funding sources as well as providing OIE with management responses to its reports.
Putting Evaluation to Use: transparency, feedback and follow-upThere are several ways that evaluation can be, and is being used. As John Mayne has pointed out in his many publications on the issue,190 when we talk of evaluation use, we are mainly thinking about its Instrumental use—use made to directly improve programming and performance. But there is also conceptual use - use which often goes unnoticed or more precisely, unmeasured. This refers to the kind of use made to enhance knowledge about the type of intervention under study in a more general way. Or even Reflective use— this refers to using discussions or workshops to encourage and support reflection on the evaluation findings to see how they might contribute to future strategies.
In the case of the CDB there is some evidence to suggest that “use” is not only instrumental, but other types are also developing. For example, in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessons. (Reflective use)
Another important use, as recommended by the ECG, is that from time to time a synthesis of lessons is drawn from a number of evaluations and made available publically. In fact the Panel was impressed to hear that in the past, the evaluation unit had done this drawing on lessons from evaluations of the power sector. (Conceptual use) Although nothing has happened since, it is now on the “to do list” for 2016 (OIE’s 2016 Work Plan).
As for instrumental use, responsibility for using the knowledge generated through evaluation and for possibly drawing up an action plan of what should be done is up to CDB senior management and the relevant CDB department and division. Oversight on applying recommendations and picking up on the lessons drawn is the responsibility of the OAC.
Evidence on how evaluations have actually contributed to decisions or negotiations is lacking or confusing, Certainly the OIE is unaware of the extent to which its evaluations are put to use. On the one hand, the OAC minutes sometimes indicate that lessons learned are integrated into the next phase. On the other hand, the reviewers were told that often in the past, the evaluation results were “too old” to be of use as the lessons had already been drawn and used way before the report was completed. Similarly, people’s gaps in memory on how well the evaluative information from previous studies may have been used may also account for the scarcity of evidence.
In response, the Panel questioned CDB staff and the OIE about a particular study, the Technical and Vocational Education and Training Assessment. The feedback was somewhat contradictory. On the one hand, the study was criticised as “confirming” news rather than bringing “new news”. However, on the other, we learned that In October 2015, the Board of Directors approved a proposal for the revision of CDB’s Education and Training Policy and Strategy. Work on this has already begun and an external consultant has been engaged to lead the process.
190 See for example, his opening chapter to Enhancing Evaluation use: Insights from internal Evaluation Units, Läubli Loud, M. and Mayne, J. 2014, Sage Publications
332
Although it is one of the OIE’s tasks to set up a database on results and lessons learned from evaluations, so far this has not been a priority. There is also currently no systematic tracking of lessons or recommendations arising from the evaluations, or on any progress in their uptake. (The Panel has already referred above to OAC’s lack of oversight in the use of evaluation.)
The OIE’s role in supporting CDB’s organisational learning is clearly specified in the Evaluation Policy, with many good suggestions for knowledge sharing activities such as “brown-bag lunches, workshops, pamphlets and short issues papers” (p. 19). So far, however, the OIE’s lead role on the knowledge sharing side appears to be quite limited. It has provided advisory input in Loan Committee discussions, and organises workshops together with the relative operations department for discussing the implications of evaluation studies. Ultimately, of course, the uptake of evaluation results and knowledge is in the hands of management. But the evaluation unit has an important role to play in terms of knowledge broker and knowledge manager. Both have tended to be underplayed in OIE’s work plan so far.
Transparency: The Communication Strategy
In recent times and with the approval of its new Disclosure Policy, the CDB has started to post its independent evaluation reports on its website. (There is nothing on the self-evaluations). The website also presents a good overview of the role and function of the OIE and evaluation within the CDB. This is a step in the right direction for sharing information. However, in our view, the CDB’s communication strategy is the weakest part of the evaluation system to date.
The Panel has already commended the OIE in its efforts to engage the CDB and the OAC in evaluation work. But reporting and communicating the lessons seem to be entirely targeted at the Board and the CDB. Moreover, the 2015 budget provides only US$2’000 for communication – nothing of which is intended for outreach.
Reviewers feel that actively engaging with the more indirect stakeholders, for example project implementers in the BMCs, NGOs or project beneficiaries is relatively weak191. There appears to be little reflection on drawing out significant messages for the broader group of stakeholders, or on how then to transmit them to the “right” people in the “right” way (knowledge brokerage).
To conclude, evidence on the uptake of evaluation is either confusing or sparse. It is unfortunate that so far no systematic record keeping system has been put into place to track lessons learned or the uptake of recommendations (or actions agreed from management responses). The OIE plays a weak role in brokering the knowledge generated through evaluations to the benefit of external partners and in managing such knowledge. Although the Evaluation Policy specifies the need for “distilling evaluation findings and lessons learned in appropriate formats for targeted audiences both within and outside the CDB” (p.19) such a targeted communication strategy has yet to be developed and budgeted.
Strengthening Evaluation Capacities and Networking From the onset in 2012, the OIE has stressed the importance of developing and strengthening evaluation capacities within the OIE, the CDB and, subject to available resources, in borrowing member countries. Building evaluation capacity in BMCs and the CDB is one of the OIE’s mandated tasks. It has been a priority that figures on the work plan from the beginning (Work Programme and Budget 2012-2104) The idea of developing an internship programme for graduates from the Caribbean region was one idea that was advanced to help build local evaluation resources. However, the capacity-building has primarily been focused on OIE and CDB staff to date. One of the OIE’s two objectives for 2015 therefore was to take up the challenge and “strengthen evaluation capacities and networking” to include reaching out to the BMCs.
191 A broader communication strategy is one of the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
333
Developing OIE staff capacities
The change from project level to strategic and thematic evaluations does require different evaluative skills and competencies. The MDB Evaluation Pyramid presented below in Figure 3 shows the different types of evaluation and changing resource needs as one ascends the pyramid. Implicit here also is the change in the type of expertise and competencies needed as evaluation aspires to the higher levels.
Consequently for 2015, the OIE set itself the objective of networking and developing working partnerships with regional and international evaluation entities and academic institutions. The rationale was twofold: (1) secure further support and guidance as well as (2) increase its outreach and coverage through joint work and international exposure. Another implicit aim was to benefit from partners’ contacts in the BMCs wherever possible so as to improve data collection and quality.
Figure 3: The MDB Evaluation Pyramid192
The OIE has therefore linked up with Carleton University in Canada and the University of the West Indies, Barbados campus. The OIE was also approached by the Development Bank of South Africa to exchange experiences about setting up an evaluation entity in a “small” development bank. However, its attempt to become a member of the Evaluation Cooperation Group was not successful for reasons beyond its control.
The OIE is to be commended in addressing the issue of staff competencies and professional development more generally. New developments in evaluation as well as new developments in the scope of OIE’s work may necessitate new competencies. For this reason, organisations such as the International Developmental Evaluation Association have recommended that the
192 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
334
competencies of evaluators and evaluation managers should be periodically reviewed. Several publications now exist on competency requirements and suggestions for the periodic review of staff competencies.193
It is not within this remit to compare and contrast OIE’s competencies with those recommended by international and national agencies. However, what we can say is that the OIE demonstrates great forethought in taking this on board.
Capacity building within CDB
The OIE’s objective also consists of continuing to develop measures for improving the monitoring and self-evaluation side of CDB’s work. OIE’s strategy here is to use the windows of opportunity on offer through some of the training sessions that are being organised by CDB as part of its shift towards MfDR e.g. by Corporate Planning Services and Technical Assistance. For 2016 it is also planned to have the OIE present at the annual staff meeting and Learning Forum.
The OIE also organises some ad hoc training with operations, for example to help understand new tools e.g. for drawing out lessons from self-evaluation reports and, more generally, in helping staff appreciate how evaluation can add value to the organisation’s work. Measures include providing advisory services on demand, and providing training alongside the introduction of new or revised tools.
Capacity building in the BMCs
This is an ambitious task and would require additional investment; from the bi-annual work plans to be effective. A modest attempt has been made in 2015; from what we understand, the OIE has joined together with the Carleton University and the University of the West Indies, using their networks in some of the BMCs, to try to develop this aspect.
To conclude, we cannot comment on the quality or reaction to such training, but can commend the OIE for making capacity building one of its priority objectives. From both the Policy and the documents we reviewed, we note that capacity building was always seen to be an important aspect of OIE’s work, but hitherto has received little strategic focus. But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.
Adequacy of the OIE’s human and financial resources to support its work
OIE’s Human Resources;
The OIE is has a staff of 5; the head, 1 senior evaluation officer and two evaluation managers, plus one administrative assistant. Three of the five were recruited from within the CDB. The limited capacity means that it is not feasible to cover all the types of evaluation activities outlined in the Evaluation Policy. Yet there is some indication from the Board that OIE should embark on impact evaluations at some future stage. An increasing demand for evaluation and for impact evaluations in particular, would run the risk of overstretching the OIE’s capacity to deliver credible and useful evaluations. Moreover, there are many other designated OIE activities that should be recognised as valuable work; the validations, building CDB and BMC evaluation capacity, providing supervision, advice, knowledge management and brokerage as well as managing evaluation contracts, The time needs of dealing with all of these may be underestimated in OIE’s budgets; all are important for assuring best value from evaluation. The Panel is concerned that a demand for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level evaluations may undermine the importance and time needs of other 193 E.g. IDEAS, (2012) Competencies for Development Evaluation Evaluators, Managers and Commissioners, the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice (2010) and the Swiss Evaluation Society’s Evaluation Managers Competencies Framework (2014)
335
essential tasks.
Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses.
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.
Self- and independent evaluationSelf-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes. Both types of evaluation are important as they are at the very heart of the evaluation function; they are said to be the building blocks for the more strategic evaluations that the OIE is now undertaking.
The Evaluation Coordination Group recommends that the self-evaluations be carried out by the relevant operations department and in turn, reviewed and validated by the organisation’s independent evaluation office. The CDB’s Evaluation Policy therefore talks of “validating all self-evaluations” as being one of OIE’s essential oversight tasks.
336
Within CDB, the self-evaluations should provide management with performance assessments and thereby serve an accountability function to the CDB and Board. To support the process, the OIE provides operations with manuals and checklists for guidance. Once a self-evaluation report is to hand, it is given over to the OIE for the validation of its technical quality and credibility.194
However, in the CDB case, there are well-documented issues that have affected the quality and timeliness of the self-evaluations on the one hand, and therefore the quality of the foundation on which to build the independent evaluations. Paucity of documentation within CDB, paucity of data collected and available in the Borrowing Member Countries (BMCs), time delays in producing completion reports and in turn, having them validated by the OIE - all such issues were systematically raised during interviews and in some of the independent evaluation reports. There appears to be little incentive to complete self-evaluations in a timelier manner.
Generally speaking, many of the monitoring data problems appear to be due to a lack of management oversight. For example, with the introduction of results-based management, the logic frame and monitoring and data needs are systematically being built into intervention design. However, the BMCs are not delivering the data as contractually agreed at the outset. Incentives to support any significant change towards building a results-based culture seem to be weak and sanctions seem to be rarely enforced when the supply of data is lacking or lengthy delays to the projects occur. Although we can appreciate the complexities of trying to enforce monitoring compliance, this means that often, project deadlines have had to be extended, data gaps are not being satisfactorily dealt with and in turn, there has been a void in the quality and quantity of available evidence for the CDB’s self-assessment of project performance. For some time, this lack of oversight has been tolerated. Part of the problem is the low priority accorded to completing the self-evaluation reports by operations, coupled with the absence of any focal point within senior management to drive the process and deal with the problems.
No record is kept of how the self-evaluation results are actually used. They do not appear on the CDB website, but we were told that the findings are integrated into the following project designs. Hence we are somewhat unclear as to the utility of these reports at present. The situation is exacerbated by a rather confused image of evaluation: some operations staff consider OIE’s input (through validations or independent evaluations) to be sometimes over-critical, regulatory and adding little value; it is a threat rather than an opportunity for learning. Yet at the same time, evaluation is recognized as an integral part of result-based management.
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.
Some managers however seem to start changing the status quo. For example a revised and simplified template for producing project completion reports is being considered, and mid-term project reviews are expected to be more stringent in looking at monitoring plans and practices and tying disbursements to performance. In some cases we also learned of incentives being introduced to encourage project managers to complete their reports in a timelier manner. But much remains to be done and, since the OIE is no longer responsible for monitoring and project evaluations, there is a void that needs to be filled. It is up to line managers to drive this work forward.
To conclude, it is fair to say that in view of a number of “frustrations” between the OIE and operations, which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring data, the added value that evaluation might offer to the operations area is ill recognized. Moreover, the link between self-evaluation as the
194 According to the Evaluation Policy, OIE should validate all PCRs and CCRs but due to the backlog of reports and the delay in completing them (sometimes years later) since October 2015, the OIE has secured OAC agreement to validate a maximum of 6 per year, which are selected in consultation with the OAC.
337
building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
338
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”195
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
195 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
339
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”196 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 197 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
196 CDB (2011) Evaluation Policy (p.2)197
340
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
341
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
342
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
343
mendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
344
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
345
APPENDICES
Appendix I - The External Review Mandate – Terms of Reference and Approach Paper
Appendix II -Review Approach, Data collection and Analysis, and Limitations
Appendix III – Overview of OIE Evaluation Practice
Appendix IV - List of Persons Interviewed
Appendix V - List of Documents Reviewed
Appendix VI- List of Topics used to guide interviews with members of CDB Board of Directors
Appendix VII - List of Topics used to guide interviews with CDB staff
346
Appendix III – Overview of OIE Evaluation Practice (prepared by the OIE in response to Reviewer’s request)
Caribbean Development Bank, Office of Independent Evaluation - OIE
Category Response
Percentage of projects subject to project (self-) evaluation
100% - Project Completion Reports (PCR)
Percentage of projects subject to validation by OIE
Approximately 40-50%
About 15 projects exit portfolio annually. Evaluation Policy calls for all PCR to be validated. However, OIE resources insufficient. Validation process reviewed in 2014. Now OAC (Board committee) selects a sample of 6-8 PCR for validation each year.
Percentage/number of projects subject to in-depth review by OIE
None – unless specifically requested by OAC
Due to limited resources, focus of OIE evaluation work programme is on PCR validations and high-level evaluations – including country strategy and programme evaluations (CSPE).
Number of high-level evaluations conducted by OIE (e.g. sector, thematic, geographic)
1-2 per year since 2011
Plan is 2-4 per year from 2016. This would include CSPE (1st planned for Q1 2016: Haiti)
Number of project impact evaluations conducted by OIE
None
OIE includes “impact questions” in high-level evaluations.
Number of project impact evaluations conducted by Bank staff or other non-OIE staff
OIE is not aware of any impact evaluation conducted by the Bank.
However, OIE provides technical support to the Basic Needs Trust Fund (BNTF) in its design of an M&E framework that entails impact evaluations.
Budget In USD mn: 0.78 in 2015; 0.82 in 2016. This is equivalent to about 2.5% of total CDB Administrative Budget.
75% of the budget is for Staff salaries (4 Professionals, 1 Support staff), leaving around USD 190,000 (in 2015) for other expenses, including consultants e.g. for external evaluations. Additional funding is accessed via the Special Development Fund (SDF). This varies according to type and scope of the evaluation, e.g. the ongoing SDF 6/7 Evaluation is SDF funded at USD 255,000.
Budget determined by Board, not separate from administrative budget.
SDF funding for evaluations is considered separately and subject to Bank internal approval process. SDF funding cannot be used to cover OIE expenses such as staff time or travel. Country eligibility for SDF funding is also a consideration. OIE expressed concerns about this funding track in respect to predictability, independence and eligibility limitations.
Head of OIE reports to Board, with administrative link to the President
Terms of appointment for Head
347
5 year term, renewable once. Appointed by the President with the agreement of the Board.
Right of Return for Head Not eligible for other staff positions.
Consultants as proportions of OIE budget
2015: 19% (USD 145,000)
Plus SDF funding. SDF funded evaluations are outsourced.
Last external evaluation (or peer review) of OIE
No external evaluation, though a review of the function was done in 2011, leading to the Evaluation Policy.
OIE External Review completed in April, 2016
Departments or special programmes supporting impact evaluation
None
348
Appendix IV – List of Persons Interviewed
Name Function relative to OIE Type interview
Mrs. Colleen Wainwright Member CDB Board of Directors (UK)
Face to face
Mrs. Cherianne Clarke Alternate MemberCDB Board of Directors (UK)
Face to face
Mrs. Jean McCardle Member CDB Board of Directors (Canada)
Face to face
Dr. Louis Woodroofe MemberCDB Board of Directors (Barbados)
Mr. A: de Brigard Former Member CDB Board of Directors
Skype interview
Mr. H. Illi Fromer Member CDB Board ofDirectors
Telephone interview
Mrs. Claudia Reyes Nieto Member CDB Board of Directors
Telephone interview
Mr. Bu Yu alternate DirectorCDB Board of Directors
Face to face
Mr. Michael Schroll(Barbados)
Head OIE
series of interviews viaSkype and face-to-face
Mr. Mark Clayton OIE Senior Evaluation Officer Focus GroupMrs. Egene Baccus Latchman OIE Evaluation OfficerMr. Everton Clinton OIE Evaluation OfficerMrs. Valerie Pilgrim OIE Evaluation Officer
Dr. Justin Ram CDB Director Economics Department
Face to face
Mr. Ian Durant CDB Deputy Director Economics Dept Face to faceDr. Wm Warren Smith CDB President
Joint interviewFace to face
Mrs. Yvette Lemonias-Seale CDB Vice President Corporate Services & Bank Secretariat
Mr. Denis Bergevin CDB Deputy DirectorInternal Audit
Face to face
Mr. Edward Greene CDB Division Chief, Technical Cooperation Division
Face to face
Mrs. Monica La Bennett CDB Deputy Director Corporate Planning Face to faceMrs. Patricia McKenzie CDB Vice President Operations Face to faceMs. Deidre Clarendon CDB Division Chief
Social Sector DivisionFace to face
Mrs. Cheryl Dixon CDB Co-ordinator, Environmental Sustainability Unit
Focus group
Mrs. Denise Noel- Debique CDB Gender Equality Advisor Mrs. Tessa Williams-Robertson CDB Head Renewable EnergyMrs. Klao Bell-Lewis CDB Head Corporate Communications Face to faceMr. Daniel Best CDB Director
Projects DepartmentFace to face
Mr. Carlyle Assue CDB Director Face to face
349
Finance Department
350
Appendix VI - Interview Guide: Members of CDB Board of Directors
Below are a list of themes that I should like to raise with you based on your experience and knowledge of the CDB’s independent evaluation function (Office of Independent
Evaluation).
In each case, I should be grateful if you could illustrate your responses with examples or help this Review by, wherever possible, sending me (or telling me where I can find) any
documents that could support your responses.
This guide is being sent to you in advance to help prepare our meeting. However, our interview will be conducted more in the style of a conversation. The following sub-questions will be used to GUIDE the interview. Please feel encouraged to raise any
additional issues that you feel we should take into account
On the governance and Independence of CDB’s evaluation functionWhat mechanisms are there in place to support its independence?
How satisfactory are the current arrangements in your opinion?
How is the balance between independence and the need for interaction with line management dealt with by the system? For example, what mechanisms exist to ensure that the OIE is kept up to date with decisions, policy / programme changes, other contextual changes etc that could have an affect on OIE evaluation studies / evaluation planning?
On the OIE’s Evaluation PolicyThe CDB’s Evaluation Policy was established in 2011. To what degree do you feel it is adequate? Still relevant?
What suggestions do you have for any improvements?
In your opinion, how adequate is the current quality assurance system for over viewing the evaluation function?
On the quality and credibility of evaluation studiesTo what degree do you believe the reports are fair and impartial?
Do you consider them to be of good quality? Are they credible?
Are you adequately consulted/involved on evaluations of interest to you?
On the relevance and usefulness of evaluations How well does the OIE engage with you / your committee during the preparation, implementation and reporting of an evaluation study to assure that it will be useful to the CDB?
How are the priorities set for the independent evaluations? What criteria are used? Are you satisfied with the current procedure?
351
When OIE evaluation studies are outsourced to external consultants, what criteria are used to make this decision?
How are the priorities for the OIE’s 3.year rolling work plan agreed? In your opinion, is the current plan adequate in terms of coverage and diversity?
In your opinion, do the evaluations address important and pressing programs and issues?
To what extent do you feel that the OIE’s evaluations integrate the cross-cutting theme such as gender, energy efficiency/renewable energy, climate change? What improvements might be made and how?
On the dissemination and uptake of evaluation findings and recommendationsTo what extent do you feel that evaluation findings are communicated to the CDB and its stakeholders in a
a) useful, b) constructive andc) timely manner?
Are evaluation recommendations useful? Realistic?
What mechanisms are in place to assure that evaluation results are taken into account in decision making and planning? What improvements do you feel could be made?
How have you used the findings from any evaluations? Examples?
To what degree do you feel that evaluation contributes to institutional learning? And what about to institutional accountability? Any examples?
What mechanisms are in place to ensure that knowledge from evaluation is accessible toCDB staff and other relevant stakeholders? Are the current arrangements satisfactory?
How satisfied are you with current arrangements? What expectations do you have for the future?
On resourcesHow is the OIE resourced financially and is this satisfactory?
What about the OIE staff, are all the important areas of expertise represented in the team?
On this Review of the Office of Independent EvaluationWhat are your expectations? What are you particularly hoping to learn from it?
Thank you very much for your cooperation and input
352
Appendix VII : Interview Pro-Forma – CDB Staff membersThis presents a list of the topics raised during interviews. It was used to guide the open-ended
discussion – this means that the sequence and exact wording of the questions may not necessarily have followed in this order or been asked in exactly this way.
Changeover to an Independent Evaluation Office? Expectations? Advantages and disadvantages??
Satisfaction with working relations between operations and the OIE from your perspective?
Process of dealing with the PCRs and CCRs? Advantages and limitations?
Quality and credibility of the validation process?
How are the self-evaluation reports used?
Credibility and Quality of OIE’s evaluation reports
Communication of self and OIE independent evaluations? To whom, in what way? Possible improvements?
actual or potential conflict of interestThe PanelIt must be s
The PanelisThe; this affects also of y, Work PracticesThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. therefore The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation
353
pyramid198 are brought out in the remaining sections of this Review, not least given the limited resources available.
But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities. sTheThes before completing the final version. However, p are only submitted to the CDB line and senior managers.Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the following the recommendations of professional good practices and standards on participative approaches; it has succeeded in , The Panelsstaff fromas well as There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. OIE doesarrange .sthe Panel also wishes tonewly appointed rsA was evidentthey expressed interest in In one case, interest was followed up in practice; can be improvedfostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that can offer can add value to understanding the strengths and weaknesses of such strategies. Tduring this transitional phase, Manual to guide and support the independent evaluation process.and operations staff sevaluation activities. ,oOIE’sjudginga . As such, theyare thatAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An is processed as follows;the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
SThisrItand Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
The rmade
- is still considered to be good practice to have the elaborated in the initial design documents the199 such as Developmental Evaluation (Patton, 2010200)
-)
said abovePCHowever, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s
198 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).199 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.200 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press
354
agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessonsthe Panelas done this ing on lessonsAlthough nothing has happened since, it is , sometimes indicate (Panel has already referred above to ’s lack of oversight in the use of evaluation.)
sThe Panelsevaluation work Moreover, the 2015 budget provides only US$2’000 for
communication – nothing of which is intended for outreach.Reviewerseither confusing or and
budgetedConsequently for 2015, theFigure 3: The MDB Evaluation Pyramid201
201 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
355
ys to be effective. A modest attempt has been made in 2015; OIE hased But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.man and financial resources to support its work
OIE’s Human Resources;
5eThree of the five were recruited from within the CDB. edfrom the Board that OIE
should embark on ee and for impact evaluations in particular,OIE’s ee Moreover, there
are many other designated OIE activities that should be recognised as valuable work; the
validations, building CDB and BMC evaluation capacity, providing supervision, advice,
knowledge management and brokerage as well as managing evaluation contracts, The
time needs of dealing with all of these may be underestimated in OIE’s budgets; all are
important for assuring best value from evaluation. The Panel is concerned that a demand
for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level
evaluations may undermine the importance and time needs of other essential
tasks.Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses.
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies
356
does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.Self-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes.types of evaluation y There appears to be little incentive to complete self-evaluations in a timelier manner.
.
; it is a threat rather than an opportunity for learning. Yis recognized as
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.a number of , which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring dataadded value that evaluation might offer to the operations area is ill recognized Moreover, the link between self-evaluation as the building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
357
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”202
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
202 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
358
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”203 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 204 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
203 CDB (2011) Evaluation Policy (p.2)204
359
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
360
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
361
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
362
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
Independence of the Office of Independent Evaluation (OIEIndependence is absolutely central to the integrity and trustworthiness of evaluation. It is an agreed requirement within the development agencies and in the evaluation community as a whole. In examining the issue of independence and good practice, reviewers are guided by the Evaluation Cooperation Group’s recommendations on good practices, the CDB’s Evaluation Policy and by the 2011 consultancy review of independence relative to the CDB’s evaluation and oversight division205. The appraisal is based on a comparison of the ECG’s recommendations on independence206 and the current OIE status.
OIE and Independence: Recommendations from the OECD Evaluation Cooperation Group (ECG)
The ECG’s considers the issue of independence according to three specific areas: organisational, or structural independence, behavioural, or functional independence and protection from outside interference, or operational independence.
Organizational independence, ensures that the evaluation unit and staff are protected against any influence or control by senior or line management, and have unrestricted access to all documents and information sources needed for conducting their evaluations. Also, that the scope of evaluations selected can cover all relevant aspects of their institution.
Behavioural independence, generally refers to the evaluation unit’s autonomy in selecting and conducting setting its work programme and in producing quality reports which can be delivered without management interference.
Protection from outside interference refers to the extent to which the evaluation function is autonomous in setting its priorities, and conducting its studies and processes and in reaching its judgments, and in managing its human and budget resources without management interference.
Conflict of interest safeguards refers to protection against staff conflict of interests be they current, immediate, future or prior professional and personal relationships and considerations or financial interests for which there should be provision in the institution’s human resource policies.
The OIE’s Independence in Practice
Organisational / structural independenceOn the whole, the Panel acknowledges and commends the efforts being made by the CDB to assure OIE’s organisational independence. The CDB’s Evaluation Policy provides for the OIE’s organisational independence from line management and the interview data suggests that there is also wide acceptance and acknowledgement of why the OIE should have such independent status. Table 1 below provides our overall assessment of this aspect of OIE’s independence when compared with ECG recommendations. 207
205 Osvaldo Feinstein & Patrick G. Grasso, Consultants, May 2011 Consultancy to Review the Independence of the Evaluation and Oversight Division of the Caribbean Development Bank206 ECG 2014 Evaluation Good Practice Standards, Template for Assessing the Independence of Evaluation Organizations, Annexe II.1 207 Based on ECG (2014) Template for Assessing the Independence of Evaluation Organizations, Evaluation Good
Practice Standards, Annexe II.1
363
Table 1: OIE organisational independence compared with ECG recommendations
Aspects Indicators CDB Evaluation Policy (EP) and Practice
The structure and role of evaluation unit
Whether the evaluation unit has a mandate statement that makes clear its scope of responsibility extends to all operations of the organization, and that its reporting line, staff, budget and functions are organizationally independent from the organization’s operational, policy, and strategy departments and related decision-making
Partially Complies The Policy is broad enough to cover the full range of MDB type of evaluations. However in practice this would not be possible without additional human and budget resources
The unit is accountable to, and reports evaluation results to, the head or deputy head of the organization or its governing Board
Whether there is a direct reporting relationship between the unit, and
a) the Management, and/or
b) Board or
c) relevant Board Committee, of the institution
Complies - OIE reports to the Board of Directors (BoD) through its Oversight Assurance Committee (OAC)
The unit is located organizationally outside the staff or line management function of the program, activity or entity being evaluated
The unit’s position in the organization relative to the program, activity or entity being evaluated
Complies - The OIE is located outside, and is therefore independent of CDB line management
The unit reports regularly to the larger organization’s audit committee or other oversight body
Reporting relationship and frequency of reporting to the oversight body
Complies - The OIE reports x 5 per year to the OAC . Board approval for an additional executive meeting between the Head of the OIE and the OAC at least once per year was given in October 2015
The unit is sufficiently removed from political pressures to be able to report findings without fear of repercussions
Extent to which the evaluation unit and its staff are not accountable to political authorities, and are insulated from participation in political activities
Complies
Unit staffers are protected by a personnel system in which compensation, training, tenure and advancement are based on merit
Extent to which a merit system covering compensation, training, tenure and advancement is in place and enforced
Partially Complies - with CDB human resource policy. However the skill needs of OIE staff ought to be regularly reviewed in light of its move towards higher-level evaluations. Appraisal of skill needs and hiring of relevant staff should be completely under the authority of the Head of Evaluation. This is not sufficiently clear in the Policy or other documents we reviewed.
364
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Unit has access to all needed information and information sources
Extent to which the evaluation unit has access to the organization’s
a) staff, records, and project sites;
b) co-financiers and other partners, clients; and
c) programs, activities, or entities it funds or sponsors
Complies –The available evidence suggests that there is no reason to doubt such access. But systematic and easily accessible documentation is lacking in the CDB; it is one of its weak points.. Delays in getting hold of the relevant documents can have consequences on the timeliness of evaluation studies
However, independence should not mean isolation: There appears to be a detachment between the OIE and CDB that is of concern to the Panel; on the one hand, between the OIE and operations staff, and (2) on the other, in terms of the structural arrangements between the OIE and senior management.
19) In agreeing for the OIE to concentrate on strategic and thematic, in-depth evaluations, responsibility for project monitoring and evaluation were given over to operations. The division is clear and respected. However, it has its drawbacks. With the OIE no longer systematically involved at the front-end of project design, the monitoring data needs are likely to be poorly defined. Weak monitoring data will contribute to weaker evaluations. (More on this point under the heading self and independent evaluations.)
In the reviewers’ opinion, it is a common misunderstanding to assume that providing evaluator advice on monitoring and evaluation data will comprise evaluator independence. On the contrary, evaluation input into project design is essential to assure that the logic, indicators and data needs are addressed so that at some future point in time an evaluation of the achievements can be empirically grounded.
This is not to say that the OIE no longer has any influence at the front-end design stage; it has merely shifted the point of focus. The OIE is now systematically providing such input more generally to the corporate planning teams for the tools and systems they are developing to support the MfDR framework. The monitoring data for projects and their implementation should be improved once the Project Performance Evaluation System (PPES) and the Portfolio Performance Management System (PPMS) are updated and operational.
20) In the second place, the OIE has limited formal access to the Advisory Management Team (AMT) weekly meetings where the President and senior management gather to exchange up-to-date information on the dynamics of CDB policy and practice. The OIE is not regularly invited in any capacity to these meetings or given a copy of the agenda or minutes; the OIE is occasionally invited to attend in order to discuss an evaluation report or management feedback. For the OIE, this means that it is unlikely to pick up on the ‘when’ and ‘what’ of key decisional issues or provide input into the discussion based on evaluative information. Its observer status at Loans Committee meetings, or as a participant informer at the OAC and BoD meetings and discussions do not necessarily provide the same insight as to the dynamics of management actions and/or decisions. .
To respond to this situation, the President has agreed to meet regularly with the Head of the OIE in order to keep him up to date with CDB strategic thinking. This is a welcomed change.
OIE Independence and Behavioural Issues The Panel has concerns about some behavioural issues. For example, through both the interviews and documentary review, we learned of considerable delays in processing both the
365
independent evaluation reports as well as OIE’s validation of the CDB’s self-evaluations. Delays are generally due to receiving feedback on the independent reports from first, the relevant operational department, then from the AMT, and then on providing the OIE with a management response that is initially drafted by operations staff before being reviewed by the AMT. (OIE reports cannot be submitted to the OAC without the relevant management response). This two-layer process for preparing submissions to the Board is inefficient and could potentially be a threat to evaluation’s independence in the future by delaying OIE’s timely reporting to the OAC.
OIE validations of the CDB self-evaluations are also submitted to the OAC, but it is in both sides’ interest to clear up any misunderstandings beforehand. Despite attempts to improve the timeframe for completing these validations, delays are more the norm than the exception. Table 2 below summarises our assessment of the behavioural aspects of independence.
Table 2: OIE and Behavioural Independence
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Ability and willingness to issue strong, high quality, and uncompromising reports
Extent to which the evaluation unit:
a) has issued high quality reports that invite public scrutiny (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk) of the lessons from the organization’s programs and activities;
b) proposes standards for performance that are in advance of those in current use by the organization; and
c) critiques the outcomes of the organization’s programs, activities and entities
Partially complies – paucity of data and documentation sometimes hinder the quality of reports. The OIE emphasizes the learning part of evaluation, and is cautious in its criticism recognising that management is going through a transitory stage and can still be overly defensive.
Ability to report candidly
Extent to which the organization’s mandate provides that the evaluation unit transmits its reports to the Management/Board after review and comment by relevant corporate units but without management-imposed restrictions on their scope and comments
Partially complies - as sometimes reporting to the Board is compromised by delays in the review/comment process between the OIE and the CDB. Any delay with the production of a Management Response will also mean that submitting a report to the Board in a timely manner is impaired since the two have to be submitted together.
Transparency in the reporting of evaluation findings
Extent to which the organization’s disclosure rules permit the evaluation unit to report significant findings to concerned stakeholders, both internal and external (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk).
Who determines evaluation unit’s disclosure policy and procedures: Board, relevant committee, or management.
Partially complies - The OIE’s conforms to the CDB’s disclosure policy. However, the dissemination of evaluation findings appears to be currently restricted to website publication and reports to the Board. A more targeted communication strategy to include other key stakeholders, e.g. project implementers in the BMCs should be developed and put in place.
Self-selection of items for work program
Procedures for selection of work program items are chosen, through systematic or purposive means, by the
Complies - The OIE also ensures that its work program is drawn up after consultation with both CDB Management
366
Aspects Indicators CDB Evaluation Policy (EP) and Practice
evaluation organization; consultation on work program with Management and Board
and Board to seek their input on relevant topics and themes.
Protection of administrative budget, and other budget sources, for evaluation function
Line item of administrative budget for evaluation determined in accordance with a clear policy parameter, and preserved at an indicated level or proportion; access to additional sources of funding with only formal review of content of submissions
Partially complies - The administrative budget for supporting OIE work is protected. Access to additional sources of funding is possible if well argued and justified. But the approval process is complex and inefficient. (See Figure 1 below)
OIE and Protection from External influence or interference
Our overall assessment is provided in Table 3 below. The OIE’s independence in the design, conduct and content of its evaluations does not appear to be subjected to any external interference. But securing funding from any sources outside the OIE’s administrative budget, i.e. from the Social Development Fund, is an unduly complex and long process. As such we consider that the current funding process can affect the OIE’s choice with regard to the type of evaluations it can undertake. (See Figures 1 and 2 below)
Table 3: OIE and its Independence from External influence or interference
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Proper design and execution of an evaluation
Extent to which the evaluation unit is able to determine the design, scope, timing and conduct of evaluations without Management interference
Complies – however within limits of restricted human and financial resources available
Evaluation study funding
Extent to which the evaluation unit is unimpeded by restrictions on funds or other resources that would adversely affect its ability to carry out its responsibilities
Partially Complies - OIE must work within the limits of the agreed administrative budget wherever possible. If additional resources are needed for studies it must seek alternative funds elsewhere. The budget limitations can have an affect on the type of evaluations undertaken and therefore its independence in terms of choice.
Judgments made by the evaluators
Extent to which the evaluator’s judgment as to the appropriate content of a report is not subject to overruling or influence by an external authority
Complies – the evidence available suggests that the Board and Management accept the evaluators’ independent interpretation and conclusions Management responses are agreed to be the accepted place to raise any difference of opinion.
Evaluation unit head hiring/firing, term of office, performance review and compensation
Mandate or equivalent document specifies procedures for the
a) hiring, firing,
b) term of office,
c) performance review, and d). compensation of the evaluation unit head that ensure independence from operational management
Complies – the Head of OIE is appointed by the CDB President in agreement with the OAC for a 5 year period which is renewable x 1. The Head could be removed from Office by the President or the Board but only with the agreement of both parties.
However the Head reports to the President for all administrative and personnel matters. Even though this was not recommended in the Osvaldo Feinstein & Patrick G. Grasso report on Independence in 2011, the BoD accepted CDB’s reasons for keeping this arrangement. (e.g.most OAC members are non residents and cannot
367
Aspects Indicators CDB Evaluation Policy (EP) and Practice
oversee day-to-day work)
. Extent to which the evaluation unit has control over:
a) staff hiring,
b) promotion, pay increases, and
c) firing, within a merit system
Partially complies - All OIE staff members are treated in the same way as other CDB staff. The Head has limited control over the hiring, firing or promotion of OIE staff.
Continued staff employment
Extent to which the evaluator’s continued employment is based only on reasons related to job performance, competency or the need for evaluator services
Partially complies - Whilst the EP is clear about procedures for hiring, firing and promotion, all of which must conform with CDB human resource policy, there is nothing mentioned about any difference of opinion between the CDB and the Head of the OIE with regard to continued staff employment subject to changes in the level of technical or interpersonal competencies needed to meet new demands.
Avoidance of Financial, Personal or Professional conflicts of interest
This particular aspect refers to the organisation’s Human Resources Policy; there must be provisions in place to protect against actual or potential conflict of interest. The Panel requested via the OIE, to have evidence from human resources on any such provisions but did not receive an answer. It must be assumes that this aspect of independence, past or present, does indeed form part of normal CDB Human Resource Policies
To conclude: The Panel is impressed with the measures CDB has taken to assure the organisational independence of the OIE. Its independent status is accepted and respected by senior and line management. The OIE’s budget is not independent from the overall CDB administrative budget; this affects its choice of evaluation types or approaches. Some of the behavioural issues affecting independence were also of concern, especially due to the delays in the exchange of documents, between the OIE and operations departments, which has a direct effect on timely reporting to the OAC. As for protection from outside interference, our concerns are largely to do with OIE’s independence over staffing issue; there are potential loopholes in current arrangements that could undermine OIE’s autonomy over its staff.
OIE’s Strategy, Work Practices and Work ProgrammeThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. Much of the period 2012 to 2015 has therefore been taken up with preparing OIE’s shift in focus from project-based evaluations to the high-level thematic and in-depth strategic studies. This has meant adopting a three-way approach; (1) for self-evaluations, reducing its time input to support the process and (2) for independent evaluations, taking stock of the gaps in coverage and expertise, and (3) networking to share experiences with centres of expertise and align OIE with international practices. In addition, amongst other duties, it has been supporting the development of MfDR tools and systems such as the Project Performance Assessment System by providing advice and input on programme logic and monitoring needs. The OIE plans to conduct 2-4 high-level studies per year from 2016. The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
368
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation pyramid208 are brought out in the remaining sections of this Review, not least given the limited resources available.
To conclude: The OIE has made a first step in proposing a strategy for establishing itself as an independent evaluation resource. But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities.
The Value / Usefulness of OIE’s Independent EvaluationsEvaluation is a powerful tool that can provide useful, evidence-based information to help inform and influence policy and practice. But useful evaluations depend not only on the evaluators’ skills, but on several other important factors as well; 1) on planning evaluations to be relevant to the priorities of the organisation’s work and for their results to be delivered in time to be useful; on the degree of 2) consultation and ultimately ownership by those who seek evaluative information; on the 3) tools used to support the evaluation process per se; and on the 4) credibility and quality of the evaluation products209.
1. Planning relevant and timely evaluationsThe OIE is now working on a 3 year rolling work plan that sets out the broad areas for enquiry. So far, there are no agreed criteria for making the selection of the specific topics for independent evaluation, although the priorities tend to reflect those of the CDB’s strategic plan. Nevertheless decision-making is rather arbitrary based on a process of dialogue between the OIE and the CDB and the OIE and the Board.
One of the OIE’s two objectives for 2015 therefore, was to define a work plan and agree priorities based on an approach that is “utilisation-focused”. This means that the studies are selected and planned to be relevant and useful to the organisation’s needs.
The OIE has achieved this objective with respect to its latest studies, which concerns the Social Development Fund (SDF) Multicycle 6&7 Evaluation, the Haiti Country Strategy evaluation and the evaluation of the CDB’s Policy Based Operations. Each of these three have been planned to deliver their results in time to provide the CDB Board of Directors with relevant information for negotiating the next round of funding. In spite of some delays due to a myriad of reasons, not least to the extra effort needed to secure essential data, the studies are expected to deliver on time.
The processes for agreeing OIE’s work plan and specific evaluations on the one hand, and, in securing alternative funding on the other, are shown in Figure 1 below. The Panel was surprised at learning how bureaucratic (the internal approval process), and inefficient (in view of the time it takes) the process seems to be. The concern here is that such a process could possibly pose a threat to assuring the Board of “timely studies.”
208 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).209 These aspects reflect the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
369
Figure 1: Selection of Evaluation Topics and Funding Source
Consultation with CDB Operations and OAC/Board for selection of
evaluation topic
Internal review of Approach Paper
Specific Evaluation Study Design and Budgeting
OIE Draft Terms of Reference / Approach
Paper
Detailed ToR or Final Approach Paper if sufficiently detailed.
Finalise Approach Paper and submit to OAC/Board
Final Approach Paper
OAC ApprovalOAC minutes
Paper
Funding Track
Final Approach Paper/ToR
3-year Work Programme and Budget (approved by Board)
Board approval necessary If above USD
150,000
Board notification only if USD 150,000 or
below
Board Approval
Board Paper
Annual OIE report and work plan
submission to OAC
OIE – Selection of consultants (if any) contracting
OIE Admin Budget or …
… SDF
Prepare TA Paper (content similar to Approach Paper but different
format.
TA Paper
Approval – Internal Loans Committee
OIE – Selection of consultants (if any)
contracting
370
2. Consultation and ownership“The credibility of evaluations depends to some degree on whether and how the organization’s approach
to evaluation fosters partnership and helps build ownership and capacity in developing countries.”
(ECG good practices)
The OIE engages with the OAC, CDB senior management and operations for agreeing its 3-year work plan and then for selecting the specific topics and themes. It also discusses the evaluation approach paper (design and implementation plan) with the CDB and OAC before completing the final version. However, preliminary and final drafts of the report are only submitted to the CDB line and senior managers for comment and factual errors. Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the results and their implications. Discussions with the OAC are more limited due to the overburdened agenda of OAC and Board meetings, as previously discussed.
In short, the OIE is to be commended for following the recommendations of professional good practices and standards on participative approaches; it has succeeded in having introduced a modus operandi that involves the key players in the selection of evaluation topics, the evaluation designs and their results. Figure 2 below provides an overview of the evaluation implementation and stakeholder engagement processes.
Figure 2: Evaluation Study Implementation and Feedback Loops
Arrangement AFully outsourced / external
consultants; oversight by OIE
Preparations:Detailed evaluation plan (incl tools,
timeline, etc.) and logistics
Production of Inception Report / Approach Paper
Arrangement BConducted by OIE
staff
Arrangement CJointly: external
consultants and OIE
Terms of Reference
Prepares Inception Report /
Approach Paper
Presentation/workshop:Interim findings and conclusions for immediate feedback and validation
Data Collection and Analysis
OIE
Summary and ppt for workshop presentation
and discussion with CDBSubmission of Draft Final
Report to OIE
Board notification only if USD 150,000 or
below
Draft Final Report
Review loops – OIE and CDB (potentially also BMC)
371
17.18.
Notes to Figure 2
17. The OIE informed the Panel that this is an abbreviated version as there are e.g. additional steps (secondary processes) when evaluations are procured (tendering or single source), when there are additional review loops and updates to OAC etc.
18. OAC may also decide to return the report to OIE, the Panel were informed, or demand from Management specific actions based on the report.
This process is engaging and appears to have secured senior management and OAC interest and buy-in as witnessed in the latest studies. But there is the downside too! The process takes much time and, in our view, is partly unnecessary. The Panel appreciates that staff from operations as well as the AMT may both want to confer on an appropriate management response, but this should not be the case for reviewing an independent report for factual errors. The two-phase approach seems somewhat inefficient and unnecessary in our opinion.
Contact between the OIE, the CDB and/or the OAC during the actual study implementation is most often restricted to the occasional progress report, particularly when studies run behind time. There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. The OIE does, however, arrange discussions for reflecting on emerging findings, but we are not sure of how systematic this feedback loop is.
Prepare for disclosure and dissemination
Final OIE approved report to CDB Senior Management for Management Response
Feedback to evaluation lead
Submission of Final Report to
OIE
Final Report
Final Report and Management Response submitted to
OAC/BoardFinal Report and
Mgt. Resp.
Management Response
OIE ApprovalFinal Report and Management Response considered by CDB
AMT
OAC/Board endorsed
372
More generally speaking, outside of an evaluation study, the OIE has limited dealings with operations. The OIE has an advisory role in providing them with help, particularly with providing training, guidelines and tools to support self-evaluations. We are nevertheless concerned about the seeming distance between these two and how this has affected the perceived value of evaluation. (For further on this point, please see the section below on “Self- and Independent Evaluations”)
But the Panel also wishes to stress that this is not the case for newly appointed senior managers. A much more open attitude to evaluation and appreciation of its potential value was evident; they expressed interest in drawing out important lessons on what works, how, for whom, and under what conditions. In one case, interest was followed up in practice; the OIE was recently invited by a senior manager to share evaluative knowledge and experience with his staff regarding policy based operations.
Certainly, we can say that overall, the key stakeholders within the CDB are adequately integrated into the evaluation process as to foster their buy-in and ownership. But more generally, we feel that the utility of independent evaluations can be improved by fostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that evaluation can offer can add value to understanding the strengths and weaknesses of such strategies. This however cannot be done overnight and takes a long time.
3. Tools to support the evaluation processSo far, during this transitional phase, the OIE has mainly focussed on improving the tools to support the operations areas’ self-evaluations. This has left the OIE with little time to produce the checklists or tools to support its own studies. There are plans to develop an OIE Manual to guide and support the independent evaluation process. Such plans should be encouraged, as these documents will form a very important part of training, particularly for newcomers to the OIE team.
In the meantime, the OIE and operations staff refers to the Performance Assessment System (PAS) Manuals for evaluation activities. The manuals are based on DAC criteria and ECG principles. Much emphasis is given to the rating system and how and what should be rated. However we find them lengthy, unwieldy and overcomplicated. Moreover, such manuals should be used for reference, but cannot and should not replace first-hand training in how to plan, conduct and manage the evaluation process.
Quality Assessment (QA) and Quality at Entry (QaE)
There was a transition period between 2012 and 2014 to establish the OIE. Work on the PAS, QaE, PCRs, ARPP, which had started earlier, was therefore completed after OIE came into existence, but it effectively had no formal ‘home’ in operations. The Panel was told that there had been some discussions about creating a Quality Assurance unit within CDB (OPS) but the current status is unclear.
The QaE Guidance Questionnaire was developed before and completed by the OIE. It was used to assess the documents that came across to the OIE for comments at the Review Stage. The results were then sent to the Portfolio Manager/Project Coordinator indicating any gaps/issues that needed to be addressed or clarified. QaE Guidance Questionnaires were developed for all the Bank’s lending products, CSP and to assess the quality of supervision.
After the QaE was launched bank wide, several operations officers saw the merit in using the QaE Guidance Questionnaire in the field and adopted it as a tool for their use during the appraisal mission in order to cross check and test their data collection and analysis.
OIE’s use of the QaE was discontinued in 2014 due to limited resources and a stronger focus on evaluations. It still sometimes comments on specific appraisals, but very selectively.
373
Both QaE and QaS (quality at supervision) are also addressed in the PAS Manuals. In addition the QaE and PAS have been incorporated in Volume 2 of the Operations Manual OPPM.
The Review Panel assessed the QaE forms. They are relatively standard, adapted to the specificities of the CDB. They contribute to judging a project’s expected quality in a relatively objective way. As such, they are are helpful, as a benchmark, in the ex-post assessment of projects.
The Panel considers that the lack of an established Quality Unit in the CDB (and independent from OIE) is a weakness that should be addressed in the near future.
4. Credibility and Quality of Evaluation ProductsAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An independent evaluation is processed as follows; the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
Since 2012, the OIE has produced a range of studies and approach papers. This review is based on those listed below as provided by the OIE, and cover the period from May 2012 to December 2015. It includes 3 evaluations (in blue), 4 Assessment studies (in brown) 14 validations of self-evaluations (in green) and 3 Approach Papers (in purple) for upcoming evaluations. These are listed below in Table 4.
Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
Board Meeting
Date Type / Topic
251 May 2012 Ex-Post Evaluation Report on Road Improvement and Maintenance Project, Nevis -St. Kitts and Nevis.
Validation of Project Completion Report on Sites and Services – Grenada. Assessment of Effectiveness of Implementation of Poverty Reduction
Strategy 2004-09.253 Oct. 2012 Assessment of Extent and Effectiveness of Mainstreaming Environment,
Climate Change, Disaster Management at CDB.254 Dec. 2012 Assessment of the Implementation Effectiveness of the Gender Equality
Policy and Operational Strategy of the Caribbean Development Bank. Validation of Project Completion Report on Enhancement of Technical and
Vocational Education and Training – Belize. Validation of Project Completion Report on Fourth Road (Northern Coastal
Highway Improvement Section 1 of Segment II) Project – Jamaica. Assessment of the Effectiveness of the Policy-based Lending Instrument.
256 May 2013 Validation of Project Completion Report on Expansion of Grantley Adams International Airport – Barbados.
374
Validation of Project Completion Report on Fifth Water Supply Project – Saint Lucia.
261 May 2014 Validation of Project Completion Report on Immediate Response Loan, Tropical Storm Gustav, Jamaica.
Validation of Project Completion Report on Social Investment Fund, Jamaica.
Validation of Project Completion Report on Disaster Mitigation and Restoration – Rockfall and Landslip, Grenada.
263 Oct. 2014 Validation of Project Completion Report on Basic Education Project – Antigua and Barbuda
263 Oct. 2014 Approach Paper for SDF 6 & 7 Multicycle Evaluation
264 Dec. 2014 Validation of Project Completion Report on Policy-Based Loan – Anguilla Validation of Project Completion Report on Immediate Response Loan -
Tropical Storm Arthur – Belize. Evaluation of Technical Assistance Interventions of the Caribbean
Development Bank Related To Tax Administration and Tax Reform in The Borrowing Member Countries 2005-2012.
265 March
2015
Approach Paper for the Evaluation of Policy Based Operations
266 May 2015 Validation of Project Completion Report on Upgrading of Ecotourism Sites – Dominica
The Evaluation of the Caribbean Development Bank’s Intervention in Technical and Vocational Education and Training (1990-2012)
267 July 2015 Validation of Project Completion Report on The Belize Social Investment Fund I Project − Belize
268 Oct.2015 Approach Paper Country Strategy and Programme Evaluation, Haiti
The review and analysis of these documents is based on the UNEG Quality Checklist for Evaluation Reports (http://www.uneval.org/document/detail/607) as well as on ECG guidance (Big Book on Good Practice Standards).
Approach Papers
Three Approach Papers (APs) were made available to the panel (see Table [ref] above). An AP describes the rationale for the evaluation, the background to the topic evaluated, the evaluation framework (criteria and questions) and approach. It also describes the team and provides an initial planning. Being the first main deliverable of OIE’s evaluation process, APs are the starting point and therefore a major determining element in the roll-out of each evaluation. Therefore APs “have to get it right”.
The APs examined are clearly written, well-structured and of reasonable length.210 We were surprised to find, however, that they do not make explicit the objectives of the evaluated intervention(s), e.g., through a clear objective tree, or through an explicit theory of change, intervention logic or logframe. Whilst one of the APs contains, in an appendix, a results framework for the evaluation, the results framework for the intervention (PBO) itself is lacking.
Inception reports
Only one Inception Report was given to the Panel for review (SDF 6&7). This gives an in-depth description of the evaluated programme and provides a clear Theory of Change. It is good
210 Opportunities remain of course to be more concise and to move parts to appendices, e.g., detailed descriptions of the evaluation team or part of the description of the evaluated intervention.
375
practice that this is established after a pilot field mission, which helps to amend the initial AP on the basis of field observations and sharpen the evaluation questions if needed.
However, it is still considered to be good practice to have the Theory of Change elaborated in the initial design documents . This would facilitate OIE evaluations after project completion. Establishing the Theory of Change of any intervention would be included in the QaE form more explicitly, to be developed between the Quality unit referred to above, and OIE.
Evaluations and Assessments
Three evaluations and four assessment reports completed during the review period were considered. Assessments are similar to evaluations but have a narrower scope; they focus on a limited set of aspects or judgment criteria, mainly effectiveness, i.e. achievement of objectives. Evaluations generally base their judgment on the internationally recognised DAC criteria as well as aspects of the CDB and BMC’s management of the intervention.
In general, these reports are of reasonable quality. In the main, they explain the evaluated object211 and provide evaluation objectives. The findings are organised around the evaluation criteria or questions detailed in the scope and objectives section of the report. They are based on evidence derived from data collection and analysis methods as described in the methodology section. The reports tend to dwell on the limitations that the evaluation encountered, but without becoming defensive. In one case (PBL Assessment) the report starts with a summary of the reviews on the topic done by other MDBs. This was a pleasant surprise and indeed a good practice that could well be adopted in future evaluations too.
However, the reports also show several significant weaknesses:
- Reports do not always provide clear (reconstructed) intervention logics or theories of change for the intervention(s) evaluated.212 Evaluation criteria and questions are defined at a fairly general level. They are translated into more precise “research questions” (in an “Evaluation Design Matrix”, for each project for each criterion). However, it is unclear how these questions relate to the intervention logic (as this is not made explicit). This may be done in inception reports (of which, as noted above, only one was available for review), but should be done also in the final reports.
- The reports do not describe the link from the evaluation questions to the answers, how the evaluation judgments are made and how these ultimately transform into ratings for each criterion and each project. In other words, the explanation provided in the evaluation frameworks is inadequate. The “evaluation design matrix” currently used does not provide sufficient insight into how ultimately an intervention’s performance is judged.213 Links between findings, conclusions and recommendations could be improved by making this more explicit. In other words, reports should include the story on how the evaluand is credibly linked to any observed outcomes and impacts, and should be clear on how causal claims are made.
- With the exception of the PBL Assessment, reports are lengthy and detailed. One reason for this is an over-emphasis on ratings. Their detailed discussion, project by project, criterion by criterion, occupies a very prominent position in the evaluation reports’ main body of text. Although ratings are traditionally an important element in evaluations of MDBs, too strong
211 Sometimes in great length: for instance with the SDF 6&7 multicycle evaluation report it is only at page 30 that we find the beginning of the report on findings…212 Again with the SDF 6&7 evaluation, it is said to be guided by a “Logic Model” which is not explained.213 Marlène: I moreover have the idea that the methodology (often described as “visits”) is based on interviews and little hard evidence. Any view on this?.JM: My “interview-based evaluations”!!
376
an emphasis can be tedious and may distract the reader from the real lessons to be drawn. The detailed discussion of ratings, and their evidence base, would be better placed in an Appendix, with a brief summary in the main report. This would help give the lessons and recommendations a more prominent position than is now the case. This would also help make the evaluation reports not only shorter but also more interesting to read; this could help add value to evaluation’s image within the organisation.
- The reviewers feel that the OIE evaluations tend to over-emphasise objective-based evaluation214 and the DAC criteria to the exclusions of considering other evaluation approaches such as Developmental Evaluation (Patton, 2010215); evaluation should be case specific and answer the actual information needs of managers and other decisions makers rather than always concentrating on final performance.
- Related to the previous point (and again with the exception of the PBL Assessment) executive summaries (approximately 8 pages) are too long. For the evaluation report to increase potential impact, they would need to be reduced to 2 to 3 pages and be more focused; again this could be done by dwelling less on the individual ratings of projects and more on key findings, lessons and conclusions. More generally, reports could be better adapted to the needs of the different audiences. Although not strictly limited to evaluations, The Health Evidence Network Reports216 are a model that could be adapted for evaluation reporting purposes; they are specifically geared towards addressing policy and decision-making.
- The “Recommendations to BMCs” are an interesting feature of the reports, (although we are unsure to what degree such recommendations could be effectively followed up by OIE or the Bank, but certainly could taken up with BMC Board members.
- Reports (e.g. the evaluation report on Technical Assistance) focus much on technical problems that were encountered during the evaluation. Although these are important issues, again to improve the report’s flow and “readability” this section would be better placed in the Appendix. What counts is the story of the intervention, not the story of the evaluation (see “Limitations” section in the TA report for instance)
OIE Validations of Project and Country Strategy Programme Completion Reports (referred to globally as PCRs hereafter)
As said above, the OIE has the mandate to validate the Project and Economic departments PCRs and CSPCRs. However, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
The validations tend to repeat the different items reported in the PCRs and then provide extensive comment on each. The PCVRs go into great depth and detail, which makes the documents rich and complete. This is their strength – but also their weakness. The depth and
214 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.215 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press216 See the reports available at the WHO’s Health Evidence Netowkr at http://www.euro.who.int/en/data-and-evidence/evidence-informed-policy-making/health-evidence-network-hen
377
level of detail, as well as the repetitions from the original PCRs, makes PCVRs (overly) lengthy (20-40 pages) and difficult to read. The OIE reported spending approximately 27.2% of its time on validating PCRs in 2015 compared with 44.4% on its core work, i.e. doing or managing the higher level evaluations. That is more than half of its evaluation work is being spent on the validation process. Finally, the PCVRs now seem to be, to a great extent, a standalone output of OIE. It is not always clear to us how they are being used as the “building blocks” for the OIE’s independent evaluations. Making this clearer in the independent evaluations would help show the link and therefore the value of the time being spent on the self-evaluation validations.
To conclude, the review finds that the OIE has taken steps to improve the perceived utility of evaluation in several ways. In the first instance, by planning its work to provide relevant and timely evidence geared towards helping the Board with its oversight and decision making tasks. The topics are selected through dialogue between the OIE and key CDB stakeholders and reflect priorities of the CDBs strategic plan. Secondly, by securing the interest and consequently the buy-in of the OAC and CDB senior management through engaging their input throughout the evaluation process. This is evidenced by the reported interest in the latest three studies, the Country strategy programme in Haiti, the evaluation of policy-based operations and the SDF 6& 7 multicycle assessment.
The OIE products are of an acceptable quality and could be even better if some of the shortcomings were addressed. However, the products themselves do not impair the utility of OIE’s work; this is undermined in several ways: (1) by the time delays in commenting on PCRs (OIE) and providing feedback to the independent evaluations (operations and management) (2) by the inefficient processes for agreeing topics and funding sources as well as providing OIE with management responses to its reports.
Putting Evaluation to Use: transparency, feedback and follow-upThere are several ways that evaluation can be, and is being used. As John Mayne has pointed out in his many publications on the issue,217 when we talk of evaluation use, we are mainly thinking about its Instrumental use—use made to directly improve programming and performance. But there is also conceptual use - use which often goes unnoticed or more precisely, unmeasured. This refers to the kind of use made to enhance knowledge about the type of intervention under study in a more general way. Or even Reflective use— this refers to using discussions or workshops to encourage and support reflection on the evaluation findings to see how they might contribute to future strategies.
In the case of the CDB there is some evidence to suggest that “use” is not only instrumental, but other types are also developing. For example, in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessons. (Reflective use)
Another important use, as recommended by the ECG, is that from time to time a synthesis of lessons is drawn from a number of evaluations and made available publically. In fact the Panel was impressed to hear that in the past, the evaluation unit had done this drawing on lessons from evaluations of the power sector. (Conceptual use) Although nothing has happened since, it is now on the “to do list” for 2016 (OIE’s 2016 Work Plan).
As for instrumental use, responsibility for using the knowledge generated through evaluation and for possibly drawing up an action plan of what should be done is up to CDB senior management and the relevant CDB department and division. Oversight on applying recommendations and picking up on the lessons drawn is the responsibility of the OAC.
Evidence on how evaluations have actually contributed to decisions or negotiations is lacking or confusing, Certainly the OIE is unaware of the extent to which its evaluations are put to use. On
217 See for example, his opening chapter to Enhancing Evaluation use: Insights from internal Evaluation Units, Läubli Loud, M. and Mayne, J. 2014, Sage Publications
378
the one hand, the OAC minutes sometimes indicate that lessons learned are integrated into the next phase. On the other hand, the reviewers were told that often in the past, the evaluation results were “too old” to be of use as the lessons had already been drawn and used way before the report was completed. Similarly, people’s gaps in memory on how well the evaluative information from previous studies may have been used may also account for the scarcity of evidence.
In response, the Panel questioned CDB staff and the OIE about a particular study, the Technical and Vocational Education and Training Assessment. The feedback was somewhat contradictory. On the one hand, the study was criticised as “confirming” news rather than bringing “new news”. However, on the other, we learned that In October 2015, the Board of Directors approved a proposal for the revision of CDB’s Education and Training Policy and Strategy. Work on this has already begun and an external consultant has been engaged to lead the process.
Although it is one of the OIE’s tasks to set up a database on results and lessons learned from evaluations, so far this has not been a priority. There is also currently no systematic tracking of lessons or recommendations arising from the evaluations, or on any progress in their uptake. (The Panel has already referred above to OAC’s lack of oversight in the use of evaluation.)
The OIE’s role in supporting CDB’s organisational learning is clearly specified in the Evaluation Policy, with many good suggestions for knowledge sharing activities such as “brown-bag lunches, workshops, pamphlets and short issues papers” (p. 19). So far, however, the OIE’s lead role on the knowledge sharing side appears to be quite limited. It has provided advisory input in Loan Committee discussions, and organises workshops together with the relative operations department for discussing the implications of evaluation studies. Ultimately, of course, the uptake of evaluation results and knowledge is in the hands of management. But the evaluation unit has an important role to play in terms of knowledge broker and knowledge manager. Both have tended to be underplayed in OIE’s work plan so far.
Transparency: The Communication Strategy
In recent times and with the approval of its new Disclosure Policy, the CDB has started to post its independent evaluation reports on its website. (There is nothing on the self-evaluations). The website also presents a good overview of the role and function of the OIE and evaluation within the CDB. This is a step in the right direction for sharing information. However, in our view, the CDB’s communication strategy is the weakest part of the evaluation system to date.
The Panel has already commended the OIE in its efforts to engage the CDB and the OAC in evaluation work. But reporting and communicating the lessons seem to be entirely targeted at the Board and the CDB. Moreover, the 2015 budget provides only US$2’000 for communication – nothing of which is intended for outreach.
Reviewers feel that actively engaging with the more indirect stakeholders, for example project implementers in the BMCs, NGOs or project beneficiaries is relatively weak218. There appears to be little reflection on drawing out significant messages for the broader group of stakeholders, or on how then to transmit them to the “right” people in the “right” way (knowledge brokerage).
To conclude, evidence on the uptake of evaluation is either confusing or sparse. It is unfortunate that so far no systematic record keeping system has been put into place to track lessons learned or the uptake of recommendations (or actions agreed from management responses). The OIE plays a weak role in brokering the knowledge generated through evaluations to the benefit of external partners and in managing such knowledge. Although the Evaluation Policy specifies the need for “distilling evaluation findings and lessons learned in appropriate formats for targeted audiences both within and outside the CDB” (p.19) such a targeted communication strategy has yet to be developed and budgeted.
218 A broader communication strategy is one of the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
379
Strengthening Evaluation Capacities and Networking From the onset in 2012, the OIE has stressed the importance of developing and strengthening evaluation capacities within the OIE, the CDB and, subject to available resources, in borrowing member countries. Building evaluation capacity in BMCs and the CDB is one of the OIE’s mandated tasks. It has been a priority that figures on the work plan from the beginning (Work Programme and Budget 2012-2104) The idea of developing an internship programme for graduates from the Caribbean region was one idea that was advanced to help build local evaluation resources. However, the capacity-building has primarily been focused on OIE and CDB staff to date. One of the OIE’s two objectives for 2015 therefore was to take up the challenge and “strengthen evaluation capacities and networking” to include reaching out to the BMCs.
Developing OIE staff capacities
The change from project level to strategic and thematic evaluations does require different evaluative skills and competencies. The MDB Evaluation Pyramid presented below in Figure 3 shows the different types of evaluation and changing resource needs as one ascends the pyramid. Implicit here also is the change in the type of expertise and competencies needed as evaluation aspires to the higher levels.
Consequently for 2015, the OIE set itself the objective of networking and developing working partnerships with regional and international evaluation entities and academic institutions. The rationale was twofold: (1) secure further support and guidance as well as (2) increase its outreach and coverage through joint work and international exposure. Another implicit aim was to benefit from partners’ contacts in the BMCs wherever possible so as to improve data collection and quality.
Figure 3: The MDB Evaluation Pyramid219
219 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
380
The OIE has therefore linked up with Carleton University in Canada and the University of the West Indies, Barbados campus. The OIE was also approached by the Development Bank of South Africa to exchange experiences about setting up an evaluation entity in a “small” development bank. However, its attempt to become a member of the Evaluation Cooperation Group was not successful for reasons beyond its control.
The OIE is to be commended in addressing the issue of staff competencies and professional development more generally. New developments in evaluation as well as new developments in the scope of OIE’s work may necessitate new competencies. For this reason, organisations such as the International Developmental Evaluation Association have recommended that the competencies of evaluators and evaluation managers should be periodically reviewed. Several publications now exist on competency requirements and suggestions for the periodic review of staff competencies.220
It is not within this remit to compare and contrast OIE’s competencies with those recommended by international and national agencies. However, what we can say is that the OIE demonstrates great forethought in taking this on board.
Capacity building within CDB
The OIE’s objective also consists of continuing to develop measures for improving the monitoring and self-evaluation side of CDB’s work. OIE’s strategy here is to use the windows of opportunity on offer through some of the training sessions that are being organised by CDB as part of its shift towards MfDR e.g. by Corporate Planning Services and Technical Assistance. For 2016 it is also planned to have the OIE present at the annual staff meeting and Learning Forum.
The OIE also organises some ad hoc training with operations, for example to help understand new tools e.g. for drawing out lessons from self-evaluation reports and, more generally, in helping staff appreciate how evaluation can add value to the organisation’s work. Measures include providing advisory services on demand, and providing training alongside the introduction of new or revised tools.
Capacity building in the BMCs
This is an ambitious task and would require additional investment; from the bi-annual work plans to be effective. A modest attempt has been made in 2015; from what we understand, the OIE has joined together with the Carleton University and the University of the West Indies, using their networks in some of the BMCs, to try to develop this aspect.
To conclude, we cannot comment on the quality or reaction to such training, but can commend the OIE for making capacity building one of its priority objectives. From both the Policy and the documents we reviewed, we note that capacity building was always seen to be an important aspect of OIE’s work, but hitherto has received little strategic focus. But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.
Adequacy of the OIE’s human and financial resources to support its work
OIE’s Human Resources;
The OIE is has a staff of 5; the head, 1 senior evaluation officer and two evaluation managers, plus one administrative assistant. Three of the five were recruited from within the CDB. The
220 E.g. IDEAS, (2012) Competencies for Development Evaluation Evaluators, Managers and Commissioners, the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice (2010) and the Swiss Evaluation Society’s Evaluation Managers Competencies Framework (2014)
381
limited capacity means that it is not feasible to cover all the types of evaluation activities outlined in the Evaluation Policy. Yet there is some indication from the Board that OIE should embark on impact evaluations at some future stage. An increasing demand for evaluation and for impact evaluations in particular, would run the risk of overstretching the OIE’s capacity to deliver credible and useful evaluations. Moreover, there are many other designated OIE activities that should be recognised as valuable work; the validations, building CDB and BMC evaluation capacity, providing supervision, advice, knowledge management and brokerage as well as managing evaluation contracts, The time needs of dealing with all of these may be underestimated in OIE’s budgets; all are important for assuring best value from evaluation. The Panel is concerned that a demand for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level evaluations may undermine the importance and time needs of other essential tasks.
Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses.
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.
382
Self- and independent evaluationSelf-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes. Both types of evaluation are important as they are at the very heart of the evaluation function; they are said to be the building blocks for the more strategic evaluations that the OIE is now undertaking.
The Evaluation Coordination Group recommends that the self-evaluations be carried out by the relevant operations department and in turn, reviewed and validated by the organisation’s independent evaluation office. The CDB’s Evaluation Policy therefore talks of “validating all self-evaluations” as being one of OIE’s essential oversight tasks.
Within CDB, the self-evaluations should provide management with performance assessments and thereby serve an accountability function to the CDB and Board. To support the process, the OIE provides operations with manuals and checklists for guidance. Once a self-evaluation report is to hand, it is given over to the OIE for the validation of its technical quality and credibility.221
However, in the CDB case, there are well-documented issues that have affected the quality and timeliness of the self-evaluations on the one hand, and therefore the quality of the foundation on which to build the independent evaluations. Paucity of documentation within CDB, paucity of data collected and available in the Borrowing Member Countries (BMCs), time delays in producing completion reports and in turn, having them validated by the OIE - all such issues were systematically raised during interviews and in some of the independent evaluation reports. There appears to be little incentive to complete self-evaluations in a timelier manner.
Generally speaking, many of the monitoring data problems appear to be due to a lack of management oversight. For example, with the introduction of results-based management, the logic frame and monitoring and data needs are systematically being built into intervention design. However, the BMCs are not delivering the data as contractually agreed at the outset. Incentives to support any significant change towards building a results-based culture seem to be weak and sanctions seem to be rarely enforced when the supply of data is lacking or lengthy delays to the projects occur. Although we can appreciate the complexities of trying to enforce monitoring compliance, this means that often, project deadlines have had to be extended, data gaps are not being satisfactorily dealt with and in turn, there has been a void in the quality and quantity of available evidence for the CDB’s self-assessment of project performance. For some time, this lack of oversight has been tolerated. Part of the problem is the low priority accorded to completing the self-evaluation reports by operations, coupled with the absence of any focal point within senior management to drive the process and deal with the problems.
No record is kept of how the self-evaluation results are actually used. They do not appear on the CDB website, but we were told that the findings are integrated into the following project designs. Hence we are somewhat unclear as to the utility of these reports at present. The situation is exacerbated by a rather confused image of evaluation: some operations staff consider OIE’s input (through validations or independent evaluations) to be sometimes over-critical, regulatory and adding little value; it is a threat rather than an opportunity for learning. Yet at the same time, evaluation is recognized as an integral part of result-based management.
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.
Some managers however seem to start changing the status quo. For example a revised and simplified template for producing project completion reports is being considered, and mid-term
221 According to the Evaluation Policy, OIE should validate all PCRs and CCRs but due to the backlog of reports and the delay in completing them (sometimes years later) since October 2015, the OIE has secured OAC agreement to validate a maximum of 6 per year, which are selected in consultation with the OAC.
383
project reviews are expected to be more stringent in looking at monitoring plans and practices and tying disbursements to performance. In some cases we also learned of incentives being introduced to encourage project managers to complete their reports in a timelier manner. But much remains to be done and, since the OIE is no longer responsible for monitoring and project evaluations, there is a void that needs to be filled. It is up to line managers to drive this work forward.
To conclude, it is fair to say that in view of a number of “frustrations” between the OIE and operations, which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring data, the added value that evaluation might offer to the operations area is ill recognized. Moreover, the link between self-evaluation as the building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
384
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”222
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
222 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
385
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”223 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 224 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
223 CDB (2011) Evaluation Policy (p.2)224
386
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
387
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
388
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
389
mendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
390
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
391
APPENDICES
Appendix I - The External Review Mandate – Terms of Reference and Approach Paper
Appendix II -Review Approach, Data collection and Analysis, and Limitations
Appendix III – Overview of OIE Evaluation Practice
Appendix IV - List of Persons Interviewed
Appendix V - List of Documents Reviewed
Appendix VI- List of Topics used to guide interviews with members of CDB Board of Directors
Appendix VII - List of Topics used to guide interviews with CDB staff
392
Appendix III – Overview of OIE Evaluation Practice (prepared by the OIE in response to Reviewer’s request)
Caribbean Development Bank, Office of Independent Evaluation - OIE
Category Response
Percentage of projects subject to project (self-) evaluation
100% - Project Completion Reports (PCR)
Percentage of projects subject to validation by OIE
Approximately 40-50%
About 15 projects exit portfolio annually. Evaluation Policy calls for all PCR to be validated. However, OIE resources insufficient. Validation process reviewed in 2014. Now OAC (Board committee) selects a sample of 6-8 PCR for validation each year.
Percentage/number of projects subject to in-depth review by OIE
None – unless specifically requested by OAC
Due to limited resources, focus of OIE evaluation work programme is on PCR validations and high-level evaluations – including country strategy and programme evaluations (CSPE).
Number of high-level evaluations conducted by OIE (e.g. sector, thematic, geographic)
1-2 per year since 2011
Plan is 2-4 per year from 2016. This would include CSPE (1st planned for Q1 2016: Haiti)
Number of project impact evaluations conducted by OIE
None
OIE includes “impact questions” in high-level evaluations.
Number of project impact evaluations conducted by Bank staff or other non-OIE staff
OIE is not aware of any impact evaluation conducted by the Bank.
However, OIE provides technical support to the Basic Needs Trust Fund (BNTF) in its design of an M&E framework that entails impact evaluations.
Budget In USD mn: 0.78 in 2015; 0.82 in 2016. This is equivalent to about 2.5% of total CDB Administrative Budget.
75% of the budget is for Staff salaries (4 Professionals, 1 Support staff), leaving around USD 190,000 (in 2015) for other expenses, including consultants e.g. for external evaluations. Additional funding is accessed via the Special Development Fund (SDF). This varies according to type and scope of the evaluation, e.g. the ongoing SDF 6/7 Evaluation is SDF funded at USD 255,000.
Budget determined by Board, not separate from administrative budget.
SDF funding for evaluations is considered separately and subject to Bank internal approval process. SDF funding cannot be used to cover OIE expenses such as staff time or travel. Country eligibility for SDF funding is also a consideration. OIE expressed concerns about this funding track in respect to predictability, independence and eligibility limitations.
Head of OIE reports to Board, with administrative link to the President
Terms of appointment for Head
393
5 year term, renewable once. Appointed by the President with the agreement of the Board.
Right of Return for Head Not eligible for other staff positions.
Consultants as proportions of OIE budget
2015: 19% (USD 145,000)
Plus SDF funding. SDF funded evaluations are outsourced.
Last external evaluation (or peer review) of OIE
No external evaluation, though a review of the function was done in 2011, leading to the Evaluation Policy.
OIE External Review completed in April, 2016
Departments or special programmes supporting impact evaluation
None
394
Appendix IV – List of Persons Interviewed
Name Function relative to OIE Type interview
Mrs. Colleen Wainwright Member CDB Board of Directors (UK)
Face to face
Mrs. Cherianne Clarke Alternate MemberCDB Board of Directors (UK)
Face to face
Mrs. Jean McCardle Member CDB Board of Directors (Canada)
Face to face
Dr. Louis Woodroofe MemberCDB Board of Directors (Barbados)
Mr. A: de Brigard Former Member CDB Board of Directors
Skype interview
Mr. H. Illi Fromer Member CDB Board ofDirectors
Telephone interview
Mrs. Claudia Reyes Nieto Member CDB Board of Directors
Telephone interview
Mr. Bu Yu alternate DirectorCDB Board of Directors
Face to face
Mr. Michael Schroll(Barbados)
Head OIE
series of interviews viaSkype and face-to-face
Mr. Mark Clayton OIE Senior Evaluation Officer Focus GroupMrs. Egene Baccus Latchman OIE Evaluation OfficerMr. Everton Clinton OIE Evaluation OfficerMrs. Valerie Pilgrim OIE Evaluation Officer
Dr. Justin Ram CDB Director Economics Department
Face to face
Mr. Ian Durant CDB Deputy Director Economics Dept Face to faceDr. Wm Warren Smith CDB President
Joint interviewFace to face
Mrs. Yvette Lemonias-Seale CDB Vice President Corporate Services & Bank Secretariat
Mr. Denis Bergevin CDB Deputy DirectorInternal Audit
Face to face
Mr. Edward Greene CDB Division Chief, Technical Cooperation Division
Face to face
Mrs. Monica La Bennett CDB Deputy Director Corporate Planning Face to faceMrs. Patricia McKenzie CDB Vice President Operations Face to faceMs. Deidre Clarendon CDB Division Chief
Social Sector DivisionFace to face
Mrs. Cheryl Dixon CDB Co-ordinator, Environmental Sustainability Unit
Focus group
Mrs. Denise Noel- Debique CDB Gender Equality Advisor Mrs. Tessa Williams-Robertson CDB Head Renewable EnergyMrs. Klao Bell-Lewis CDB Head Corporate Communications Face to faceMr. Daniel Best CDB Director
Projects DepartmentFace to face
Mr. Carlyle Assue CDB Director Face to face
395
Finance Department
396
Appendix VI - Interview Guide: Members of CDB Board of Directors
Below are a list of themes that I should like to raise with you based on your experience and knowledge of the CDB’s independent evaluation function (Office of Independent
Evaluation).
In each case, I should be grateful if you could illustrate your responses with examples or help this Review by, wherever possible, sending me (or telling me where I can find) any
documents that could support your responses.
This guide is being sent to you in advance to help prepare our meeting. However, our interview will be conducted more in the style of a conversation. The following sub-questions will be used to GUIDE the interview. Please feel encouraged to raise any
additional issues that you feel we should take into account
On the governance and Independence of CDB’s evaluation functionWhat mechanisms are there in place to support its independence?
How satisfactory are the current arrangements in your opinion?
How is the balance between independence and the need for interaction with line management dealt with by the system? For example, what mechanisms exist to ensure that the OIE is kept up to date with decisions, policy / programme changes, other contextual changes etc that could have an affect on OIE evaluation studies / evaluation planning?
On the OIE’s Evaluation PolicyThe CDB’s Evaluation Policy was established in 2011. To what degree do you feel it is adequate? Still relevant?
What suggestions do you have for any improvements?
In your opinion, how adequate is the current quality assurance system for over viewing the evaluation function?
On the quality and credibility of evaluation studiesTo what degree do you believe the reports are fair and impartial?
Do you consider them to be of good quality? Are they credible?
Are you adequately consulted/involved on evaluations of interest to you?
On the relevance and usefulness of evaluations How well does the OIE engage with you / your committee during the preparation, implementation and reporting of an evaluation study to assure that it will be useful to the CDB?
How are the priorities set for the independent evaluations? What criteria are used? Are you satisfied with the current procedure?
397
When OIE evaluation studies are outsourced to external consultants, what criteria are used to make this decision?
How are the priorities for the OIE’s 3.year rolling work plan agreed? In your opinion, is the current plan adequate in terms of coverage and diversity?
In your opinion, do the evaluations address important and pressing programs and issues?
To what extent do you feel that the OIE’s evaluations integrate the cross-cutting theme such as gender, energy efficiency/renewable energy, climate change? What improvements might be made and how?
On the dissemination and uptake of evaluation findings and recommendationsTo what extent do you feel that evaluation findings are communicated to the CDB and its stakeholders in a
a) useful, b) constructive andc) timely manner?
Are evaluation recommendations useful? Realistic?
What mechanisms are in place to assure that evaluation results are taken into account in decision making and planning? What improvements do you feel could be made?
How have you used the findings from any evaluations? Examples?
To what degree do you feel that evaluation contributes to institutional learning? And what about to institutional accountability? Any examples?
What mechanisms are in place to ensure that knowledge from evaluation is accessible toCDB staff and other relevant stakeholders? Are the current arrangements satisfactory?
How satisfied are you with current arrangements? What expectations do you have for the future?
On resourcesHow is the OIE resourced financially and is this satisfactory?
What about the OIE staff, are all the important areas of expertise represented in the team?
On this Review of the Office of Independent EvaluationWhat are your expectations? What are you particularly hoping to learn from it?
Thank you very much for your cooperation and input
398
Appendix VII : Interview Pro-Forma – CDB Staff membersThis presents a list of the topics raised during interviews. It was used to guide the open-ended
discussion – this means that the sequence and exact wording of the questions may not necessarily have followed in this order or been asked in exactly this way.
Changeover to an Independent Evaluation Office? Expectations? Advantages and disadvantages??
Satisfaction with working relations between operations and the OIE from your perspective?
Process of dealing with the PCRs and CCRs? Advantages and limitations?
Quality and credibility of the validation process?
How are the self-evaluation reports used?
Credibility and Quality of OIE’s evaluation reports
Communication of self and OIE independent evaluations? To whom, in what way? Possible improvements?
Independence of the Office of Independent Evaluation (OIEIndependence is absolutely central to the integrity and trustworthiness of evaluation. It is an agreed requirement within the development agencies and in the evaluation community as a whole. In examining the issue of independence and good practice, reviewers are guided by the Evaluation Cooperation Group’s recommendations on good practices, the CDB’s Evaluation Policy and by the 2011 consultancy review of independence relative to the CDB’s evaluation and oversight division225. The appraisal is based on a comparison of the ECG’s recommendations on independence226 and the current OIE status.
OIE and Independence: Recommendations from the OECD Evaluation Cooperation Group (ECG)
The ECG’s considers the issue of independence according to three specific areas: organisational, or structural independence, behavioural, or functional independence and protection from outside interference, or operational independence.
225 Osvaldo Feinstein & Patrick G. Grasso, Consultants, May 2011 Consultancy to Review the Independence of the Evaluation and Oversight Division of the Caribbean Development Bank226 ECG 2014 Evaluation Good Practice Standards, Template for Assessing the Independence of Evaluation Organizations, Annexe II.1
399
Organizational independence, ensures that the evaluation unit and staff are protected against any influence or control by senior or line management, and have unrestricted access to all documents and information sources needed for conducting their evaluations. Also, that the scope of evaluations selected can cover all relevant aspects of their institution.
Behavioural independence, generally refers to the evaluation unit’s autonomy in selecting and conducting setting its work programme and in producing quality reports which can be delivered without management interference.
Protection from outside interference refers to the extent to which the evaluation function is autonomous in setting its priorities, and conducting its studies and processes and in reaching its judgments, and in managing its human and budget resources without management interference.
Conflict of interest safeguards refers to protection against staff conflict of interests be they current, immediate, future or prior professional and personal relationships and considerations or financial interests for which there should be provision in the institution’s human resource policies.
The OIE’s Independence in Practice
Organisational / structural independenceOn the whole, the Panel acknowledges and commends the efforts being made by the CDB to assure OIE’s organisational independence. The CDB’s Evaluation Policy provides for the OIE’s organisational independence from line management and the interview data suggests that there is also wide acceptance and acknowledgement of why the OIE should have such independent status. Table 1 below provides our overall assessment of this aspect of OIE’s independence when compared with ECG recommendations. 227
Table 1: OIE organisational independence compared with ECG recommendations
Aspects Indicators CDB Evaluation Policy (EP) and Practice
The structure and role of evaluation unit
Whether the evaluation unit has a mandate statement that makes clear its scope of responsibility extends to all operations of the organization, and that its reporting line, staff, budget and functions are organizationally independent from the organization’s operational, policy, and strategy departments and related decision-making
Partially Complies The Policy is broad enough to cover the full range of MDB type of evaluations. However in practice this would not be possible without additional human and budget resources
The unit is accountable to, and reports evaluation results to, the head or deputy head of the organization or its governing Board
Whether there is a direct reporting relationship between the unit, and
a) the Management, and/or
b) Board or
c) relevant Board Committee, of the institution
Complies - OIE reports to the Board of Directors (BoD) through its Oversight Assurance Committee (OAC)
The unit is located organizationally outside the staff or line management function of
The unit’s position in the organization relative to the program, activity or entity being evaluated
Complies - The OIE is located outside, and is therefore independent of CDB line management
227 Based on ECG (2014) Template for Assessing the Independence of Evaluation Organizations, Evaluation Good
Practice Standards, Annexe II.1
400
Aspects Indicators CDB Evaluation Policy (EP) and Practice
the program, activity or entity being evaluated
The unit reports regularly to the larger organization’s audit committee or other oversight body
Reporting relationship and frequency of reporting to the oversight body
Complies - The OIE reports x 5 per year to the OAC . Board approval for an additional executive meeting between the Head of the OIE and the OAC at least once per year was given in October 2015
The unit is sufficiently removed from political pressures to be able to report findings without fear of repercussions
Extent to which the evaluation unit and its staff are not accountable to political authorities, and are insulated from participation in political activities
Complies
Unit staffers are protected by a personnel system in which compensation, training, tenure and advancement are based on merit
Extent to which a merit system covering compensation, training, tenure and advancement is in place and enforced
Partially Complies - with CDB human resource policy. However the skill needs of OIE staff ought to be regularly reviewed in light of its move towards higher-level evaluations. Appraisal of skill needs and hiring of relevant staff should be completely under the authority of the Head of Evaluation. This is not sufficiently clear in the Policy or other documents we reviewed.
Unit has access to all needed information and information sources
Extent to which the evaluation unit has access to the organization’s
a) staff, records, and project sites;
b) co-financiers and other partners, clients; and
c) programs, activities, or entities it funds or sponsors
Complies –The available evidence suggests that there is no reason to doubt such access. But systematic and easily accessible documentation is lacking in the CDB; it is one of its weak points.. Delays in getting hold of the relevant documents can have consequences on the timeliness of evaluation studies
However, independence should not mean isolation: There appears to be a detachment between the OIE and CDB that is of concern to the Panel; on the one hand, between the OIE and operations staff, and (2) on the other, in terms of the structural arrangements between the OIE and senior management.
21) In agreeing for the OIE to concentrate on strategic and thematic, in-depth evaluations, responsibility for project monitoring and evaluation were given over to operations. The division is clear and respected. However, it has its drawbacks. With the OIE no longer systematically involved at the front-end of project design, the monitoring data needs are likely to be poorly defined. Weak monitoring data will contribute to weaker evaluations. (More on this point under the heading self and independent evaluations.)
In the reviewers’ opinion, it is a common misunderstanding to assume that providing evaluator advice on monitoring and evaluation data will comprise evaluator independence. On the contrary, evaluation input into project design is essential to assure that the logic, indicators and data needs are addressed so that at some future point in time an evaluation of the achievements can be empirically grounded.
401
This is not to say that the OIE no longer has any influence at the front-end design stage; it has merely shifted the point of focus. The OIE is now systematically providing such input more generally to the corporate planning teams for the tools and systems they are developing to support the MfDR framework. The monitoring data for projects and their implementation should be improved once the Project Performance Evaluation System (PPES) and the Portfolio Performance Management System (PPMS) are updated and operational.
22) In the second place, the OIE has limited formal access to the Advisory Management Team (AMT) weekly meetings where the President and senior management gather to exchange up-to-date information on the dynamics of CDB policy and practice. The OIE is not regularly invited in any capacity to these meetings or given a copy of the agenda or minutes; the OIE is occasionally invited to attend in order to discuss an evaluation report or management feedback. For the OIE, this means that it is unlikely to pick up on the ‘when’ and ‘what’ of key decisional issues or provide input into the discussion based on evaluative information. Its observer status at Loans Committee meetings, or as a participant informer at the OAC and BoD meetings and discussions do not necessarily provide the same insight as to the dynamics of management actions and/or decisions. .
To respond to this situation, the President has agreed to meet regularly with the Head of the OIE in order to keep him up to date with CDB strategic thinking. This is a welcomed change.
OIE Independence and Behavioural Issues The Panel has concerns about some behavioural issues. For example, through both the interviews and documentary review, we learned of considerable delays in processing both the independent evaluation reports as well as OIE’s validation of the CDB’s self-evaluations. Delays are generally due to receiving feedback on the independent reports from first, the relevant operational department, then from the AMT, and then on providing the OIE with a management response that is initially drafted by operations staff before being reviewed by the AMT. (OIE reports cannot be submitted to the OAC without the relevant management response). This two-layer process for preparing submissions to the Board is inefficient and could potentially be a threat to evaluation’s independence in the future by delaying OIE’s timely reporting to the OAC.
OIE validations of the CDB self-evaluations are also submitted to the OAC, but it is in both sides’ interest to clear up any misunderstandings beforehand. Despite attempts to improve the timeframe for completing these validations, delays are more the norm than the exception. Table 2 below summarises our assessment of the behavioural aspects of independence.
Table 2: OIE and Behavioural Independence
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Ability and willingness to issue strong, high quality, and uncompromising reports
Extent to which the evaluation unit:
a) has issued high quality reports that invite public scrutiny (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk) of the lessons from the organization’s programs and activities;
b) proposes standards for performance that are in advance of those in current use by the organization; and
c) critiques the outcomes of the
Partially complies – paucity of data and documentation sometimes hinder the quality of reports. The OIE emphasizes the learning part of evaluation, and is cautious in its criticism recognising that management is going through a transitory stage and can still be overly defensive.
402
Aspects Indicators CDB Evaluation Policy (EP) and Practice
organization’s programs, activities and entities
Ability to report candidly
Extent to which the organization’s mandate provides that the evaluation unit transmits its reports to the Management/Board after review and comment by relevant corporate units but without management-imposed restrictions on their scope and comments
Partially complies - as sometimes reporting to the Board is compromised by delays in the review/comment process between the OIE and the CDB. Any delay with the production of a Management Response will also mean that submitting a report to the Board in a timely manner is impaired since the two have to be submitted together.
Transparency in the reporting of evaluation findings
Extent to which the organization’s disclosure rules permit the evaluation unit to report significant findings to concerned stakeholders, both internal and external (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk).
Who determines evaluation unit’s disclosure policy and procedures: Board, relevant committee, or management.
Partially complies - The OIE’s conforms to the CDB’s disclosure policy. However, the dissemination of evaluation findings appears to be currently restricted to website publication and reports to the Board. A more targeted communication strategy to include other key stakeholders, e.g. project implementers in the BMCs should be developed and put in place.
Self-selection of items for work program
Procedures for selection of work program items are chosen, through systematic or purposive means, by the evaluation organization; consultation on work program with Management and Board
Complies - The OIE also ensures that its work program is drawn up after consultation with both CDB Management and Board to seek their input on relevant topics and themes.
Protection of administrative budget, and other budget sources, for evaluation function
Line item of administrative budget for evaluation determined in accordance with a clear policy parameter, and preserved at an indicated level or proportion; access to additional sources of funding with only formal review of content of submissions
Partially complies - The administrative budget for supporting OIE work is protected. Access to additional sources of funding is possible if well argued and justified. But the approval process is complex and inefficient. (See Figure 1 below)
OIE and Protection from External influence or interference
Our overall assessment is provided in Table 3 below. The OIE’s independence in the design, conduct and content of its evaluations does not appear to be subjected to any external interference. But securing funding from any sources outside the OIE’s administrative budget, i.e. from the Social Development Fund, is an unduly complex and long process. As such we consider that the current funding process can affect the OIE’s choice with regard to the type of evaluations it can undertake. (See Figures 1 and 2 below)
Table 3: OIE and its Independence from External influence or interference
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Proper design and execution of an evaluation
Extent to which the evaluation unit is able to determine the design, scope, timing and conduct of evaluations without Management
Complies – however within limits of restricted human and financial resources available
403
Aspects Indicators CDB Evaluation Policy (EP) and Practice
interference
Evaluation study funding
Extent to which the evaluation unit is unimpeded by restrictions on funds or other resources that would adversely affect its ability to carry out its responsibilities
Partially Complies - OIE must work within the limits of the agreed administrative budget wherever possible. If additional resources are needed for studies it must seek alternative funds elsewhere. The budget limitations can have an affect on the type of evaluations undertaken and therefore its independence in terms of choice.
Judgments made by the evaluators
Extent to which the evaluator’s judgment as to the appropriate content of a report is not subject to overruling or influence by an external authority
Complies – the evidence available suggests that the Board and Management accept the evaluators’ independent interpretation and conclusions Management responses are agreed to be the accepted place to raise any difference of opinion.
Evaluation unit head hiring/firing, term of office, performance review and compensation
Mandate or equivalent document specifies procedures for the
a) hiring, firing,
b) term of office,
c) performance review, and d). compensation of the evaluation unit head that ensure independence from operational management
Complies – the Head of OIE is appointed by the CDB President in agreement with the OAC for a 5 year period which is renewable x 1. The Head could be removed from Office by the President or the Board but only with the agreement of both parties.
However the Head reports to the President for all administrative and personnel matters. Even though this was not recommended in the Osvaldo Feinstein & Patrick G. Grasso report on Independence in 2011, the BoD accepted CDB’s reasons for keeping this arrangement. (e.g.most OAC members are non residents and cannot oversee day-to-day work)
. Extent to which the evaluation unit has control over:
a) staff hiring,
b) promotion, pay increases, and
c) firing, within a merit system
Partially complies - All OIE staff members are treated in the same way as other CDB staff. The Head has limited control over the hiring, firing or promotion of OIE staff.
Continued staff employment
Extent to which the evaluator’s continued employment is based only on reasons related to job performance, competency or the need for evaluator services
Partially complies - Whilst the EP is clear about procedures for hiring, firing and promotion, all of which must conform with CDB human resource policy, there is nothing mentioned about any difference of opinion between the CDB and the Head of the OIE with regard to continued staff employment subject to changes in the level of technical or interpersonal competencies needed to meet new demands.
Avoidance of Financial, Personal or Professional conflicts of interest
This particular aspect refers to the organisation’s Human Resources Policy; there must be provisions in place to protect against actual or potential conflict of interest. The Panel requested via the OIE, to have evidence from human resources on any such provisions but did not receive an answer. It must be assumes that this aspect of independence, past or present, does indeed form part of normal CDB Human Resource Policies
To conclude: The Panel is impressed with the measures CDB has taken to assure the organisational independence of the OIE. Its independent status is accepted and respected by
404
senior and line management. The OIE’s budget is not independent from the overall CDB administrative budget; this affects its choice of evaluation types or approaches. Some of the behavioural issues affecting independence were also of concern, especially due to the delays in the exchange of documents, between the OIE and operations departments, which has a direct effect on timely reporting to the OAC. As for protection from outside interference, our concerns are largely to do with OIE’s independence over staffing issue; there are potential loopholes in current arrangements that could undermine OIE’s autonomy over its staff.
OIE’s Strategy, Work Practices and Work ProgrammeThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. Much of the period 2012 to 2015 has therefore been taken up with preparing OIE’s shift in focus from project-based evaluations to the high-level thematic and in-depth strategic studies. This has meant adopting a three-way approach; (1) for self-evaluations, reducing its time input to support the process and (2) for independent evaluations, taking stock of the gaps in coverage and expertise, and (3) networking to share experiences with centres of expertise and align OIE with international practices. In addition, amongst other duties, it has been supporting the development of MfDR tools and systems such as the Project Performance Assessment System by providing advice and input on programme logic and monitoring needs. The OIE plans to conduct 2-4 high-level studies per year from 2016. The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation pyramid228 are brought out in the remaining sections of this Review, not least given the limited resources available.
To conclude: The OIE has made a first step in proposing a strategy for establishing itself as an independent evaluation resource. But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities.
The Value / Usefulness of OIE’s Independent EvaluationsEvaluation is a powerful tool that can provide useful, evidence-based information to help inform and influence policy and practice. But useful evaluations depend not only on the evaluators’ skills, but on several other important factors as well; 1) on planning evaluations to be relevant to the priorities of the organisation’s work and for their results to be delivered in time to be useful; on the degree of 2) consultation and ultimately ownership by those who seek evaluative information; on the 3) tools used to support the evaluation process per se; and on the 4) credibility and quality of the evaluation products229.
228 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).229 These aspects reflect the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
405
1. Planning relevant and timely evaluationsThe OIE is now working on a 3 year rolling work plan that sets out the broad areas for enquiry. So far, there are no agreed criteria for making the selection of the specific topics for independent evaluation, although the priorities tend to reflect those of the CDB’s strategic plan. Nevertheless decision-making is rather arbitrary based on a process of dialogue between the OIE and the CDB and the OIE and the Board.
One of the OIE’s two objectives for 2015 therefore, was to define a work plan and agree priorities based on an approach that is “utilisation-focused”. This means that the studies are selected and planned to be relevant and useful to the organisation’s needs.
The OIE has achieved this objective with respect to its latest studies, which concerns the Social Development Fund (SDF) Multicycle 6&7 Evaluation, the Haiti Country Strategy evaluation and the evaluation of the CDB’s Policy Based Operations. Each of these three have been planned to deliver their results in time to provide the CDB Board of Directors with relevant information for negotiating the next round of funding. In spite of some delays due to a myriad of reasons, not least to the extra effort needed to secure essential data, the studies are expected to deliver on time.
The processes for agreeing OIE’s work plan and specific evaluations on the one hand, and, in securing alternative funding on the other, are shown in Figure 1 below. The Panel was surprised at learning how bureaucratic (the internal approval process), and inefficient (in view of the time it takes) the process seems to be. The concern here is that such a process could possibly pose a threat to assuring the Board of “timely studies.”
Figure 1: Selection of Evaluation Topics and Funding Source
Consultation with CDB Operations and OAC/Board for selection of
evaluation topic
Internal review of Approach Paper
Specific Evaluation Study Design and Budgeting
OIE Draft Terms of Reference / Approach
Paper
Finalise Approach Paper and submit to OAC/Board
Final Approach Paper
OAC ApprovalOAC minutes
Paper
3-year Work Programme and Budget (approved by Board)
Board approval necessary If above USD
150,000
Board notification only if USD 150,000 or
below
Board Paper
Annual OIE report and work plan
submission to OAC
406
2. Consultation and ownership“The credibility of evaluations depends to some degree on whether and how the organization’s approach
to evaluation fosters partnership and helps build ownership and capacity in developing countries.”
(ECG good practices)
The OIE engages with the OAC, CDB senior management and operations for agreeing its 3-year work plan and then for selecting the specific topics and themes. It also discusses the evaluation approach paper (design and implementation plan) with the CDB and OAC before completing the final version. However, preliminary and final drafts of the report are only submitted to the CDB line and senior managers for comment and factual errors. Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the results and their implications. Discussions with the OAC are more limited due to the overburdened agenda of OAC and Board meetings, as previously discussed.
In short, the OIE is to be commended for following the recommendations of professional good practices and standards on participative approaches; it has succeeded in having introduced a modus operandi that involves the key players in the selection of evaluation topics, the evaluation designs and their results. Figure 2 below provides an overview of the evaluation implementation and stakeholder engagement processes.
Figure 2: Evaluation Study Implementation and Feedback Loops
Detailed ToR or Final Approach Paper if sufficiently detailed.
Funding Track
Final Approach Paper/ToR Board
Approval
OIE – Selection of consultants (if any) contracting
OIE Admin Budget or …
… SDF
Prepare TA Paper (content similar to Approach Paper but different
format.
TA Paper
Approval – Internal Loans Committee
Arrangement AFully outsourced / external
consultants; oversight by OIE
Arrangement BConducted by OIE
staff
Arrangement CJointly: external
consultants and OIE
Terms of Reference
OIE – Selection of consultants (if any)
contracting
407
19.20.
Preparations:Detailed evaluation plan (incl tools,
timeline, etc.) and logistics
Production of Inception Report / Approach Paper
Prepares Inception Report /
Approach Paper
Prepare for disclosure and dissemination
Presentation/workshop:Interim findings and conclusions for immediate feedback and validation
Data Collection and Analysis
OIE
Summary and ppt for workshop presentation
and discussion with CDBSubmission of Draft Final
Report to OIE
Final OIE approved report to CDB Senior Management for Management Response
Board notification only if USD 150,000 or
below
Draft Final Report
Review loops – OIE and CDB (potentially also BMC)
Feedback to evaluation lead
Submission of Final Report to
OIE
Final Report
Final Report and Management Response submitted to
OAC/BoardFinal Report and
Mgt. Resp.
Management Response
OIE ApprovalFinal Report and Management Response considered by CDB
AMT
OAC/Board endorsed
408
Notes to Figure 2
19. The OIE informed the Panel that this is an abbreviated version as there are e.g. additional steps (secondary processes) when evaluations are procured (tendering or single source), when there are additional review loops and updates to OAC etc.
20. OAC may also decide to return the report to OIE, the Panel were informed, or demand from Management specific actions based on the report.
This process is engaging and appears to have secured senior management and OAC interest and buy-in as witnessed in the latest studies. But there is the downside too! The process takes much time and, in our view, is partly unnecessary. The Panel appreciates that staff from operations as well as the AMT may both want to confer on an appropriate management response, but this should not be the case for reviewing an independent report for factual errors. The two-phase approach seems somewhat inefficient and unnecessary in our opinion.
Contact between the OIE, the CDB and/or the OAC during the actual study implementation is most often restricted to the occasional progress report, particularly when studies run behind time. There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. The OIE does, however, arrange discussions for reflecting on emerging findings, but we are not sure of how systematic this feedback loop is.
More generally speaking, outside of an evaluation study, the OIE has limited dealings with operations. The OIE has an advisory role in providing them with help, particularly with providing training, guidelines and tools to support self-evaluations. We are nevertheless concerned about the seeming distance between these two and how this has affected the perceived value of evaluation. (For further on this point, please see the section below on “Self- and Independent Evaluations”)
But the Panel also wishes to stress that this is not the case for newly appointed senior managers. A much more open attitude to evaluation and appreciation of its potential value was evident; they expressed interest in drawing out important lessons on what works, how, for whom, and under what conditions. In one case, interest was followed up in practice; the OIE was recently invited by a senior manager to share evaluative knowledge and experience with his staff regarding policy based operations.
Certainly, we can say that overall, the key stakeholders within the CDB are adequately integrated into the evaluation process as to foster their buy-in and ownership. But more generally, we feel that the utility of independent evaluations can be improved by fostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that evaluation can offer can add value to understanding the strengths and weaknesses of such strategies. This however cannot be done overnight and takes a long time.
3. Tools to support the evaluation processSo far, during this transitional phase, the OIE has mainly focussed on improving the tools to support the operations areas’ self-evaluations. This has left the OIE with little time to produce the checklists or tools to support its own studies. There are plans to develop an OIE Manual to guide and support the independent evaluation process. Such plans should be encouraged, as these documents will form a very important part of training, particularly for newcomers to the OIE team.
In the meantime, the OIE and operations staff refers to the Performance Assessment System (PAS) Manuals for evaluation activities. The manuals are based on DAC criteria and ECG principles. Much emphasis is given to the rating system and how and what should be rated.
409
However we find them lengthy, unwieldy and overcomplicated. Moreover, such manuals should be used for reference, but cannot and should not replace first-hand training in how to plan, conduct and manage the evaluation process.
Quality Assessment (QA) and Quality at Entry (QaE)
There was a transition period between 2012 and 2014 to establish the OIE. Work on the PAS, QaE, PCRs, ARPP, which had started earlier, was therefore completed after OIE came into existence, but it effectively had no formal ‘home’ in operations. The Panel was told that there had been some discussions about creating a Quality Assurance unit within CDB (OPS) but the current status is unclear.
The QaE Guidance Questionnaire was developed before and completed by the OIE. It was used to assess the documents that came across to the OIE for comments at the Review Stage. The results were then sent to the Portfolio Manager/Project Coordinator indicating any gaps/issues that needed to be addressed or clarified. QaE Guidance Questionnaires were developed for all the Bank’s lending products, CSP and to assess the quality of supervision.
After the QaE was launched bank wide, several operations officers saw the merit in using the QaE Guidance Questionnaire in the field and adopted it as a tool for their use during the appraisal mission in order to cross check and test their data collection and analysis.
OIE’s use of the QaE was discontinued in 2014 due to limited resources and a stronger focus on evaluations. It still sometimes comments on specific appraisals, but very selectively.
Both QaE and QaS (quality at supervision) are also addressed in the PAS Manuals. In addition the QaE and PAS have been incorporated in Volume 2 of the Operations Manual OPPM.
The Review Panel assessed the QaE forms. They are relatively standard, adapted to the specificities of the CDB. They contribute to judging a project’s expected quality in a relatively objective way. As such, they are are helpful, as a benchmark, in the ex-post assessment of projects.
The Panel considers that the lack of an established Quality Unit in the CDB (and independent from OIE) is a weakness that should be addressed in the near future.
4. Credibility and Quality of Evaluation ProductsAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An independent evaluation is processed as follows; the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
Since 2012, the OIE has produced a range of studies and approach papers. This review is based on those listed below as provided by the OIE, and cover the period from May 2012 to December 2015. It includes 3 evaluations (in blue), 4 Assessment studies (in brown) 14 validations of self-evaluations (in green) and 3 Approach Papers (in purple) for upcoming evaluations. These are listed below in Table 4.
410
Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
Board Meeting
Date Type / Topic
251 May 2012 Ex-Post Evaluation Report on Road Improvement and Maintenance Project, Nevis -St. Kitts and Nevis.
Validation of Project Completion Report on Sites and Services – Grenada. Assessment of Effectiveness of Implementation of Poverty Reduction
Strategy 2004-09.253 Oct. 2012 Assessment of Extent and Effectiveness of Mainstreaming Environment,
Climate Change, Disaster Management at CDB.254 Dec. 2012 Assessment of the Implementation Effectiveness of the Gender Equality
Policy and Operational Strategy of the Caribbean Development Bank. Validation of Project Completion Report on Enhancement of Technical and
Vocational Education and Training – Belize. Validation of Project Completion Report on Fourth Road (Northern Coastal
Highway Improvement Section 1 of Segment II) Project – Jamaica. Assessment of the Effectiveness of the Policy-based Lending Instrument.
256 May 2013 Validation of Project Completion Report on Expansion of Grantley Adams International Airport – Barbados.
Validation of Project Completion Report on Fifth Water Supply Project – Saint Lucia.
261 May 2014 Validation of Project Completion Report on Immediate Response Loan, Tropical Storm Gustav, Jamaica.
Validation of Project Completion Report on Social Investment Fund, Jamaica.
Validation of Project Completion Report on Disaster Mitigation and Restoration – Rockfall and Landslip, Grenada.
263 Oct. 2014 Validation of Project Completion Report on Basic Education Project – Antigua and Barbuda
263 Oct. 2014 Approach Paper for SDF 6 & 7 Multicycle Evaluation
264 Dec. 2014 Validation of Project Completion Report on Policy-Based Loan – Anguilla Validation of Project Completion Report on Immediate Response Loan -
Tropical Storm Arthur – Belize. Evaluation of Technical Assistance Interventions of the Caribbean
Development Bank Related To Tax Administration and Tax Reform in The Borrowing Member Countries 2005-2012.
265 March
2015
Approach Paper for the Evaluation of Policy Based Operations
266 May 2015 Validation of Project Completion Report on Upgrading of Ecotourism Sites – Dominica
The Evaluation of the Caribbean Development Bank’s Intervention in Technical and Vocational Education and Training (1990-2012)
267 July 2015 Validation of Project Completion Report on The Belize Social Investment Fund I Project − Belize
268 Oct.2015 Approach Paper Country Strategy and Programme Evaluation, Haiti
The review and analysis of these documents is based on the UNEG Quality Checklist for Evaluation Reports (http://www.uneval.org/document/detail/607) as well as on ECG guidance (Big Book on Good Practice Standards).
411
Approach Papers
Three Approach Papers (APs) were made available to the panel (see Table [ref] above). An AP describes the rationale for the evaluation, the background to the topic evaluated, the evaluation framework (criteria and questions) and approach. It also describes the team and provides an initial planning. Being the first main deliverable of OIE’s evaluation process, APs are the starting point and therefore a major determining element in the roll-out of each evaluation. Therefore APs “have to get it right”.
The APs examined are clearly written, well-structured and of reasonable length.230 We were surprised to find, however, that they do not make explicit the objectives of the evaluated intervention(s), e.g., through a clear objective tree, or through an explicit theory of change, intervention logic or logframe. Whilst one of the APs contains, in an appendix, a results framework for the evaluation, the results framework for the intervention (PBO) itself is lacking.
Inception reports
Only one Inception Report was given to the Panel for review (SDF 6&7). This gives an in-depth description of the evaluated programme and provides a clear Theory of Change. It is good practice that this is established after a pilot field mission, which helps to amend the initial AP on the basis of field observations and sharpen the evaluation questions if needed.
However, it is still considered to be good practice to have the Theory of Change elaborated in the initial design documents . This would facilitate OIE evaluations after project completion. Establishing the Theory of Change of any intervention would be included in the QaE form more explicitly, to be developed between the Quality unit referred to above, and OIE.
Evaluations and Assessments
Three evaluations and four assessment reports completed during the review period were considered. Assessments are similar to evaluations but have a narrower scope; they focus on a limited set of aspects or judgment criteria, mainly effectiveness, i.e. achievement of objectives. Evaluations generally base their judgment on the internationally recognised DAC criteria as well as aspects of the CDB and BMC’s management of the intervention.
In general, these reports are of reasonable quality. In the main, they explain the evaluated object231 and provide evaluation objectives. The findings are organised around the evaluation criteria or questions detailed in the scope and objectives section of the report. They are based on evidence derived from data collection and analysis methods as described in the methodology section. The reports tend to dwell on the limitations that the evaluation encountered, but without becoming defensive. In one case (PBL Assessment) the report starts with a summary of the reviews on the topic done by other MDBs. This was a pleasant surprise and indeed a good practice that could well be adopted in future evaluations too.
However, the reports also show several significant weaknesses:
- Reports do not always provide clear (reconstructed) intervention logics or theories of change for the intervention(s) evaluated.232 Evaluation criteria and questions are defined at a fairly general level. They are translated into more precise “research questions” (in an “Evaluation Design Matrix”, for each project for each criterion). However, it is unclear how these questions relate to the intervention logic (as this is not made explicit). This may be
230 Opportunities remain of course to be more concise and to move parts to appendices, e.g., detailed descriptions of the evaluation team or part of the description of the evaluated intervention.231 Sometimes in great length: for instance with the SDF 6&7 multicycle evaluation report it is only at page 30 that we find the beginning of the report on findings…232 Again with the SDF 6&7 evaluation, it is said to be guided by a “Logic Model” which is not explained.
412
done in inception reports (of which, as noted above, only one was available for review), but should be done also in the final reports.
- The reports do not describe the link from the evaluation questions to the answers, how the evaluation judgments are made and how these ultimately transform into ratings for each criterion and each project. In other words, the explanation provided in the evaluation frameworks is inadequate. The “evaluation design matrix” currently used does not provide sufficient insight into how ultimately an intervention’s performance is judged.233 Links between findings, conclusions and recommendations could be improved by making this more explicit. In other words, reports should include the story on how the evaluand is credibly linked to any observed outcomes and impacts, and should be clear on how causal claims are made.
- With the exception of the PBL Assessment, reports are lengthy and detailed. One reason for this is an over-emphasis on ratings. Their detailed discussion, project by project, criterion by criterion, occupies a very prominent position in the evaluation reports’ main body of text. Although ratings are traditionally an important element in evaluations of MDBs, too strong an emphasis can be tedious and may distract the reader from the real lessons to be drawn. The detailed discussion of ratings, and their evidence base, would be better placed in an Appendix, with a brief summary in the main report. This would help give the lessons and recommendations a more prominent position than is now the case. This would also help make the evaluation reports not only shorter but also more interesting to read; this could help add value to evaluation’s image within the organisation.
- The reviewers feel that the OIE evaluations tend to over-emphasise objective-based evaluation234 and the DAC criteria to the exclusions of considering other evaluation approaches such as Developmental Evaluation (Patton, 2010235); evaluation should be case specific and answer the actual information needs of managers and other decisions makers rather than always concentrating on final performance.
- Related to the previous point (and again with the exception of the PBL Assessment) executive summaries (approximately 8 pages) are too long. For the evaluation report to increase potential impact, they would need to be reduced to 2 to 3 pages and be more focused; again this could be done by dwelling less on the individual ratings of projects and more on key findings, lessons and conclusions. More generally, reports could be better adapted to the needs of the different audiences. Although not strictly limited to evaluations, The Health Evidence Network Reports236 are a model that could be adapted for evaluation reporting purposes; they are specifically geared towards addressing policy and decision-making.
- The “Recommendations to BMCs” are an interesting feature of the reports, (although we are unsure to what degree such recommendations could be effectively followed up by OIE or the Bank, but certainly could taken up with BMC Board members.
233 Marlène: I moreover have the idea that the methodology (often described as “visits”) is based on interviews and little hard evidence. Any view on this?.JM: My “interview-based evaluations”!!234 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.235 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press236 See the reports available at the WHO’s Health Evidence Netowkr at http://www.euro.who.int/en/data-and-evidence/evidence-informed-policy-making/health-evidence-network-hen
413
- Reports (e.g. the evaluation report on Technical Assistance) focus much on technical problems that were encountered during the evaluation. Although these are important issues, again to improve the report’s flow and “readability” this section would be better placed in the Appendix. What counts is the story of the intervention, not the story of the evaluation (see “Limitations” section in the TA report for instance)
OIE Validations of Project and Country Strategy Programme Completion Reports (referred to globally as PCRs hereafter)
As said above, the OIE has the mandate to validate the Project and Economic departments PCRs and CSPCRs. However, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
The validations tend to repeat the different items reported in the PCRs and then provide extensive comment on each. The PCVRs go into great depth and detail, which makes the documents rich and complete. This is their strength – but also their weakness. The depth and level of detail, as well as the repetitions from the original PCRs, makes PCVRs (overly) lengthy (20-40 pages) and difficult to read. The OIE reported spending approximately 27.2% of its time on validating PCRs in 2015 compared with 44.4% on its core work, i.e. doing or managing the higher level evaluations. That is more than half of its evaluation work is being spent on the validation process. Finally, the PCVRs now seem to be, to a great extent, a standalone output of OIE. It is not always clear to us how they are being used as the “building blocks” for the OIE’s independent evaluations. Making this clearer in the independent evaluations would help show the link and therefore the value of the time being spent on the self-evaluation validations.
To conclude, the review finds that the OIE has taken steps to improve the perceived utility of evaluation in several ways. In the first instance, by planning its work to provide relevant and timely evidence geared towards helping the Board with its oversight and decision making tasks. The topics are selected through dialogue between the OIE and key CDB stakeholders and reflect priorities of the CDBs strategic plan. Secondly, by securing the interest and consequently the buy-in of the OAC and CDB senior management through engaging their input throughout the evaluation process. This is evidenced by the reported interest in the latest three studies, the Country strategy programme in Haiti, the evaluation of policy-based operations and the SDF 6& 7 multicycle assessment.
The OIE products are of an acceptable quality and could be even better if some of the shortcomings were addressed. However, the products themselves do not impair the utility of OIE’s work; this is undermined in several ways: (1) by the time delays in commenting on PCRs (OIE) and providing feedback to the independent evaluations (operations and management) (2) by the inefficient processes for agreeing topics and funding sources as well as providing OIE with management responses to its reports.
Putting Evaluation to Use: transparency, feedback and follow-upThere are several ways that evaluation can be, and is being used. As John Mayne has pointed out in his many publications on the issue,237 when we talk of evaluation use, we are mainly thinking about its Instrumental use—use made to directly improve programming and performance. But there is also conceptual use - use which often goes unnoticed or more precisely, unmeasured. This refers to the kind of use made to enhance knowledge about the type of intervention under study in a more general way. Or even Reflective use— this refers to using discussions or
237 See for example, his opening chapter to Enhancing Evaluation use: Insights from internal Evaluation Units, Läubli Loud, M. and Mayne, J. 2014, Sage Publications
414
workshops to encourage and support reflection on the evaluation findings to see how they might contribute to future strategies.
In the case of the CDB there is some evidence to suggest that “use” is not only instrumental, but other types are also developing. For example, in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessons. (Reflective use)
Another important use, as recommended by the ECG, is that from time to time a synthesis of lessons is drawn from a number of evaluations and made available publically. In fact the Panel was impressed to hear that in the past, the evaluation unit had done this drawing on lessons from evaluations of the power sector. (Conceptual use) Although nothing has happened since, it is now on the “to do list” for 2016 (OIE’s 2016 Work Plan).
As for instrumental use, responsibility for using the knowledge generated through evaluation and for possibly drawing up an action plan of what should be done is up to CDB senior management and the relevant CDB department and division. Oversight on applying recommendations and picking up on the lessons drawn is the responsibility of the OAC.
Evidence on how evaluations have actually contributed to decisions or negotiations is lacking or confusing, Certainly the OIE is unaware of the extent to which its evaluations are put to use. On the one hand, the OAC minutes sometimes indicate that lessons learned are integrated into the next phase. On the other hand, the reviewers were told that often in the past, the evaluation results were “too old” to be of use as the lessons had already been drawn and used way before the report was completed. Similarly, people’s gaps in memory on how well the evaluative information from previous studies may have been used may also account for the scarcity of evidence.
In response, the Panel questioned CDB staff and the OIE about a particular study, the Technical and Vocational Education and Training Assessment. The feedback was somewhat contradictory. On the one hand, the study was criticised as “confirming” news rather than bringing “new news”. However, on the other, we learned that In October 2015, the Board of Directors approved a proposal for the revision of CDB’s Education and Training Policy and Strategy. Work on this has already begun and an external consultant has been engaged to lead the process.
Although it is one of the OIE’s tasks to set up a database on results and lessons learned from evaluations, so far this has not been a priority. There is also currently no systematic tracking of lessons or recommendations arising from the evaluations, or on any progress in their uptake. (The Panel has already referred above to OAC’s lack of oversight in the use of evaluation.)
The OIE’s role in supporting CDB’s organisational learning is clearly specified in the Evaluation Policy, with many good suggestions for knowledge sharing activities such as “brown-bag lunches, workshops, pamphlets and short issues papers” (p. 19). So far, however, the OIE’s lead role on the knowledge sharing side appears to be quite limited. It has provided advisory input in Loan Committee discussions, and organises workshops together with the relative operations department for discussing the implications of evaluation studies. Ultimately, of course, the uptake of evaluation results and knowledge is in the hands of management. But the evaluation unit has an important role to play in terms of knowledge broker and knowledge manager. Both have tended to be underplayed in OIE’s work plan so far.
Transparency: The Communication Strategy
In recent times and with the approval of its new Disclosure Policy, the CDB has started to post its independent evaluation reports on its website. (There is nothing on the self-evaluations). The website also presents a good overview of the role and function of the OIE and evaluation within the CDB. This is a step in the right direction for sharing information. However, in our view, the CDB’s communication strategy is the weakest part of the evaluation system to date.
415
The Panel has already commended the OIE in its efforts to engage the CDB and the OAC in evaluation work. But reporting and communicating the lessons seem to be entirely targeted at the Board and the CDB. Moreover, the 2015 budget provides only US$2’000 for communication – nothing of which is intended for outreach.
Reviewers feel that actively engaging with the more indirect stakeholders, for example project implementers in the BMCs, NGOs or project beneficiaries is relatively weak238. There appears to be little reflection on drawing out significant messages for the broader group of stakeholders, or on how then to transmit them to the “right” people in the “right” way (knowledge brokerage).
To conclude, evidence on the uptake of evaluation is either confusing or sparse. It is unfortunate that so far no systematic record keeping system has been put into place to track lessons learned or the uptake of recommendations (or actions agreed from management responses). The OIE plays a weak role in brokering the knowledge generated through evaluations to the benefit of external partners and in managing such knowledge. Although the Evaluation Policy specifies the need for “distilling evaluation findings and lessons learned in appropriate formats for targeted audiences both within and outside the CDB” (p.19) such a targeted communication strategy has yet to be developed and budgeted.
Strengthening Evaluation Capacities and Networking From the onset in 2012, the OIE has stressed the importance of developing and strengthening evaluation capacities within the OIE, the CDB and, subject to available resources, in borrowing member countries. Building evaluation capacity in BMCs and the CDB is one of the OIE’s mandated tasks. It has been a priority that figures on the work plan from the beginning (Work Programme and Budget 2012-2104) The idea of developing an internship programme for graduates from the Caribbean region was one idea that was advanced to help build local evaluation resources. However, the capacity-building has primarily been focused on OIE and CDB staff to date. One of the OIE’s two objectives for 2015 therefore was to take up the challenge and “strengthen evaluation capacities and networking” to include reaching out to the BMCs.
Developing OIE staff capacities
The change from project level to strategic and thematic evaluations does require different evaluative skills and competencies. The MDB Evaluation Pyramid presented below in Figure 3 shows the different types of evaluation and changing resource needs as one ascends the pyramid. Implicit here also is the change in the type of expertise and competencies needed as evaluation aspires to the higher levels.
Consequently for 2015, the OIE set itself the objective of networking and developing working partnerships with regional and international evaluation entities and academic institutions. The rationale was twofold: (1) secure further support and guidance as well as (2) increase its outreach and coverage through joint work and international exposure. Another implicit aim was to benefit from partners’ contacts in the BMCs wherever possible so as to improve data collection and quality.
Figure 3: The MDB Evaluation Pyramid239
238 A broader communication strategy is one of the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.239 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
416
The OIE has therefore linked up with Carleton University in Canada and the University of the West Indies, Barbados campus. The OIE was also approached by the Development Bank of South Africa to exchange experiences about setting up an evaluation entity in a “small” development bank. However, its attempt to become a member of the Evaluation Cooperation Group was not successful for reasons beyond its control.
The OIE is to be commended in addressing the issue of staff competencies and professional development more generally. New developments in evaluation as well as new developments in the scope of OIE’s work may necessitate new competencies. For this reason, organisations such as the International Developmental Evaluation Association have recommended that the competencies of evaluators and evaluation managers should be periodically reviewed. Several publications now exist on competency requirements and suggestions for the periodic review of staff competencies.240
It is not within this remit to compare and contrast OIE’s competencies with those recommended by international and national agencies. However, what we can say is that the OIE demonstrates great forethought in taking this on board.
Capacity building within CDB
The OIE’s objective also consists of continuing to develop measures for improving the monitoring and self-evaluation side of CDB’s work. OIE’s strategy here is to use the windows of opportunity on offer through some of the training sessions that are being organised by CDB as part of its shift towards MfDR e.g. by Corporate Planning Services and Technical Assistance. For 2016 it is also planned to have the OIE present at the annual staff meeting and Learning Forum.
The OIE also organises some ad hoc training with operations, for example to help understand new tools e.g. for drawing out lessons from self-evaluation reports and, more generally, in
240 E.g. IDEAS, (2012) Competencies for Development Evaluation Evaluators, Managers and Commissioners, the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice (2010) and the Swiss Evaluation Society’s Evaluation Managers Competencies Framework (2014)
417
helping staff appreciate how evaluation can add value to the organisation’s work. Measures include providing advisory services on demand, and providing training alongside the introduction of new or revised tools.
Capacity building in the BMCs
This is an ambitious task and would require additional investment; from the bi-annual work plans to be effective. A modest attempt has been made in 2015; from what we understand, the OIE has joined together with the Carleton University and the University of the West Indies, using their networks in some of the BMCs, to try to develop this aspect.
To conclude, we cannot comment on the quality or reaction to such training, but can commend the OIE for making capacity building one of its priority objectives. From both the Policy and the documents we reviewed, we note that capacity building was always seen to be an important aspect of OIE’s work, but hitherto has received little strategic focus. But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.
Adequacy of the OIE’s human and financial resources to support its work
OIE’s Human Resources;
The OIE is has a staff of 5; the head, 1 senior evaluation officer and two evaluation managers, plus one administrative assistant. Three of the five were recruited from within the CDB. The limited capacity means that it is not feasible to cover all the types of evaluation activities outlined in the Evaluation Policy. Yet there is some indication from the Board that OIE should embark on impact evaluations at some future stage. An increasing demand for evaluation and for impact evaluations in particular, would run the risk of overstretching the OIE’s capacity to deliver credible and useful evaluations. Moreover, there are many other designated OIE activities that should be recognised as valuable work; the validations, building CDB and BMC evaluation capacity, providing supervision, advice, knowledge management and brokerage as well as managing evaluation contracts, The time needs of dealing with all of these may be underestimated in OIE’s budgets; all are important for assuring best value from evaluation. The Panel is concerned that a demand for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level evaluations may undermine the importance and time needs of other essential tasks.
Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses.
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the
418
backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.
Self- and independent evaluationSelf-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes. Both types of evaluation are important as they are at the very heart of the evaluation function; they are said to be the building blocks for the more strategic evaluations that the OIE is now undertaking.
The Evaluation Coordination Group recommends that the self-evaluations be carried out by the relevant operations department and in turn, reviewed and validated by the organisation’s independent evaluation office. The CDB’s Evaluation Policy therefore talks of “validating all self-evaluations” as being one of OIE’s essential oversight tasks.
Within CDB, the self-evaluations should provide management with performance assessments and thereby serve an accountability function to the CDB and Board. To support the process, the OIE provides operations with manuals and checklists for guidance. Once a self-evaluation report is to hand, it is given over to the OIE for the validation of its technical quality and credibility.241
However, in the CDB case, there are well-documented issues that have affected the quality and timeliness of the self-evaluations on the one hand, and therefore the quality of the foundation on which to build the independent evaluations. Paucity of documentation within CDB, paucity of data collected and available in the Borrowing Member Countries (BMCs), time delays in producing completion reports and in turn, having them validated by the OIE - all such issues were systematically raised during interviews and in some of the independent evaluation reports. There appears to be little incentive to complete self-evaluations in a timelier manner.
Generally speaking, many of the monitoring data problems appear to be due to a lack of management oversight. For example, with the introduction of results-based management, the logic frame and monitoring and data needs are systematically being built into intervention design. However, the BMCs are not delivering the data as contractually agreed at the outset.
241 According to the Evaluation Policy, OIE should validate all PCRs and CCRs but due to the backlog of reports and the delay in completing them (sometimes years later) since October 2015, the OIE has secured OAC agreement to validate a maximum of 6 per year, which are selected in consultation with the OAC.
419
Incentives to support any significant change towards building a results-based culture seem to be weak and sanctions seem to be rarely enforced when the supply of data is lacking or lengthy delays to the projects occur. Although we can appreciate the complexities of trying to enforce monitoring compliance, this means that often, project deadlines have had to be extended, data gaps are not being satisfactorily dealt with and in turn, there has been a void in the quality and quantity of available evidence for the CDB’s self-assessment of project performance. For some time, this lack of oversight has been tolerated. Part of the problem is the low priority accorded to completing the self-evaluation reports by operations, coupled with the absence of any focal point within senior management to drive the process and deal with the problems.
No record is kept of how the self-evaluation results are actually used. They do not appear on the CDB website, but we were told that the findings are integrated into the following project designs. Hence we are somewhat unclear as to the utility of these reports at present. The situation is exacerbated by a rather confused image of evaluation: some operations staff consider OIE’s input (through validations or independent evaluations) to be sometimes over-critical, regulatory and adding little value; it is a threat rather than an opportunity for learning. Yet at the same time, evaluation is recognized as an integral part of result-based management.
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.
Some managers however seem to start changing the status quo. For example a revised and simplified template for producing project completion reports is being considered, and mid-term project reviews are expected to be more stringent in looking at monitoring plans and practices and tying disbursements to performance. In some cases we also learned of incentives being introduced to encourage project managers to complete their reports in a timelier manner. But much remains to be done and, since the OIE is no longer responsible for monitoring and project evaluations, there is a void that needs to be filled. It is up to line managers to drive this work forward.
To conclude, it is fair to say that in view of a number of “frustrations” between the OIE and operations, which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring data, the added value that evaluation might offer to the operations area is ill recognized. Moreover, the link between self-evaluation as the building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
420
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”242
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
242 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
421
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”243 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 244 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
243 CDB (2011) Evaluation Policy (p.2)244
422
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
423
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
424
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
425
mendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
426
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
427
APPENDICES
Appendix I - The External Review Mandate – Terms of Reference and Approach Paper
Appendix II -Review Approach, Data collection and Analysis, and Limitations
Appendix III – Overview of OIE Evaluation Practice
Appendix IV - List of Persons Interviewed
Appendix V - List of Documents Reviewed
Appendix VI- List of Topics used to guide interviews with members of CDB Board of Directors
Appendix VII - List of Topics used to guide interviews with CDB staff
428
Appendix III – Overview of OIE Evaluation Practice (prepared by the OIE in response to Reviewer’s request)
Caribbean Development Bank, Office of Independent Evaluation - OIE
Category Response
Percentage of projects subject to project (self-) evaluation
100% - Project Completion Reports (PCR)
Percentage of projects subject to validation by OIE
Approximately 40-50%
About 15 projects exit portfolio annually. Evaluation Policy calls for all PCR to be validated. However, OIE resources insufficient. Validation process reviewed in 2014. Now OAC (Board committee) selects a sample of 6-8 PCR for validation each year.
Percentage/number of projects subject to in-depth review by OIE
None – unless specifically requested by OAC
Due to limited resources, focus of OIE evaluation work programme is on PCR validations and high-level evaluations – including country strategy and programme evaluations (CSPE).
Number of high-level evaluations conducted by OIE (e.g. sector, thematic, geographic)
1-2 per year since 2011
Plan is 2-4 per year from 2016. This would include CSPE (1st planned for Q1 2016: Haiti)
Number of project impact evaluations conducted by OIE
None
OIE includes “impact questions” in high-level evaluations.
Number of project impact evaluations conducted by Bank staff or other non-OIE staff
OIE is not aware of any impact evaluation conducted by the Bank.
However, OIE provides technical support to the Basic Needs Trust Fund (BNTF) in its design of an M&E framework that entails impact evaluations.
Budget In USD mn: 0.78 in 2015; 0.82 in 2016. This is equivalent to about 2.5% of total CDB Administrative Budget.
75% of the budget is for Staff salaries (4 Professionals, 1 Support staff), leaving around USD 190,000 (in 2015) for other expenses, including consultants e.g. for external evaluations. Additional funding is accessed via the Special Development Fund (SDF). This varies according to type and scope of the evaluation, e.g. the ongoing SDF 6/7 Evaluation is SDF funded at USD 255,000.
Budget determined by Board, not separate from administrative budget.
SDF funding for evaluations is considered separately and subject to Bank internal approval process. SDF funding cannot be used to cover OIE expenses such as staff time or travel. Country eligibility for SDF funding is also a consideration. OIE expressed concerns about this funding track in respect to predictability, independence and eligibility limitations.
Head of OIE reports to Board, with administrative link to the President
Terms of appointment for Head
429
5 year term, renewable once. Appointed by the President with the agreement of the Board.
Right of Return for Head Not eligible for other staff positions.
Consultants as proportions of OIE budget
2015: 19% (USD 145,000)
Plus SDF funding. SDF funded evaluations are outsourced.
Last external evaluation (or peer review) of OIE
No external evaluation, though a review of the function was done in 2011, leading to the Evaluation Policy.
OIE External Review completed in April, 2016
Departments or special programmes supporting impact evaluation
None
430
Appendix IV – List of Persons Interviewed
Name Function relative to OIE Type interview
Mrs. Colleen Wainwright Member CDB Board of Directors (UK)
Face to face
Mrs. Cherianne Clarke Alternate MemberCDB Board of Directors (UK)
Face to face
Mrs. Jean McCardle Member CDB Board of Directors (Canada)
Face to face
Dr. Louis Woodroofe MemberCDB Board of Directors (Barbados)
Mr. A: de Brigard Former Member CDB Board of Directors
Skype interview
Mr. H. Illi Fromer Member CDB Board ofDirectors
Telephone interview
Mrs. Claudia Reyes Nieto Member CDB Board of Directors
Telephone interview
Mr. Bu Yu alternate DirectorCDB Board of Directors
Face to face
Mr. Michael Schroll(Barbados)
Head OIE
series of interviews viaSkype and face-to-face
Mr. Mark Clayton OIE Senior Evaluation Officer Focus GroupMrs. Egene Baccus Latchman OIE Evaluation OfficerMr. Everton Clinton OIE Evaluation OfficerMrs. Valerie Pilgrim OIE Evaluation Officer
Dr. Justin Ram CDB Director Economics Department
Face to face
Mr. Ian Durant CDB Deputy Director Economics Dept Face to faceDr. Wm Warren Smith CDB President
Joint interviewFace to face
Mrs. Yvette Lemonias-Seale CDB Vice President Corporate Services & Bank Secretariat
Mr. Denis Bergevin CDB Deputy DirectorInternal Audit
Face to face
Mr. Edward Greene CDB Division Chief, Technical Cooperation Division
Face to face
Mrs. Monica La Bennett CDB Deputy Director Corporate Planning Face to faceMrs. Patricia McKenzie CDB Vice President Operations Face to faceMs. Deidre Clarendon CDB Division Chief
Social Sector DivisionFace to face
Mrs. Cheryl Dixon CDB Co-ordinator, Environmental Sustainability Unit
Focus group
Mrs. Denise Noel- Debique CDB Gender Equality Advisor Mrs. Tessa Williams-Robertson CDB Head Renewable EnergyMrs. Klao Bell-Lewis CDB Head Corporate Communications Face to faceMr. Daniel Best CDB Director
Projects DepartmentFace to face
Mr. Carlyle Assue CDB Director Face to face
431
Finance Department
432
Appendix VI - Interview Guide: Members of CDB Board of Directors
Below are a list of themes that I should like to raise with you based on your experience and knowledge of the CDB’s independent evaluation function (Office of Independent
Evaluation).
In each case, I should be grateful if you could illustrate your responses with examples or help this Review by, wherever possible, sending me (or telling me where I can find) any
documents that could support your responses.
This guide is being sent to you in advance to help prepare our meeting. However, our interview will be conducted more in the style of a conversation. The following sub-questions will be used to GUIDE the interview. Please feel encouraged to raise any
additional issues that you feel we should take into account
On the governance and Independence of CDB’s evaluation functionWhat mechanisms are there in place to support its independence?
How satisfactory are the current arrangements in your opinion?
How is the balance between independence and the need for interaction with line management dealt with by the system? For example, what mechanisms exist to ensure that the OIE is kept up to date with decisions, policy / programme changes, other contextual changes etc that could have an affect on OIE evaluation studies / evaluation planning?
On the OIE’s Evaluation PolicyThe CDB’s Evaluation Policy was established in 2011. To what degree do you feel it is adequate? Still relevant?
What suggestions do you have for any improvements?
In your opinion, how adequate is the current quality assurance system for over viewing the evaluation function?
On the quality and credibility of evaluation studiesTo what degree do you believe the reports are fair and impartial?
Do you consider them to be of good quality? Are they credible?
Are you adequately consulted/involved on evaluations of interest to you?
On the relevance and usefulness of evaluations How well does the OIE engage with you / your committee during the preparation, implementation and reporting of an evaluation study to assure that it will be useful to the CDB?
How are the priorities set for the independent evaluations? What criteria are used? Are you satisfied with the current procedure?
433
When OIE evaluation studies are outsourced to external consultants, what criteria are used to make this decision?
How are the priorities for the OIE’s 3.year rolling work plan agreed? In your opinion, is the current plan adequate in terms of coverage and diversity?
In your opinion, do the evaluations address important and pressing programs and issues?
To what extent do you feel that the OIE’s evaluations integrate the cross-cutting theme such as gender, energy efficiency/renewable energy, climate change? What improvements might be made and how?
On the dissemination and uptake of evaluation findings and recommendationsTo what extent do you feel that evaluation findings are communicated to the CDB and its stakeholders in a
a) useful, b) constructive andc) timely manner?
Are evaluation recommendations useful? Realistic?
What mechanisms are in place to assure that evaluation results are taken into account in decision making and planning? What improvements do you feel could be made?
How have you used the findings from any evaluations? Examples?
To what degree do you feel that evaluation contributes to institutional learning? And what about to institutional accountability? Any examples?
What mechanisms are in place to ensure that knowledge from evaluation is accessible toCDB staff and other relevant stakeholders? Are the current arrangements satisfactory?
How satisfied are you with current arrangements? What expectations do you have for the future?
On resourcesHow is the OIE resourced financially and is this satisfactory?
What about the OIE staff, are all the important areas of expertise represented in the team?
On this Review of the Office of Independent EvaluationWhat are your expectations? What are you particularly hoping to learn from it?
Thank you very much for your cooperation and input
434
Appendix VII : Interview Pro-Forma – CDB Staff membersThis presents a list of the topics raised during interviews. It was used to guide the open-ended
discussion – this means that the sequence and exact wording of the questions may not necessarily have followed in this order or been asked in exactly this way.
Changeover to an Independent Evaluation Office? Expectations? Advantages and disadvantages??
Satisfaction with working relations between operations and the OIE from your perspective?
Process of dealing with the PCRs and CCRs? Advantages and limitations?
Quality and credibility of the validation process?
How are the self-evaluation reports used?
Credibility and Quality of OIE’s evaluation reports
Communication of self and OIE independent evaluations? To whom, in what way? Possible improvements?
In response, the OIE has greatly improved the presentation of technical reports by summarising the main points in its “Brief Reports” (e.g. the Tax Administration and Tax Reform and Technical and Vocational Education and Training evaluation). This is commendable and certainly a step in the right direction although the Panel considers that they should have a sharper focus on the strategic issues (which are the end of the brief rather than the beginning), be condensed and be made more “reader friendly”.
The Panel was also surprised to find that despite expressions of support for rigorous evaluation and its importance to the CDB, the OAC do not appear to be taking any firm position with regard to the paucity of available data. OAC has been made aware of the data problems in the BMCs (e.g. lack of rigorous monitoring and statistical data and the consequent effect on the rigour of OIE’s evaluations) as well as the delays in the submission of self-evaluations and their validations, yet there appears to be no OAC attempt to deal with such problems e.g. exerting any pressure on the CDB or on the BMCs through their representatives on the Board.
To conclude: The OAC has an expressed interest in advancing the role of evaluation as a strategic tool for CDB management. However, it is not performing its oversight function with sufficient firmness to bring about any change to the problems raised through evaluations, especially with regard to data issues and reporting delays. More generally there is a lack of any systematic
435
report on “follow up of actions agreed” which could be particularly useful for tracking changes as a consequence of an evaluation and management’s response. The OIE could do better justice to its oversight responsibly if it were to receive all background documents systematically at least two weeks before its meetings. Moreover, the volume and length of documents received at any one time is considered to be overwhelming. The number and/or importance of agenda items competing for attention at any one session is an additional handicap.
the Panelss
the PanelssUnit has access to all needed information and information sources
Extent to which the evaluation unit has access to the organization’s
a) staff, records, and project sites;
b) co-financiers and other partners, clients; and
c) programs, activities, or entities it funds or sponsors
Complies –The available evidence suggests that there is no reason to doubt such access. But systematic and easily accessible documentation is lacking in the CDB; it is one of its weak points.. Delays in getting hold of the relevant documents can have consequences on the timeliness of evaluation studies
23) There appears to be a that is of concern to the Panels (More on this point under the heading self and independent evaluations.)The OIE is not regularly invited in any capacity to these meetings or given a copy of the agenda or minutes; the OIE is occasionally invited to attend in order to discuss an evaluation report or management feedback. Its observer status at meetingsor as a do not necessarily provide the same insight as to the dynamics of management actions and/or decisions.
. Extent to which the evaluation unit has control over:
a) staff hiring,
b) promotion, pay increases, and
c) firing, within a merit system
Partially complies - All OIE staff members are treated in the same way as other CDB staff. The Head has limited control over the hiring, firing or promotion of OIE staff.
actual or potential conflict of interestThe PanelIt must be s
The PanelisThe; this affects also of y, Work PracticesThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. therefore The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation pyramid245 are brought out in the remaining sections of this Review, not least given the limited resources available.
But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities. sTheThes before completing the final
245 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
436
version. However, p are only submitted to the CDB line and senior managers.Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the following the recommendations of professional good practices and standards on participative approaches; it has succeeded in , The Panelsstaff fromas well as There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. OIE doesarrange .sthe Panel also wishes tonewly appointed rsA was evidentthey expressed interest in In one case, interest was followed up in practice; can be improvedfostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that can offer can add value to understanding the strengths and weaknesses of such strategies. Tduring this transitional phase, Manual to guide and support the independent evaluation process.and operations staff sevaluation activities. ,oOIE’sjudginga . As such, theyare thatAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An is processed as follows;the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
SThisrItand Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
The rmade
- is still considered to be good practice to have the elaborated in the initial design documents the246 such as Developmental Evaluation (Patton, 2010247)
-)
said abovePCHowever, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessonsthe Panelas done this ing
246 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.247 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press
437
on lessonsAlthough nothing has happened since, it is , sometimes indicate (Panel has already referred above to ’s lack of oversight in the use of evaluation.)
sThe Panelsevaluation work Moreover, the 2015 budget provides only US$2’000 for
communication – nothing of which is intended for outreach.Reviewerseither confusing or and
budgetedConsequently for 2015, theFigure 3: The MDB Evaluation Pyramid248
248 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
438
ys to be effective. A modest attempt has been made in 2015; OIE hased But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.man and financial resources to support its work
OIE’s Human Resources;
5eThree of the five were recruited from within the CDB. edfrom the Board that OIE
should embark on ee and for impact evaluations in particular,OIE’s ee Moreover, there
are many other designated OIE activities that should be recognised as valuable work; the
validations, building CDB and BMC evaluation capacity, providing supervision, advice,
knowledge management and brokerage as well as managing evaluation contracts, The
time needs of dealing with all of these may be underestimated in OIE’s budgets; all are
important for assuring best value from evaluation. The Panel is concerned that a demand
for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level
evaluations may undermine the importance and time needs of other essential
tasks.Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses.
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies
439
does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.Self-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes.types of evaluation y There appears to be little incentive to complete self-evaluations in a timelier manner.
.
; it is a threat rather than an opportunity for learning. Yis recognized as
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.a number of , which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring dataadded value that evaluation might offer to the operations area is ill recognized Moreover, the link between self-evaluation as the building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
440
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”249
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
249 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
441
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”250 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 251 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
250 CDB (2011) Evaluation Policy (p.2)251
442
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
443
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
444
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
445
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
Independence of the Office of Independent Evaluation (OIEIndependence is absolutely central to the integrity and trustworthiness of evaluation. It is an agreed requirement within the development agencies and in the evaluation community as a whole. In examining the issue of independence and good practice, reviewers are guided by the Evaluation Cooperation Group’s recommendations on good practices, the CDB’s Evaluation Policy and by the 2011 consultancy review of independence relative to the CDB’s evaluation and oversight division252. The appraisal is based on a comparison of the ECG’s recommendations on independence253 and the current OIE status.
OIE and Independence: Recommendations from the OECD Evaluation Cooperation Group (ECG)
The ECG’s considers the issue of independence according to three specific areas: organisational, or structural independence, behavioural, or functional independence and protection from outside interference, or operational independence.
Organizational independence, ensures that the evaluation unit and staff are protected against any influence or control by senior or line management, and have unrestricted access to all documents and information sources needed for conducting their evaluations. Also, that the scope of evaluations selected can cover all relevant aspects of their institution.
Behavioural independence, generally refers to the evaluation unit’s autonomy in selecting and conducting setting its work programme and in producing quality reports which can be delivered without management interference.
Protection from outside interference refers to the extent to which the evaluation function is autonomous in setting its priorities, and conducting its studies and processes and in reaching its judgments, and in managing its human and budget resources without management interference.
Conflict of interest safeguards refers to protection against staff conflict of interests be they current, immediate, future or prior professional and personal relationships and considerations or financial interests for which there should be provision in the institution’s human resource policies.
The OIE’s Independence in Practice
Organisational / structural independenceOn the whole, the Panel acknowledges and commends the efforts being made by the CDB to assure OIE’s organisational independence. The CDB’s Evaluation Policy provides for the OIE’s organisational independence from line management and the interview data suggests that there is also wide acceptance and acknowledgement of why the OIE should have such independent status. Table 1 below provides our overall assessment of this aspect of OIE’s independence when compared with ECG recommendations. 254
252 Osvaldo Feinstein & Patrick G. Grasso, Consultants, May 2011 Consultancy to Review the Independence of the Evaluation and Oversight Division of the Caribbean Development Bank253 ECG 2014 Evaluation Good Practice Standards, Template for Assessing the Independence of Evaluation Organizations, Annexe II.1 254 Based on ECG (2014) Template for Assessing the Independence of Evaluation Organizations, Evaluation Good
Practice Standards, Annexe II.1
446
Table 1: OIE organisational independence compared with ECG recommendations
Aspects Indicators CDB Evaluation Policy (EP) and Practice
The structure and role of evaluation unit
Whether the evaluation unit has a mandate statement that makes clear its scope of responsibility extends to all operations of the organization, and that its reporting line, staff, budget and functions are organizationally independent from the organization’s operational, policy, and strategy departments and related decision-making
Partially Complies The Policy is broad enough to cover the full range of MDB type of evaluations. However in practice this would not be possible without additional human and budget resources
The unit is accountable to, and reports evaluation results to, the head or deputy head of the organization or its governing Board
Whether there is a direct reporting relationship between the unit, and
a) the Management, and/or
b) Board or
c) relevant Board Committee, of the institution
Complies - OIE reports to the Board of Directors (BoD) through its Oversight Assurance Committee (OAC)
The unit is located organizationally outside the staff or line management function of the program, activity or entity being evaluated
The unit’s position in the organization relative to the program, activity or entity being evaluated
Complies - The OIE is located outside, and is therefore independent of CDB line management
The unit reports regularly to the larger organization’s audit committee or other oversight body
Reporting relationship and frequency of reporting to the oversight body
Complies - The OIE reports x 5 per year to the OAC . Board approval for an additional executive meeting between the Head of the OIE and the OAC at least once per year was given in October 2015
The unit is sufficiently removed from political pressures to be able to report findings without fear of repercussions
Extent to which the evaluation unit and its staff are not accountable to political authorities, and are insulated from participation in political activities
Complies
Unit staffers are protected by a personnel system in which compensation, training, tenure and advancement are based on merit
Extent to which a merit system covering compensation, training, tenure and advancement is in place and enforced
Partially Complies - with CDB human resource policy. However the skill needs of OIE staff ought to be regularly reviewed in light of its move towards higher-level evaluations. Appraisal of skill needs and hiring of relevant staff should be completely under the authority of the Head of Evaluation. This is not sufficiently clear in the Policy or other documents we reviewed.
Unit has access to all needed information and information sources
Extent to which the evaluation unit has access to the organization’s
a) staff, records, and project sites;
b) co-financiers and other partners, clients; and
Complies –The available evidence suggests that there is no reason to doubt such access. But systematic and easily accessible documentation is lacking in the CDB; it is one of its weak points.. Delays in getting hold of the relevant documents can have consequences on the timeliness of
447
Aspects Indicators CDB Evaluation Policy (EP) and Practice
c) programs, activities, or entities it funds or sponsors
evaluation studies
However, independence should not mean isolation: There appears to be a detachment between the OIE and CDB that is of concern to the Panel; on the one hand, between the OIE and operations staff, and (2) on the other, in terms of the structural arrangements between the OIE and senior management.
24) In agreeing for the OIE to concentrate on strategic and thematic, in-depth evaluations, responsibility for project monitoring and evaluation were given over to operations. The division is clear and respected. However, it has its drawbacks. With the OIE no longer systematically involved at the front-end of project design, the monitoring data needs are likely to be poorly defined. Weak monitoring data will contribute to weaker evaluations. (More on this point under the heading self and independent evaluations.)
In the reviewers’ opinion, it is a common misunderstanding to assume that providing evaluator advice on monitoring and evaluation data will comprise evaluator independence. On the contrary, evaluation input into project design is essential to assure that the logic, indicators and data needs are addressed so that at some future point in time an evaluation of the achievements can be empirically grounded.
This is not to say that the OIE no longer has any influence at the front-end design stage; it has merely shifted the point of focus. The OIE is now systematically providing such input more generally to the corporate planning teams for the tools and systems they are developing to support the MfDR framework. The monitoring data for projects and their implementation should be improved once the Project Performance Evaluation System (PPES) and the Portfolio Performance Management System (PPMS) are updated and operational.
25) In the second place, the OIE has limited formal access to the Advisory Management Team (AMT) weekly meetings where the President and senior management gather to exchange up-to-date information on the dynamics of CDB policy and practice. The OIE is not regularly invited in any capacity to these meetings or given a copy of the agenda or minutes; the OIE is occasionally invited to attend in order to discuss an evaluation report or management feedback. For the OIE, this means that it is unlikely to pick up on the ‘when’ and ‘what’ of key decisional issues or provide input into the discussion based on evaluative information. Its observer status at Loans Committee meetings, or as a participant informer at the OAC and BoD meetings and discussions do not necessarily provide the same insight as to the dynamics of management actions and/or decisions. .
To respond to this situation, the President has agreed to meet regularly with the Head of the OIE in order to keep him up to date with CDB strategic thinking. This is a welcomed change.
OIE Independence and Behavioural Issues The Panel has concerns about some behavioural issues. For example, through both the interviews and documentary review, we learned of considerable delays in processing both the independent evaluation reports as well as OIE’s validation of the CDB’s self-evaluations. Delays are generally due to receiving feedback on the independent reports from first, the relevant operational department, then from the AMT, and then on providing the OIE with a management response that is initially drafted by operations staff before being reviewed by the AMT. (OIE reports cannot be submitted to the OAC without the relevant management response). This two-layer process for preparing submissions to the Board is inefficient and could potentially be a
448
threat to evaluation’s independence in the future by delaying OIE’s timely reporting to the OAC.
OIE validations of the CDB self-evaluations are also submitted to the OAC, but it is in both sides’ interest to clear up any misunderstandings beforehand. Despite attempts to improve the timeframe for completing these validations, delays are more the norm than the exception. Table 2 below summarises our assessment of the behavioural aspects of independence.
Table 2: OIE and Behavioural Independence
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Ability and willingness to issue strong, high quality, and uncompromising reports
Extent to which the evaluation unit:
a) has issued high quality reports that invite public scrutiny (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk) of the lessons from the organization’s programs and activities;
b) proposes standards for performance that are in advance of those in current use by the organization; and
c) critiques the outcomes of the organization’s programs, activities and entities
Partially complies – paucity of data and documentation sometimes hinder the quality of reports. The OIE emphasizes the learning part of evaluation, and is cautious in its criticism recognising that management is going through a transitory stage and can still be overly defensive.
Ability to report candidly
Extent to which the organization’s mandate provides that the evaluation unit transmits its reports to the Management/Board after review and comment by relevant corporate units but without management-imposed restrictions on their scope and comments
Partially complies - as sometimes reporting to the Board is compromised by delays in the review/comment process between the OIE and the CDB. Any delay with the production of a Management Response will also mean that submitting a report to the Board in a timely manner is impaired since the two have to be submitted together.
Transparency in the reporting of evaluation findings
Extent to which the organization’s disclosure rules permit the evaluation unit to report significant findings to concerned stakeholders, both internal and external (within appropriate safeguards to protect confidential or proprietary information and to mitigate institutional risk).
Who determines evaluation unit’s disclosure policy and procedures: Board, relevant committee, or management.
Partially complies - The OIE’s conforms to the CDB’s disclosure policy. However, the dissemination of evaluation findings appears to be currently restricted to website publication and reports to the Board. A more targeted communication strategy to include other key stakeholders, e.g. project implementers in the BMCs should be developed and put in place.
Self-selection of items for work program
Procedures for selection of work program items are chosen, through systematic or purposive means, by the evaluation organization; consultation on work program with Management and Board
Complies - The OIE also ensures that its work program is drawn up after consultation with both CDB Management and Board to seek their input on relevant topics and themes.
Protection of administrative budget, and other budget
Line item of administrative budget for evaluation determined in accordance with a clear policy parameter, and
Partially complies - The administrative budget for supporting OIE work is protected. Access to additional sources of
449
Aspects Indicators CDB Evaluation Policy (EP) and Practice
sources, for evaluation function
preserved at an indicated level or proportion; access to additional sources of funding with only formal review of content of submissions
funding is possible if well argued and justified. But the approval process is complex and inefficient. (See Figure 1 below)
OIE and Protection from External influence or interference
Our overall assessment is provided in Table 3 below. The OIE’s independence in the design, conduct and content of its evaluations does not appear to be subjected to any external interference. But securing funding from any sources outside the OIE’s administrative budget, i.e. from the Social Development Fund, is an unduly complex and long process. As such we consider that the current funding process can affect the OIE’s choice with regard to the type of evaluations it can undertake. (See Figures 1 and 2 below)
Table 3: OIE and its Independence from External influence or interference
Aspects Indicators CDB Evaluation Policy (EP) and Practice
Proper design and execution of an evaluation
Extent to which the evaluation unit is able to determine the design, scope, timing and conduct of evaluations without Management interference
Complies – however within limits of restricted human and financial resources available
Evaluation study funding
Extent to which the evaluation unit is unimpeded by restrictions on funds or other resources that would adversely affect its ability to carry out its responsibilities
Partially Complies - OIE must work within the limits of the agreed administrative budget wherever possible. If additional resources are needed for studies it must seek alternative funds elsewhere. The budget limitations can have an affect on the type of evaluations undertaken and therefore its independence in terms of choice.
Judgments made by the evaluators
Extent to which the evaluator’s judgment as to the appropriate content of a report is not subject to overruling or influence by an external authority
Complies – the evidence available suggests that the Board and Management accept the evaluators’ independent interpretation and conclusions Management responses are agreed to be the accepted place to raise any difference of opinion.
Evaluation unit head hiring/firing, term of office, performance review and compensation
Mandate or equivalent document specifies procedures for the
a) hiring, firing,
b) term of office,
c) performance review, and d). compensation of the evaluation unit head that ensure independence from operational management
Complies – the Head of OIE is appointed by the CDB President in agreement with the OAC for a 5 year period which is renewable x 1. The Head could be removed from Office by the President or the Board but only with the agreement of both parties.
However the Head reports to the President for all administrative and personnel matters. Even though this was not recommended in the Osvaldo Feinstein & Patrick G. Grasso report on Independence in 2011, the BoD accepted CDB’s reasons for keeping this arrangement. (e.g.most OAC members are non residents and cannot oversee day-to-day work)
. Extent to which the evaluation unit has control over:
a) staff hiring,
Partially complies - All OIE staff members are treated in the same way as other CDB staff. The Head has limited control over the hiring, firing or promotion of OIE staff.
450
Aspects Indicators CDB Evaluation Policy (EP) and Practice
b) promotion, pay increases, and
c) firing, within a merit system
Continued staff employment
Extent to which the evaluator’s continued employment is based only on reasons related to job performance, competency or the need for evaluator services
Partially complies - Whilst the EP is clear about procedures for hiring, firing and promotion, all of which must conform with CDB human resource policy, there is nothing mentioned about any difference of opinion between the CDB and the Head of the OIE with regard to continued staff employment subject to changes in the level of technical or interpersonal competencies needed to meet new demands.
Avoidance of Financial, Personal or Professional conflicts of interest
This particular aspect refers to the organisation’s Human Resources Policy; there must be provisions in place to protect against actual or potential conflict of interest. The Panel requested via the OIE, to have evidence from human resources on any such provisions but did not receive an answer. It must be assumes that this aspect of independence, past or present, does indeed form part of normal CDB Human Resource Policies
To conclude: The Panel is impressed with the measures CDB has taken to assure the organisational independence of the OIE. Its independent status is accepted and respected by senior and line management. The OIE’s budget is not independent from the overall CDB administrative budget; this affects its choice of evaluation types or approaches. Some of the behavioural issues affecting independence were also of concern, especially due to the delays in the exchange of documents, between the OIE and operations departments, which has a direct effect on timely reporting to the OAC. As for protection from outside interference, our concerns are largely to do with OIE’s independence over staffing issue; there are potential loopholes in current arrangements that could undermine OIE’s autonomy over its staff.
OIE’s Strategy, Work Practices and Work ProgrammeThe OIE has had to develop a plan to implement the Evaluation Policy. This raises such questions as what are the priorities and what is the timeframe for achieving which activities? These were partially addressed in the OIE work programme and budget 2012 to 2014, but it proved to be over ambitious. Much of the period 2012 to 2015 has therefore been taken up with preparing OIE’s shift in focus from project-based evaluations to the high-level thematic and in-depth strategic studies. This has meant adopting a three-way approach; (1) for self-evaluations, reducing its time input to support the process and (2) for independent evaluations, taking stock of the gaps in coverage and expertise, and (3) networking to share experiences with centres of expertise and align OIE with international practices. In addition, amongst other duties, it has been supporting the development of MfDR tools and systems such as the Project Performance Assessment System by providing advice and input on programme logic and monitoring needs. The OIE plans to conduct 2-4 high-level studies per year from 2016. The OIE has also chosen to increase the involvement of its professional staff in conducting independent evaluations. Outsourcing is still needed; when the study is funded by the SDF, when time is limited and when specific expertise is needed.
But plans appear to place little emphasis on the activities associated with evaluation management (e.g. knowledge management) and the relevant time needed. Other time demands mentioned in the previous sections, such as delays in completing reports, validation work etc, have also affected OIE’s plans. The more recent work plans have set the task of devliering utility-focused and timely evaluations. But it lacks clarity on how the OIE proposes to surmount the
451
time and data issues, which are far from new. In short it lacks a theory of change and timeline. The challenges that have to be dealt with to enable the OIE to move up the MDB evaluation pyramid255 are brought out in the remaining sections of this Review, not least given the limited resources available.
To conclude: The OIE has made a first step in proposing a strategy for establishing itself as an independent evaluation resource. But its strategy is lacking a theory of change and prioritisation of tasks, which should include more emphasis on evaluation management activities.
The Value / Usefulness of OIE’s Independent EvaluationsEvaluation is a powerful tool that can provide useful, evidence-based information to help inform and influence policy and practice. But useful evaluations depend not only on the evaluators’ skills, but on several other important factors as well; 1) on planning evaluations to be relevant to the priorities of the organisation’s work and for their results to be delivered in time to be useful; on the degree of 2) consultation and ultimately ownership by those who seek evaluative information; on the 3) tools used to support the evaluation process per se; and on the 4) credibility and quality of the evaluation products256.
1. Planning relevant and timely evaluationsThe OIE is now working on a 3 year rolling work plan that sets out the broad areas for enquiry. So far, there are no agreed criteria for making the selection of the specific topics for independent evaluation, although the priorities tend to reflect those of the CDB’s strategic plan. Nevertheless decision-making is rather arbitrary based on a process of dialogue between the OIE and the CDB and the OIE and the Board.
One of the OIE’s two objectives for 2015 therefore, was to define a work plan and agree priorities based on an approach that is “utilisation-focused”. This means that the studies are selected and planned to be relevant and useful to the organisation’s needs.
The OIE has achieved this objective with respect to its latest studies, which concerns the Social Development Fund (SDF) Multicycle 6&7 Evaluation, the Haiti Country Strategy evaluation and the evaluation of the CDB’s Policy Based Operations. Each of these three have been planned to deliver their results in time to provide the CDB Board of Directors with relevant information for negotiating the next round of funding. In spite of some delays due to a myriad of reasons, not least to the extra effort needed to secure essential data, the studies are expected to deliver on time.
The processes for agreeing OIE’s work plan and specific evaluations on the one hand, and, in securing alternative funding on the other, are shown in Figure 1 below. The Panel was surprised at learning how bureaucratic (the internal approval process), and inefficient (in view of the time it takes) the process seems to be. The concern here is that such a process could possibly pose a threat to assuring the Board of “timely studies.”
Figure 1: Selection of Evaluation Topics and Funding Source
255 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).256 These aspects reflect the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
Consultation with CDB Operations and OAC/Board for selection of
evaluation topic
3-year Work Programme and Budget (approved by Board)
Annual OIE report and work plan
submission to OAC
452
2. Consultation and ownership“The credibility of evaluations depends to some degree on whether and how the organization’s approach
to evaluation fosters partnership and helps build ownership and capacity in developing countries.”
(ECG good practices)
Internal review of Approach Paper
Specific Evaluation Study Design and Budgeting
OIE Draft Terms of Reference / Approach
Paper
Detailed ToR or Final Approach Paper if sufficiently detailed.
Finalise Approach Paper and submit to OAC/Board
Final Approach Paper
OAC ApprovalOAC minutes
Paper
Funding Track
Final Approach Paper/ToR
Board approval necessary If above USD
150,000
Board notification only if USD 150,000 or
below
Board Approval
Board Paper
OIE – Selection of consultants (if any) contracting
OIE Admin Budget or …
… SDF
Prepare TA Paper (content similar to Approach Paper but different
format.
TA Paper
Approval – Internal Loans Committee
OIE – Selection of consultants (if any)
contracting
453
The OIE engages with the OAC, CDB senior management and operations for agreeing its 3-year work plan and then for selecting the specific topics and themes. It also discusses the evaluation approach paper (design and implementation plan) with the CDB and OAC before completing the final version. However, preliminary and final drafts of the report are only submitted to the CDB line and senior managers for comment and factual errors. Only final versions are given over to the OAC. A series of discussions are held with the CDB first and then with the OAC on the results and their implications. Discussions with the OAC are more limited due to the overburdened agenda of OAC and Board meetings, as previously discussed.
In short, the OIE is to be commended for following the recommendations of professional good practices and standards on participative approaches; it has succeeded in having introduced a modus operandi that involves the key players in the selection of evaluation topics, the evaluation designs and their results. Figure 2 below provides an overview of the evaluation implementation and stakeholder engagement processes.
Figure 2: Evaluation Study Implementation and Feedback Loops
Arrangement AFully outsourced / external
consultants; oversight by OIE
Preparations:Detailed evaluation plan (incl tools,
timeline, etc.) and logistics
Production of Inception Report / Approach Paper
Arrangement BConducted by OIE
staff
Arrangement CJointly: external
consultants and OIE
Terms of Reference
Prepares Inception Report /
Approach Paper
Presentation/workshop:Interim findings and conclusions for immediate feedback and validation
Data Collection and Analysis
OIE
Summary and ppt for workshop presentation
and discussion with CDBSubmission of Draft Final
Report to OIE
Final OIE approved report to CDB Senior Management for Management Response
Board notification only if USD 150,000 or
below
Draft Final Report
Review loops – OIE and CDB (potentially also BMC)
Feedback to evaluation lead
Submission of Final Report to
OIE
454
21.22.
Notes to Figure 2
21. The OIE informed the Panel that this is an abbreviated version as there are e.g. additional steps (secondary processes) when evaluations are procured (tendering or single source), when there are additional review loops and updates to OAC etc.
22. OAC may also decide to return the report to OIE, the Panel were informed, or demand from Management specific actions based on the report.
This process is engaging and appears to have secured senior management and OAC interest and buy-in as witnessed in the latest studies. But there is the downside too! The process takes much time and, in our view, is partly unnecessary. The Panel appreciates that staff from operations as well as the AMT may both want to confer on an appropriate management response, but this should not be the case for reviewing an independent report for factual errors. The two-phase approach seems somewhat inefficient and unnecessary in our opinion.
Contact between the OIE, the CDB and/or the OAC during the actual study implementation is most often restricted to the occasional progress report, particularly when studies run behind time. There is no “accompanying group” for individual studies, which would include both internal and possibly external partners. Such “advisory groups” have shown their worth in a number of contexts for improving buy-in and providing strategic input as well. The OIE does, however, arrange discussions for reflecting on emerging findings, but we are not sure of how systematic this feedback loop is.
More generally speaking, outside of an evaluation study, the OIE has limited dealings with operations. The OIE has an advisory role in providing them with help, particularly with providing training, guidelines and tools to support self-evaluations. We are nevertheless concerned about the seeming distance between these two and how this has affected the perceived value of evaluation. (For further on this point, please see the section below on “Self- and Independent Evaluations”)
But the Panel also wishes to stress that this is not the case for newly appointed senior managers. A much more open attitude to evaluation and appreciation of its potential value was evident;
Prepare for disclosure and dissemination
Final Report
Final Report and Management Response submitted to
OAC/BoardFinal Report and
Mgt. Resp.
Management Response
OIE ApprovalFinal Report and Management Response considered by CDB
AMT
OAC/Board endorsed
455
they expressed interest in drawing out important lessons on what works, how, for whom, and under what conditions. In one case, interest was followed up in practice; the OIE was recently invited by a senior manager to share evaluative knowledge and experience with his staff regarding policy based operations.
Certainly, we can say that overall, the key stakeholders within the CDB are adequately integrated into the evaluation process as to foster their buy-in and ownership. But more generally, we feel that the utility of independent evaluations can be improved by fostering a supportive climate that wants to learn through calculated trial and error. The constructive criticism that evaluation can offer can add value to understanding the strengths and weaknesses of such strategies. This however cannot be done overnight and takes a long time.
3. Tools to support the evaluation processSo far, during this transitional phase, the OIE has mainly focussed on improving the tools to support the operations areas’ self-evaluations. This has left the OIE with little time to produce the checklists or tools to support its own studies. There are plans to develop an OIE Manual to guide and support the independent evaluation process. Such plans should be encouraged, as these documents will form a very important part of training, particularly for newcomers to the OIE team.
In the meantime, the OIE and operations staff refers to the Performance Assessment System (PAS) Manuals for evaluation activities. The manuals are based on DAC criteria and ECG principles. Much emphasis is given to the rating system and how and what should be rated. However we find them lengthy, unwieldy and overcomplicated. Moreover, such manuals should be used for reference, but cannot and should not replace first-hand training in how to plan, conduct and manage the evaluation process.
Quality Assessment (QA) and Quality at Entry (QaE)
There was a transition period between 2012 and 2014 to establish the OIE. Work on the PAS, QaE, PCRs, ARPP, which had started earlier, was therefore completed after OIE came into existence, but it effectively had no formal ‘home’ in operations. The Panel was told that there had been some discussions about creating a Quality Assurance unit within CDB (OPS) but the current status is unclear.
The QaE Guidance Questionnaire was developed before and completed by the OIE. It was used to assess the documents that came across to the OIE for comments at the Review Stage. The results were then sent to the Portfolio Manager/Project Coordinator indicating any gaps/issues that needed to be addressed or clarified. QaE Guidance Questionnaires were developed for all the Bank’s lending products, CSP and to assess the quality of supervision.
After the QaE was launched bank wide, several operations officers saw the merit in using the QaE Guidance Questionnaire in the field and adopted it as a tool for their use during the appraisal mission in order to cross check and test their data collection and analysis.
OIE’s use of the QaE was discontinued in 2014 due to limited resources and a stronger focus on evaluations. It still sometimes comments on specific appraisals, but very selectively.
Both QaE and QaS (quality at supervision) are also addressed in the PAS Manuals. In addition the QaE and PAS have been incorporated in Volume 2 of the Operations Manual OPPM.
The Review Panel assessed the QaE forms. They are relatively standard, adapted to the specificities of the CDB. They contribute to judging a project’s expected quality in a relatively objective way. As such, they are are helpful, as a benchmark, in the ex-post assessment of projects.
The Panel considers that the lack of an established Quality Unit in the CDB (and independent from OIE) is a weakness that should be addressed in the near future.
456
4. Credibility and Quality of Evaluation ProductsAs with many other MDBs, evaluation activities include both independent and self-evaluations; the latter are the results of completion reports on operational projects and country strategy programmes and are done by the operations staff. The OIE then validates the quality of such reports. The self-evaluations should inform the more strategic studies conducted independently by the OIE. (More on the relationship between these two is provided later in this Review).
An independent evaluation is processed as follows; the OIE prepares an Approach Paper (AP) for approval by the OAC. If the study is to be outsourced, the AP becomes the basis for a Terms of Reference (ToR), which, subject to the size of the budget, may be put to tender. The contracted evaluator then prepares an Inception Report (IR) after some desk and field research has taken place. This intermediary report is not done if the OIE itself is conducting the evaluation. Sometimes a Progress Report is submitted, but otherwise the next stage is the delivery of the final report in various drafts. (Assessments are like evaluations but more limited in scope and depth of analysis)
Since 2012, the OIE has produced a range of studies and approach papers. This review is based on those listed below as provided by the OIE, and cover the period from May 2012 to December 2015. It includes 3 evaluations (in blue), 4 Assessment studies (in brown) 14 validations of self-evaluations (in green) and 3 Approach Papers (in purple) for upcoming evaluations. These are listed below in Table 4.
Table 4: List of studies (N = 24) submitted to the Board during for the period January 2012 to December 31 2015
Board Meeting
Date Type / Topic
251 May 2012 Ex-Post Evaluation Report on Road Improvement and Maintenance Project, Nevis -St. Kitts and Nevis.
Validation of Project Completion Report on Sites and Services – Grenada. Assessment of Effectiveness of Implementation of Poverty Reduction
Strategy 2004-09.253 Oct. 2012 Assessment of Extent and Effectiveness of Mainstreaming Environment,
Climate Change, Disaster Management at CDB.254 Dec. 2012 Assessment of the Implementation Effectiveness of the Gender Equality
Policy and Operational Strategy of the Caribbean Development Bank. Validation of Project Completion Report on Enhancement of Technical and
Vocational Education and Training – Belize. Validation of Project Completion Report on Fourth Road (Northern Coastal
Highway Improvement Section 1 of Segment II) Project – Jamaica. Assessment of the Effectiveness of the Policy-based Lending Instrument.
256 May 2013 Validation of Project Completion Report on Expansion of Grantley Adams International Airport – Barbados.
Validation of Project Completion Report on Fifth Water Supply Project – Saint Lucia.
261 May 2014 Validation of Project Completion Report on Immediate Response Loan, Tropical Storm Gustav, Jamaica.
Validation of Project Completion Report on Social Investment Fund, Jamaica.
Validation of Project Completion Report on Disaster Mitigation and Restoration – Rockfall and Landslip, Grenada.
263 Oct. 2014 Validation of Project Completion Report on Basic Education Project – Antigua and Barbuda
263 Oct. 2014 Approach Paper for SDF 6 & 7 Multicycle Evaluation
264 Dec. 2014 Validation of Project Completion Report on Policy-Based Loan – Anguilla
457
Validation of Project Completion Report on Immediate Response Loan - Tropical Storm Arthur – Belize.
Evaluation of Technical Assistance Interventions of the Caribbean Development Bank Related To Tax Administration and Tax Reform in The Borrowing Member Countries 2005-2012.
265 March
2015
Approach Paper for the Evaluation of Policy Based Operations
266 May 2015 Validation of Project Completion Report on Upgrading of Ecotourism Sites – Dominica
The Evaluation of the Caribbean Development Bank’s Intervention in Technical and Vocational Education and Training (1990-2012)
267 July 2015 Validation of Project Completion Report on The Belize Social Investment Fund I Project − Belize
268 Oct.2015 Approach Paper Country Strategy and Programme Evaluation, Haiti
The review and analysis of these documents is based on the UNEG Quality Checklist for Evaluation Reports (http://www.uneval.org/document/detail/607) as well as on ECG guidance (Big Book on Good Practice Standards).
Approach Papers
Three Approach Papers (APs) were made available to the panel (see Table [ref] above). An AP describes the rationale for the evaluation, the background to the topic evaluated, the evaluation framework (criteria and questions) and approach. It also describes the team and provides an initial planning. Being the first main deliverable of OIE’s evaluation process, APs are the starting point and therefore a major determining element in the roll-out of each evaluation. Therefore APs “have to get it right”.
The APs examined are clearly written, well-structured and of reasonable length.257 We were surprised to find, however, that they do not make explicit the objectives of the evaluated intervention(s), e.g., through a clear objective tree, or through an explicit theory of change, intervention logic or logframe. Whilst one of the APs contains, in an appendix, a results framework for the evaluation, the results framework for the intervention (PBO) itself is lacking.
Inception reports
Only one Inception Report was given to the Panel for review (SDF 6&7). This gives an in-depth description of the evaluated programme and provides a clear Theory of Change. It is good practice that this is established after a pilot field mission, which helps to amend the initial AP on the basis of field observations and sharpen the evaluation questions if needed.
However, it is still considered to be good practice to have the Theory of Change elaborated in the initial design documents . This would facilitate OIE evaluations after project completion. Establishing the Theory of Change of any intervention would be included in the QaE form more explicitly, to be developed between the Quality unit referred to above, and OIE.
Evaluations and Assessments
Three evaluations and four assessment reports completed during the review period were considered. Assessments are similar to evaluations but have a narrower scope; they focus on a limited set of aspects or judgment criteria, mainly effectiveness, i.e. achievement of objectives.
257 Opportunities remain of course to be more concise and to move parts to appendices, e.g., detailed descriptions of the evaluation team or part of the description of the evaluated intervention.
458
Evaluations generally base their judgment on the internationally recognised DAC criteria as well as aspects of the CDB and BMC’s management of the intervention.
In general, these reports are of reasonable quality. In the main, they explain the evaluated object258 and provide evaluation objectives. The findings are organised around the evaluation criteria or questions detailed in the scope and objectives section of the report. They are based on evidence derived from data collection and analysis methods as described in the methodology section. The reports tend to dwell on the limitations that the evaluation encountered, but without becoming defensive. In one case (PBL Assessment) the report starts with a summary of the reviews on the topic done by other MDBs. This was a pleasant surprise and indeed a good practice that could well be adopted in future evaluations too.
However, the reports also show several significant weaknesses:
- Reports do not always provide clear (reconstructed) intervention logics or theories of change for the intervention(s) evaluated.259 Evaluation criteria and questions are defined at a fairly general level. They are translated into more precise “research questions” (in an “Evaluation Design Matrix”, for each project for each criterion). However, it is unclear how these questions relate to the intervention logic (as this is not made explicit). This may be done in inception reports (of which, as noted above, only one was available for review), but should be done also in the final reports.
- The reports do not describe the link from the evaluation questions to the answers, how the evaluation judgments are made and how these ultimately transform into ratings for each criterion and each project. In other words, the explanation provided in the evaluation frameworks is inadequate. The “evaluation design matrix” currently used does not provide sufficient insight into how ultimately an intervention’s performance is judged.260 Links between findings, conclusions and recommendations could be improved by making this more explicit. In other words, reports should include the story on how the evaluand is credibly linked to any observed outcomes and impacts, and should be clear on how causal claims are made.
- With the exception of the PBL Assessment, reports are lengthy and detailed. One reason for this is an over-emphasis on ratings. Their detailed discussion, project by project, criterion by criterion, occupies a very prominent position in the evaluation reports’ main body of text. Although ratings are traditionally an important element in evaluations of MDBs, too strong an emphasis can be tedious and may distract the reader from the real lessons to be drawn. The detailed discussion of ratings, and their evidence base, would be better placed in an Appendix, with a brief summary in the main report. This would help give the lessons and recommendations a more prominent position than is now the case. This would also help make the evaluation reports not only shorter but also more interesting to read; this could help add value to evaluation’s image within the organisation.
- The reviewers feel that the OIE evaluations tend to over-emphasise objective-based evaluation261 and the DAC criteria to the exclusions of considering other evaluation
258 Sometimes in great length: for instance with the SDF 6&7 multicycle evaluation report it is only at page 30 that we find the beginning of the report on findings…259 Again with the SDF 6&7 evaluation, it is said to be guided by a “Logic Model” which is not explained.260 Marlène: I moreover have the idea that the methodology (often described as “visits”) is based on interviews and little hard evidence. Any view on this?.JM: My “interview-based evaluations”!!261 The focus of an objectives-oriented evaluation is on specified goals and objectives and determining the extent to which these have been attained by the relevant intervention. See for example, Worthen, Sanders, & Fitzpatrick (1997) ). Program Evaluation: Alternative Approaches and Practical Guidelines. (2nd Ed). White Plains, NY: Addison Wesley Longman.
459
approaches such as Developmental Evaluation (Patton, 2010262); evaluation should be case specific and answer the actual information needs of managers and other decisions makers rather than always concentrating on final performance.
- Related to the previous point (and again with the exception of the PBL Assessment) executive summaries (approximately 8 pages) are too long. For the evaluation report to increase potential impact, they would need to be reduced to 2 to 3 pages and be more focused; again this could be done by dwelling less on the individual ratings of projects and more on key findings, lessons and conclusions. More generally, reports could be better adapted to the needs of the different audiences. Although not strictly limited to evaluations, The Health Evidence Network Reports263 are a model that could be adapted for evaluation reporting purposes; they are specifically geared towards addressing policy and decision-making.
- The “Recommendations to BMCs” are an interesting feature of the reports, (although we are unsure to what degree such recommendations could be effectively followed up by OIE or the Bank, but certainly could taken up with BMC Board members.
- Reports (e.g. the evaluation report on Technical Assistance) focus much on technical problems that were encountered during the evaluation. Although these are important issues, again to improve the report’s flow and “readability” this section would be better placed in the Appendix. What counts is the story of the intervention, not the story of the evaluation (see “Limitations” section in the TA report for instance)
OIE Validations of Project and Country Strategy Programme Completion Reports (referred to globally as PCRs hereafter)
As said above, the OIE has the mandate to validate the Project and Economic departments PCRs and CSPCRs. However, in this period of transition, much of the OIE’s work since 2012 has been dealing with the backlog of the CDB self-evaluation validations. In theory, there is an estimated 15 completion reports due each year. However, delays in submitting the reports for validation is commonplace. Therefore with the change of Head in June 2014, the OIE has secured the OAC’s agreement to reduce the number of validations to a maximum of 6 per year. However, there is a continued backlog accumulating as only 2 PCRs were given to the OIE for validation in 2015.
The validations tend to repeat the different items reported in the PCRs and then provide extensive comment on each. The PCVRs go into great depth and detail, which makes the documents rich and complete. This is their strength – but also their weakness. The depth and level of detail, as well as the repetitions from the original PCRs, makes PCVRs (overly) lengthy (20-40 pages) and difficult to read. The OIE reported spending approximately 27.2% of its time on validating PCRs in 2015 compared with 44.4% on its core work, i.e. doing or managing the higher level evaluations. That is more than half of its evaluation work is being spent on the validation process. Finally, the PCVRs now seem to be, to a great extent, a standalone output of OIE. It is not always clear to us how they are being used as the “building blocks” for the OIE’s independent evaluations. Making this clearer in the independent evaluations would help show the link and therefore the value of the time being spent on the self-evaluation validations.
To conclude, the review finds that the OIE has taken steps to improve the perceived utility of evaluation in several ways. In the first instance, by planning its work to provide relevant and timely evidence geared towards helping the Board with its oversight and decision making tasks. The topics are selected through dialogue between the OIE and key CDB stakeholders and reflect
262 Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guildford Press263 See the reports available at the WHO’s Health Evidence Netowkr at http://www.euro.who.int/en/data-and-evidence/evidence-informed-policy-making/health-evidence-network-hen
460
priorities of the CDBs strategic plan. Secondly, by securing the interest and consequently the buy-in of the OAC and CDB senior management through engaging their input throughout the evaluation process. This is evidenced by the reported interest in the latest three studies, the Country strategy programme in Haiti, the evaluation of policy-based operations and the SDF 6& 7 multicycle assessment.
The OIE products are of an acceptable quality and could be even better if some of the shortcomings were addressed. However, the products themselves do not impair the utility of OIE’s work; this is undermined in several ways: (1) by the time delays in commenting on PCRs (OIE) and providing feedback to the independent evaluations (operations and management) (2) by the inefficient processes for agreeing topics and funding sources as well as providing OIE with management responses to its reports.
Putting Evaluation to Use: transparency, feedback and follow-upThere are several ways that evaluation can be, and is being used. As John Mayne has pointed out in his many publications on the issue,264 when we talk of evaluation use, we are mainly thinking about its Instrumental use—use made to directly improve programming and performance. But there is also conceptual use - use which often goes unnoticed or more precisely, unmeasured. This refers to the kind of use made to enhance knowledge about the type of intervention under study in a more general way. Or even Reflective use— this refers to using discussions or workshops to encourage and support reflection on the evaluation findings to see how they might contribute to future strategies.
In the case of the CDB there is some evidence to suggest that “use” is not only instrumental, but other types are also developing. For example, in the review of draft evaluation reports, the process includes reflective workshops that discuss not only the findings, but also seek to draw out the important lessons. (Reflective use)
Another important use, as recommended by the ECG, is that from time to time a synthesis of lessons is drawn from a number of evaluations and made available publically. In fact the Panel was impressed to hear that in the past, the evaluation unit had done this drawing on lessons from evaluations of the power sector. (Conceptual use) Although nothing has happened since, it is now on the “to do list” for 2016 (OIE’s 2016 Work Plan).
As for instrumental use, responsibility for using the knowledge generated through evaluation and for possibly drawing up an action plan of what should be done is up to CDB senior management and the relevant CDB department and division. Oversight on applying recommendations and picking up on the lessons drawn is the responsibility of the OAC.
Evidence on how evaluations have actually contributed to decisions or negotiations is lacking or confusing, Certainly the OIE is unaware of the extent to which its evaluations are put to use. On the one hand, the OAC minutes sometimes indicate that lessons learned are integrated into the next phase. On the other hand, the reviewers were told that often in the past, the evaluation results were “too old” to be of use as the lessons had already been drawn and used way before the report was completed. Similarly, people’s gaps in memory on how well the evaluative information from previous studies may have been used may also account for the scarcity of evidence.
In response, the Panel questioned CDB staff and the OIE about a particular study, the Technical and Vocational Education and Training Assessment. The feedback was somewhat contradictory. On the one hand, the study was criticised as “confirming” news rather than bringing “new news”. However, on the other, we learned that In October 2015, the Board of Directors approved a proposal for the revision of CDB’s Education and Training Policy and Strategy. Work on this has already begun and an external consultant has been engaged to lead the process.
264 See for example, his opening chapter to Enhancing Evaluation use: Insights from internal Evaluation Units, Läubli Loud, M. and Mayne, J. 2014, Sage Publications
461
Although it is one of the OIE’s tasks to set up a database on results and lessons learned from evaluations, so far this has not been a priority. There is also currently no systematic tracking of lessons or recommendations arising from the evaluations, or on any progress in their uptake. (The Panel has already referred above to OAC’s lack of oversight in the use of evaluation.)
The OIE’s role in supporting CDB’s organisational learning is clearly specified in the Evaluation Policy, with many good suggestions for knowledge sharing activities such as “brown-bag lunches, workshops, pamphlets and short issues papers” (p. 19). So far, however, the OIE’s lead role on the knowledge sharing side appears to be quite limited. It has provided advisory input in Loan Committee discussions, and organises workshops together with the relative operations department for discussing the implications of evaluation studies. Ultimately, of course, the uptake of evaluation results and knowledge is in the hands of management. But the evaluation unit has an important role to play in terms of knowledge broker and knowledge manager. Both have tended to be underplayed in OIE’s work plan so far.
Transparency: The Communication Strategy
In recent times and with the approval of its new Disclosure Policy, the CDB has started to post its independent evaluation reports on its website. (There is nothing on the self-evaluations). The website also presents a good overview of the role and function of the OIE and evaluation within the CDB. This is a step in the right direction for sharing information. However, in our view, the CDB’s communication strategy is the weakest part of the evaluation system to date.
The Panel has already commended the OIE in its efforts to engage the CDB and the OAC in evaluation work. But reporting and communicating the lessons seem to be entirely targeted at the Board and the CDB. Moreover, the 2015 budget provides only US$2’000 for communication – nothing of which is intended for outreach.
Reviewers feel that actively engaging with the more indirect stakeholders, for example project implementers in the BMCs, NGOs or project beneficiaries is relatively weak265. There appears to be little reflection on drawing out significant messages for the broader group of stakeholders, or on how then to transmit them to the “right” people in the “right” way (knowledge brokerage).
To conclude, evidence on the uptake of evaluation is either confusing or sparse. It is unfortunate that so far no systematic record keeping system has been put into place to track lessons learned or the uptake of recommendations (or actions agreed from management responses). The OIE plays a weak role in brokering the knowledge generated through evaluations to the benefit of external partners and in managing such knowledge. Although the Evaluation Policy specifies the need for “distilling evaluation findings and lessons learned in appropriate formats for targeted audiences both within and outside the CDB” (p.19) such a targeted communication strategy has yet to be developed and budgeted.
Strengthening Evaluation Capacities and Networking From the onset in 2012, the OIE has stressed the importance of developing and strengthening evaluation capacities within the OIE, the CDB and, subject to available resources, in borrowing member countries. Building evaluation capacity in BMCs and the CDB is one of the OIE’s mandated tasks. It has been a priority that figures on the work plan from the beginning (Work Programme and Budget 2012-2104) The idea of developing an internship programme for graduates from the Caribbean region was one idea that was advanced to help build local evaluation resources. However, the capacity-building has primarily been focused on OIE and CDB staff to date. One of the OIE’s two objectives for 2015 therefore was to take up the challenge and “strengthen evaluation capacities and networking” to include reaching out to the BMCs.
265 A broader communication strategy is one of the principles and good standards of the Evaluation Coordination Group and the Evaluation Community more generally.
462
Developing OIE staff capacities
The change from project level to strategic and thematic evaluations does require different evaluative skills and competencies. The MDB Evaluation Pyramid presented below in Figure 3 shows the different types of evaluation and changing resource needs as one ascends the pyramid. Implicit here also is the change in the type of expertise and competencies needed as evaluation aspires to the higher levels.
Consequently for 2015, the OIE set itself the objective of networking and developing working partnerships with regional and international evaluation entities and academic institutions. The rationale was twofold: (1) secure further support and guidance as well as (2) increase its outreach and coverage through joint work and international exposure. Another implicit aim was to benefit from partners’ contacts in the BMCs wherever possible so as to improve data collection and quality.
Figure 3: The MDB Evaluation Pyramid266
The OIE has therefore linked up with Carleton University in Canada and the University of the West Indies, Barbados campus. The OIE was also approached by the Development Bank of South Africa to exchange experiences about setting up an evaluation entity in a “small” development bank. However, its attempt to become a member of the Evaluation Cooperation Group was not successful for reasons beyond its control.
The OIE is to be commended in addressing the issue of staff competencies and professional development more generally. New developments in evaluation as well as new developments in the scope of OIE’s work may necessitate new competencies. For this reason, organisations such as the International Developmental Evaluation Association have recommended that the
266 US Treasury Report to Congress on Evaluation Standards and Practices at the Multilateral Development Banks (September 2014, Annex C).
463
competencies of evaluators and evaluation managers should be periodically reviewed. Several publications now exist on competency requirements and suggestions for the periodic review of staff competencies.267
It is not within this remit to compare and contrast OIE’s competencies with those recommended by international and national agencies. However, what we can say is that the OIE demonstrates great forethought in taking this on board.
Capacity building within CDB
The OIE’s objective also consists of continuing to develop measures for improving the monitoring and self-evaluation side of CDB’s work. OIE’s strategy here is to use the windows of opportunity on offer through some of the training sessions that are being organised by CDB as part of its shift towards MfDR e.g. by Corporate Planning Services and Technical Assistance. For 2016 it is also planned to have the OIE present at the annual staff meeting and Learning Forum.
The OIE also organises some ad hoc training with operations, for example to help understand new tools e.g. for drawing out lessons from self-evaluation reports and, more generally, in helping staff appreciate how evaluation can add value to the organisation’s work. Measures include providing advisory services on demand, and providing training alongside the introduction of new or revised tools.
Capacity building in the BMCs
This is an ambitious task and would require additional investment; from the bi-annual work plans to be effective. A modest attempt has been made in 2015; from what we understand, the OIE has joined together with the Carleton University and the University of the West Indies, using their networks in some of the BMCs, to try to develop this aspect.
To conclude, we cannot comment on the quality or reaction to such training, but can commend the OIE for making capacity building one of its priority objectives. From both the Policy and the documents we reviewed, we note that capacity building was always seen to be an important aspect of OIE’s work, but hitherto has received little strategic focus. But the resources currently available to the OIE will limit the scope of such work in the BMCs, which in turn, will continue to hinder the production of sound evidence for the OIE’s evaluations.
Adequacy of the OIE’s human and financial resources to support its work
OIE’s Human Resources;
The OIE is has a staff of 5; the head, 1 senior evaluation officer and two evaluation managers, plus one administrative assistant. Three of the five were recruited from within the CDB. The limited capacity means that it is not feasible to cover all the types of evaluation activities outlined in the Evaluation Policy. Yet there is some indication from the Board that OIE should embark on impact evaluations at some future stage. An increasing demand for evaluation and for impact evaluations in particular, would run the risk of overstretching the OIE’s capacity to deliver credible and useful evaluations. Moreover, there are many other designated OIE activities that should be recognised as valuable work; the validations, building CDB and BMC evaluation capacity, providing supervision, advice, knowledge management and brokerage as well as managing evaluation contracts, The time needs of dealing with all of these may be underestimated in OIE’s budgets; all are important for assuring best value from evaluation. The Panel is concerned that a demand for “doing” evaluations as well as OIE’s interest in advancing its skills in high-level evaluations may undermine the importance and time needs of other 267 E.g. IDEAS, (2012) Competencies for Development Evaluation Evaluators, Managers and Commissioners, the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice (2010) and the Swiss Evaluation Society’s Evaluation Managers Competencies Framework (2014)
464
essential tasks.
Limited and unpredictable resources for independent evaluations
The OIE is funded from the general administrative budget and represents approx 2.5% of the total. Whilst this is seemingly a higher proportion than other MDBs, in real terms it is quite limited. 75% of OIE budget is for staff salaries leaving US$190,000 in 2015 for external consultants and other expenses.
CDB’s donors do not appear to specify a budget for monitoring and evaluation activities. This means that on the one hand, there is no clear external budgetary recognition of the operations’ self-evaluation work or of OIE’s time in the validation process, and on the other, that whilst donors expect to receive reports from independent evaluations, the expectation is not backed by making this clear when allocating funds.
Resources available to the OIE for hiring external consultants has dropped from $350,000 in the revised 2014 budget to US$120,000 in the 2015 indicative budget. The OIE estimates that for high-level evaluations, the cost for external consultants is between US$90,00 - $350,000. (The SDF &6&7 evaluation cost US$255,000). According to the Panel’s experience, this is a sound estimate. With one less staff during 2014-2015 coupled with OIE’s focus on dealing with the backlog of self-evaluations amongst other priorities, it was unable to execute some of the evaluations during the annual budget period. Hence, the budget was reduced for the consequent years but has proven to be insufficient to fund the OIE Work Programme. The OIE has therefore needed to turn to the only alternative source available at present, the SDF fund. But the SDF funding rules apply to specific countries and themes, which obviously restrict the OIE’s choice of evaluation subjects and themes. Since the SDF does not allow for OIE recurring costs such as staff travel, the SDF evaluations have to be outsourced. As presented in Figure 1 above, the approval process is inefficient and causes delays. The Panel learned that additional funds, for example for specific studies, could be secured from within the administrative budget during the year on condition that the request was based on sound arguments.
Whilst the Panel appreciates full well that the Bank is operating within a zero growth framework, the reviewers were surprised to learn that OIE funding is not sufficiently secured in line with its priorities and work plan. The need to seek alternative funding for individual studies does not allow for any flexibility and undermines the OIE’s independent judgment of what needs to be done.
To conclude: the OIE is inadequately resourced to meet the expectations outlined in the CDB’s Evaluation Policy. However, the Panel recognises that CDB itself has budgetary restrictions. But current arrangements to secure extra funding are complicated, inefficient and limit the OIE’s ability to exercise autonomy in the selection of its evaluation studies. Moreover, OIE budgets significantly underestimate the time needs of managing evaluations and other evaluation activities.
Self- and independent evaluationSelf-evaluations cover public sector investment, lending and technical assistance, policy based loans, and country strategy programmes. Both types of evaluation are important as they are at the very heart of the evaluation function; they are said to be the building blocks for the more strategic evaluations that the OIE is now undertaking.
The Evaluation Coordination Group recommends that the self-evaluations be carried out by the relevant operations department and in turn, reviewed and validated by the organisation’s independent evaluation office. The CDB’s Evaluation Policy therefore talks of “validating all self-evaluations” as being one of OIE’s essential oversight tasks.
465
Within CDB, the self-evaluations should provide management with performance assessments and thereby serve an accountability function to the CDB and Board. To support the process, the OIE provides operations with manuals and checklists for guidance. Once a self-evaluation report is to hand, it is given over to the OIE for the validation of its technical quality and credibility.268
However, in the CDB case, there are well-documented issues that have affected the quality and timeliness of the self-evaluations on the one hand, and therefore the quality of the foundation on which to build the independent evaluations. Paucity of documentation within CDB, paucity of data collected and available in the Borrowing Member Countries (BMCs), time delays in producing completion reports and in turn, having them validated by the OIE - all such issues were systematically raised during interviews and in some of the independent evaluation reports. There appears to be little incentive to complete self-evaluations in a timelier manner.
Generally speaking, many of the monitoring data problems appear to be due to a lack of management oversight. For example, with the introduction of results-based management, the logic frame and monitoring and data needs are systematically being built into intervention design. However, the BMCs are not delivering the data as contractually agreed at the outset. Incentives to support any significant change towards building a results-based culture seem to be weak and sanctions seem to be rarely enforced when the supply of data is lacking or lengthy delays to the projects occur. Although we can appreciate the complexities of trying to enforce monitoring compliance, this means that often, project deadlines have had to be extended, data gaps are not being satisfactorily dealt with and in turn, there has been a void in the quality and quantity of available evidence for the CDB’s self-assessment of project performance. For some time, this lack of oversight has been tolerated. Part of the problem is the low priority accorded to completing the self-evaluation reports by operations, coupled with the absence of any focal point within senior management to drive the process and deal with the problems.
No record is kept of how the self-evaluation results are actually used. They do not appear on the CDB website, but we were told that the findings are integrated into the following project designs. Hence we are somewhat unclear as to the utility of these reports at present. The situation is exacerbated by a rather confused image of evaluation: some operations staff consider OIE’s input (through validations or independent evaluations) to be sometimes over-critical, regulatory and adding little value; it is a threat rather than an opportunity for learning. Yet at the same time, evaluation is recognized as an integral part of result-based management.
According to the Evaluation Policy (p.15) “The President, with the support of the Advisory Management Team, is accountable for encouraging and providing an environment where evaluation adds value to the overall management of CDB’s activities and fosters a culture of critical analysis and learning”. But, in the CDB a learning culture appears to be still in its infancy. The leadership role as expressed in the Evaluation Policy is underdeveloped.
Some managers however seem to start changing the status quo. For example a revised and simplified template for producing project completion reports is being considered, and mid-term project reviews are expected to be more stringent in looking at monitoring plans and practices and tying disbursements to performance. In some cases we also learned of incentives being introduced to encourage project managers to complete their reports in a timelier manner. But much remains to be done and, since the OIE is no longer responsible for monitoring and project evaluations, there is a void that needs to be filled. It is up to line managers to drive this work forward.
To conclude, it is fair to say that in view of a number of “frustrations” between the OIE and operations, which are largely to do with delays in exchanging comments on the various reports as well as the paucity and/or lack of monitoring data, the added value that evaluation might offer to the operations area is ill recognized. Moreover, the link between self-evaluation as the
268 According to the Evaluation Policy, OIE should validate all PCRs and CCRs but due to the backlog of reports and the delay in completing them (sometimes years later) since October 2015, the OIE has secured OAC agreement to validate a maximum of 6 per year, which are selected in consultation with the OAC.
466
building blocks for the independent evaluation is not apparent. Thus there is little incentive or management focus to drive any change to current practices. In other words, there is a lack of leadership to advanced a learning environment in which evaluation can play a major part.
467
Part Three: General Conclusions and RecommendationsTo conclude, with regard to the Evaluation Policy and OIE’s independence, our Review finds that over the past few years, the CDB has succeeded in establishing an independent evaluation office that is credible and respected. It reports to a Board Committee and is thus organisationally independent from CDB management. Its work is grounded on an Evaluation Policy agreed by the Board and the CDB that reflects internationally recognised principles and good practices. The Policy sets out a broad scope of responsibility for the OIE which, however, seems over-ambitious given current resource constraints. The OIE clearly has both an accountability and a learning function; the latter should support the development of an organisational learning culture. (So far any monitoring the uptake of recommendations and key lessons has not been systematically recorded.) In general, on the issues of independence, we can conclude that the OIE meets the criteria for organisational and behavioural independence and is protected to a certain degree from external or contextual influences.
However, as the independent Advisory Committee for Development Impact has said, “independent evaluation needs to have clout……credibility of evaluation hinges on public perceptions as well as on reality.”269
We are therefore highlighting a few potential threats even though there is no evidence to suggest they are in any way real at present. But it would be in the OIE and CDB’s interest to have these clarified sooner rather than later. For instance,
any delays incurred in reporting self and independent evaluation results to the Board could be interpreted as operational interference.
Similarly, there is no agreed process to deal with any conflict of interests between the OIE and management in reporting results as it is expected that any disagreements will be reported in the management response.
Another possible threat is the lack of complete autonomy that the Head of the OIE has over staff; recruitment, termination, continuation, and professional development. The Policy is not sufficient clear about who has the final word in the case of disagreement.
And finally, on resources, our Review accepts the limited funds available to the CDB and the fact that the OIE’s budget is not independent but operates within the Bank’s budgetary limitations. Nevertheless, we feel that some more flexible arrangements could be devised that would allow for a less restrictive and timelier access to funds.
With regard to governance, our Review has highlighted the difficulties the OAC faces in not receiving the background papers for its meetings in sufficient time to be able to do them justice. Moreover these documents tend to be very lengthy and not necessarily “reader friendly”. The OAC’s oversight responsibility is likely to be weakened and we can already see some indication of this. For instance, requests for systematic follow-up on management actions resulting from evaluation findings have not been answered. Neither is there a systematic item for this on the OAC agenda so that such requests can easily be passed over and forgotten. The broadened responsibilities now given to the OAC also mean that there are many competing entities trying to secure the OAC’s attention. There is now provision for the OAC to call on consultants for help, which we feel may help strengthen the OAC in its oversight responsibilities.
Furthermore, in its capacity as members of the Board, the OAC should stress the urgency of developing evaluation and monitoring capacity in the BMCs since this gap is having a direct impact on OIE and CDB evaluations.
With regard to the OIE’s performance, we have to respond to the questions raised in this Review’s Terms of Reference, which basically mean answering two main questions: Is the OIE doing the right thing? And is it doing it in the right way?
269 Picciotto, R. (2008) Evaluation Independence at DFID; An independent Assessment prepared for the Independent Advisory Committee for Development Impact (IADCI) (p. 4).
468
There is no doubt that the decision to establish a credible, independent evaluation function in the CDB is the “right thing” to do; effective and useful evaluation and oversight activities can assess development effectiveness, hold the organisation accountable for results, and improve operational performance.”270 It is also a policy of the MDBs to have such a function and the CDB has now aligned itself with international standards and practice. 271 The question now therefore is the following; is the OIE going about it in the right way?
The OIE has taken the “right” steps to improve the engagement and interest of the OAC and CDB senior management from selecting the topics for its evaluations through to finalising the conclusions and recommendations in a collaborative spirit. It falls short of taking the messages emerging from the studies to “outsiders” such as those responsible for implementing CDB interventions in the BMCs.
In its oversight role, we feel that the OIE has paid insufficient attention to the actual utilisation of evaluation; it is beyond its responsibility to see that action is taken, but it is certainly within its remit to record how, and how well the lessons drawn have been taken up and used. With regard to its oversight of the self-evaluations (the validation process), the OIE has attempted to improve dialogue with the operations departments and, demonstrate the dual function of oversight and learning. It is now emphasising the learning aspect by providing tools and guidance on how to draw out lessons and integrate them into future planning. More recently it has sought ways to provide more formalised training on evaluation by working with the corporate planning services and technical assistance department to develop courses that show how, where and when evaluation plays its part within the MfDR framework.
However, one of the challenges in evaluation management is balancing its independence with facilitating buy-in and ownership at the same time. It is a fine line to walk and depends to a large degree on the climate between management and the head and staff of the independent evaluation unit in defining the tone of the collaboration. In practical terms, for the CDB this means defining the role of the OIE in relation to the self-evaluations performed by the Projects and Economics Departments. The change from the EOV to the OIE made this role change quite clear; the OIE no longer has responsibility for project monitoring and planning data needs together with the operational departments. On the other hand, to improve understanding and learning, there needs to be an interface between evaluation and management. At present, OIE’s dual role, that is advisory role in relation to operations and its strategic role towards the OAC and senior management, has not been satisfactorily resolved. The operational staff still do not appear to see any urgency in producing their completion reports or appreciate what lessons might be drawn from such reflection. The OIE is doing its best to support “learning” whilst at the same time, keeping an arm’s length. The greatest challenge the OIE faces in its new capacity is the slow development of an organisational learning and evaluation culture.
A Learning and Evaluation Culture
Evaluation utility depends on the engagement of evaluation users – those who should benefit from the knowledge generated through the studies. Useful evaluation therefore depends to a large degree on the development of an evaluation and learning culture and how well these are embedded in the organisation. This means that the organisation recognises and appreciates evaluation’s role and the functions it can have, particularly for helping understand what it is achieving and where and how improvements can be made. In short, the added value that evaluation can bring to the organisation is its ability to draw out the important lessons that can help improve the organisation’s performance.
However, whilst CDB senior management shows all the signs of embracing evaluation as an important strategic tool, there still appears to be some apprehension about receiving criticism
270 CDB (2011) Evaluation Policy (p.2)271
469
however constructive this might be. The OAC has already affirmed its interest in learning what can be” put right the next time around.” In considering accountability, the committee is asking for a more strategic approach to learning and sharing knowledge based on evidence. The CDB also shares the development goals of other MDBs, that is « to end extreme poverty and promote shared prosperity. » This means looking for new forms of problem-solving and for ways to create a “development solutions culture.” Hence there is an interest in learning from experience and exchanging knowledge about what works. This implies balancing accountability and learning; making sure they are not seen as opposites, but as compatible entities. This greater emphasis on learning requires a reframing of CDB’s thinking and dealing with the constructive criticism that evaluation can offer.
Weak evaluation culture 27. While some stakeholders seem keen on evaluation, the overall evaluation culture in UNRWA is weak. There are several aspects to it.
28. First, many of the interviewees stressed that UNRWA has a weak learning culture. The weak learning culture stems from a number of factors. One reason given is related to the cultural virtue of oral communication. This makes conveying documented experiences challenging. Another reason is language. A majority of UNRWA’s national staff is not fluent in English (evaluation reports are mostly in English). Furthermore, criticism – even if constructive - is – according to some interviewees - mainly perceived as a threat and not as an opportunity. Finally, learning is also affected by a very basic constraint – lack of time.
29. Second, there is a weak knowledge management system to systematically collect and share experience and lessons learned in UNRWA. UNRWA communities of practices do not exist. Several interviewees mentioned the use of knowledge networks outside of UNRWA, i.e. communities of practices managed by other agencies. Also, accessing evaluation reports is not easy. The UNRWA website on the Internet does not provide access to evaluation reports. While the Agency’s Intranet has a site for evaluation reports, it is not a complete depository and the Evaluation Division does not exactly know how many decentralized evaluations are being produced. In addition, there are only few evaluation plans at the level of field offices or departments.
30. Third, the Panel found that decentralized evaluations are - at least partly - perceived as donor-driven accountability instruments rather than as learning tools. In that sense, evaluations are managed as bureaucratic requirements thereby weakening the learning dimension.
31. Finally, the sensitive political context in which UNRWA operates may also discourage a strong evaluation culture as evaluative evidence can sometimes be overridden by political considerations.14 The Panel was repeatedly told that given the political context, any change is a challenge.
14 An example mentioned to the Panel was the evaluation of the Qalqilya Hospital (2013) which concluded that the Hospital should be closed. However, for
political
470
Part Three: Recommendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
471
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
link between self evaluations, validations and independent evaluation not clear now between self evaluations and QaE documents – so one wonders a bit what all the effort is for on their side. This is a real issue. They seem to do a lot of interesting and not too bad things but there is a lack of coherence. (but then I have only seen the documents, not done any interviews to get a broader picture).
This is something the EIB evaluation unit was criticised for in the past too. Since, we have started to include also “younger” projects in our samples (sometimes still on-going). We also redo the portfolio analysis right before the finalisation of the report to see if things have changed. and of course the services can in their response indicate if indeed things have changed over time.
Recommendations for improving process for study approval and funding
Give recommendations on priorities for OIE work
. Funding preferably from the administrative budget. Unused monies could then be released in the annual budgetary reviews, but this should have no affect on the budget for consequent years. SDF funding at a leveit is surprised to find that a Board approved OIE work programme and budget is inadequate; either the proposed budget per work programme
472
mendations
(Here is a list of some possible ones - to be discussed and developed within Review Panel initially and then proposed to discuss together with the OIE)
OAC’s oversight responsibility needs to be strengthened (possibly) Review Evaluation Policy to redress gaps OIE to develop 5 year strategy providing step-by-step work programme, budget and
timeframe for implementing Evaluation Policy, Stronger leadership from President to provide conducive climate for promoting added
value of evaluation to overall management and fostering a culture of critical analysis and learning. Stronger support from Advisory Management Team for evaluation function by emphasising the accountability function of CDB managers for performance results and for devising incentive schemes to support accountability function.
Set up committees to accompany study specific evaluations as a means of reinforcing ownership (advisory groups)
OIE could contribute to developing a learning culture within the CDB by adopting the role of critical friend in its dealings with the CDB operations departments and the CDB more generally. Valuable insight can be gained by having a focused conversation with your client about aspects of the evaluation that went well and those that did not go so well. Some meetings might include all relevant stakeholders in addition to the main client. This feedback can then inform a subsequent internal project team debrief (e.g., identifying internal professional development or process improvement needs). Input from all perspectives can advise planning for future projects. We then circulate a summary of these discussions so that everyone can use them for their own internal or external quality improvement purposes.
Rather than asking ‘what went wrong?,’ can talk about “what surprised us, what we’d do differently, what did we expect to happen that didn’t, and what did we not expect to happen that did”. Better means of getting at the negative aspects without placing blame.
OIE to train up and engage “champions” within CDB operations departments to help demonstrate evaluation utility and provide “on the job” training in self-evaluation to colleagues Could also set up quality control group like Picciotto suggested for DfID to carry out and develop the work of Quality Assessment and particularly, Quality at Entry monitoring.
OIE should be given the resources to build M&E capacity in BMCs as a priority, possibly in partnership with other MDBs, development organisations and the countries themselves
Possibly criticise over emphasis on using the five DAC criteria, which have been in use now for more than 15 years without going through any major revisions. Given their Importance and level of influence in the field, it is pertinent that independent professionals take a critical look at them, especially since many scholars and practitioners consider that the quality of evaluations in development aid has been quite disappointing (http://evaluation.wmich.edu/jmde/ Evaluation in International Development Journal of MultiDisciplinary Evaluation, Volume 5, Number 9 ISSN 1556-8180 March 2008 41 The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement Thomaz Chianca Independent)
Regarding the Validation of self-evaluations: It is recommended that a leaner format be developed for the PCVRs, repeating less the content of the original PCR text and focusing
473
on the validity of PCR against a limited number of key issues to be assessed for each PCR. The PCVRs are also highly redacted which may not be needed for such type of document – a more tabular form with more succinct statements could lead to a leaner process by which those PCVRs are produced, all by not losing in usefulness. The “PCR checklist would be a good starting point for this
The Panel however encourages creating such a Quality control unit the role of which cannot be fulfilled by OIE, as it lies outside the scope and present capacity of OIE – even though OIE could have an advisory/methodological role.
474
APPENDICES
Appendix I - The External Review Mandate – Terms of Reference and Approach Paper
Appendix II -Review Approach, Data collection and Analysis, and Limitations
Appendix III – Overview of OIE Evaluation Practice
Appendix IV - List of Persons Interviewed
Appendix V - List of Documents Reviewed
Appendix VI- List of Topics used to guide interviews with members of CDB Board of Directors
Appendix VII - List of Topics used to guide interviews with CDB staff
475
Appendix III – Overview of OIE Evaluation Practice (prepared by the OIE in response to Reviewer’s request)
Caribbean Development Bank, Office of Independent Evaluation - OIE
Category Response
Percentage of projects subject to project (self-) evaluation
100% - Project Completion Reports (PCR)
Percentage of projects subject to validation by OIE
Approximately 40-50%
About 15 projects exit portfolio annually. Evaluation Policy calls for all PCR to be validated. However, OIE resources insufficient. Validation process reviewed in 2014. Now OAC (Board committee) selects a sample of 6-8 PCR for validation each year.
Percentage/number of projects subject to in-depth review by OIE
None – unless specifically requested by OAC
Due to limited resources, focus of OIE evaluation work programme is on PCR validations and high-level evaluations – including country strategy and programme evaluations (CSPE).
Number of high-level evaluations conducted by OIE (e.g. sector, thematic, geographic)
1-2 per year since 2011
Plan is 2-4 per year from 2016. This would include CSPE (1st planned for Q1 2016: Haiti)
Number of project impact evaluations conducted by OIE
None
OIE includes “impact questions” in high-level evaluations.
Number of project impact evaluations conducted by Bank staff or other non-OIE staff
OIE is not aware of any impact evaluation conducted by the Bank.
However, OIE provides technical support to the Basic Needs Trust Fund (BNTF) in its design of an M&E framework that entails impact evaluations.
Budget In USD mn: 0.78 in 2015; 0.82 in 2016. This is equivalent to about 2.5% of total CDB Administrative Budget.
75% of the budget is for Staff salaries (4 Professionals, 1 Support staff), leaving around USD 190,000 (in 2015) for other expenses, including consultants e.g. for external evaluations. Additional funding is accessed via the Special Development Fund (SDF). This varies according to type and scope of the evaluation, e.g. the ongoing SDF 6/7 Evaluation is SDF funded at USD 255,000.
Budget determined by Board, not separate from administrative budget.
SDF funding for evaluations is considered separately and subject to Bank internal approval process. SDF funding cannot be used to cover OIE expenses such as staff time or travel. Country eligibility for SDF funding is also a consideration. OIE expressed concerns about this funding track in respect to predictability, independence and eligibility limitations.
Head of OIE reports to Board, with administrative link to the President
Terms of appointment for Head 5 year term, renewable once. Appointed by the President with the agreement of the Board.
476
Right of Return for Head Not eligible for other staff positions.
Consultants as proportions of OIE budget
2015: 19% (USD 145,000)
Plus SDF funding. SDF funded evaluations are outsourced.
Last external evaluation (or peer review) of OIE
No external evaluation, though a review of the function was done in 2011, leading to the Evaluation Policy.
OIE External Review completed in April, 2016
Departments or special programmes supporting impact evaluation
None
477
Appendix IV – List of Persons Interviewed
Name Function relative to OIE Type interview
Mrs. Colleen Wainwright Member CDB Board of Directors (UK)
Face to face
Mrs. Cherianne Clarke Alternate MemberCDB Board of Directors (UK)
Face to face
Mrs. Jean McCardle Member CDB Board of Directors (Canada)
Face to face
Dr. Louis Woodroofe MemberCDB Board of Directors (Barbados)
Mr. A: de Brigard Former Member CDB Board of Directors
Skype interview
Mr. H. Illi Fromer Member CDB Board ofDirectors
Telephone interview
Mrs. Claudia Reyes Nieto Member CDB Board of Directors
Telephone interview
Mr. Bu Yu alternate DirectorCDB Board of Directors
Face to face
Mr. Michael Schroll(Barbados)
Head OIE
series of interviews viaSkype and face-to-face
Mr. Mark Clayton OIE Senior Evaluation Officer Focus GroupMrs. Egene Baccus Latchman OIE Evaluation OfficerMr. Everton Clinton OIE Evaluation OfficerMrs. Valerie Pilgrim OIE Evaluation Officer
Dr. Justin Ram CDB Director Economics Department
Face to face
Mr. Ian Durant CDB Deputy Director Economics Dept Face to faceDr. Wm Warren Smith CDB President
Joint interviewFace to face
Mrs. Yvette Lemonias-Seale CDB Vice President Corporate Services & Bank Secretariat
Mr. Denis Bergevin CDB Deputy DirectorInternal Audit
Face to face
Mr. Edward Greene CDB Division Chief, Technical Cooperation Division
Face to face
Mrs. Monica La Bennett CDB Deputy Director Corporate Planning Face to faceMrs. Patricia McKenzie CDB Vice President Operations Face to faceMs. Deidre Clarendon CDB Division Chief
Social Sector DivisionFace to face
Mrs. Cheryl Dixon CDB Co-ordinator, Environmental Sustainability Unit
Focus group
Mrs. Denise Noel- Debique CDB Gender Equality Advisor Mrs. Tessa Williams-Robertson CDB Head Renewable EnergyMrs. Klao Bell-Lewis CDB Head Corporate Communications Face to faceMr. Daniel Best CDB Director
Projects DepartmentFace to face
Mr. Carlyle Assue CDB Director Face to face
478
Finance Department
479
Appendix VI - Interview Guide: Members of CDB Board of Directors
Below are a list of themes that I should like to raise with you based on your experience and knowledge of the CDB’s independent evaluation function (Office of Independent
Evaluation).
In each case, I should be grateful if you could illustrate your responses with examples or help this Review by, wherever possible, sending me (or telling me where I can find) any
documents that could support your responses.
This guide is being sent to you in advance to help prepare our meeting. However, our interview will be conducted more in the style of a conversation. The following sub-questions will be used to GUIDE the interview. Please feel encouraged to raise any
additional issues that you feel we should take into account
On the governance and Independence of CDB’s evaluation functionWhat mechanisms are there in place to support its independence?
How satisfactory are the current arrangements in your opinion?
How is the balance between independence and the need for interaction with line management dealt with by the system? For example, what mechanisms exist to ensure that the OIE is kept up to date with decisions, policy / programme changes, other contextual changes etc that could have an affect on OIE evaluation studies / evaluation planning?
On the OIE’s Evaluation PolicyThe CDB’s Evaluation Policy was established in 2011. To what degree do you feel it is adequate? Still relevant?
What suggestions do you have for any improvements?
In your opinion, how adequate is the current quality assurance system for over viewing the evaluation function?
On the quality and credibility of evaluation studiesTo what degree do you believe the reports are fair and impartial?
Do you consider them to be of good quality? Are they credible?
Are you adequately consulted/involved on evaluations of interest to you?
On the relevance and usefulness of evaluations How well does the OIE engage with you / your committee during the preparation, implementation and reporting of an evaluation study to assure that it will be useful to the CDB?
How are the priorities set for the independent evaluations? What criteria are used? Are you satisfied with the current procedure?
480
When OIE evaluation studies are outsourced to external consultants, what criteria are used to make this decision?
How are the priorities for the OIE’s 3.year rolling work plan agreed? In your opinion, is the current plan adequate in terms of coverage and diversity?
In your opinion, do the evaluations address important and pressing programs and issues?
To what extent do you feel that the OIE’s evaluations integrate the cross-cutting theme such as gender, energy efficiency/renewable energy, climate change? What improvements might be made and how?
On the dissemination and uptake of evaluation findings and recommendationsTo what extent do you feel that evaluation findings are communicated to the CDB and its stakeholders in a
a) useful, b) constructive andc) timely manner?
Are evaluation recommendations useful? Realistic?
What mechanisms are in place to assure that evaluation results are taken into account in decision making and planning? What improvements do you feel could be made?
How have you used the findings from any evaluations? Examples?
To what degree do you feel that evaluation contributes to institutional learning? And what about to institutional accountability? Any examples?
What mechanisms are in place to ensure that knowledge from evaluation is accessible toCDB staff and other relevant stakeholders? Are the current arrangements satisfactory?
How satisfied are you with current arrangements? What expectations do you have for the future?
On resourcesHow is the OIE resourced financially and is this satisfactory?
What about the OIE staff, are all the important areas of expertise represented in the team?
On this Review of the Office of Independent EvaluationWhat are your expectations? What are you particularly hoping to learn from it?
Thank you very much for your cooperation and input
481
Appendix VII : Interview Pro-Forma – CDB Staff membersThis presents a list of the topics raised during interviews. It was used to guide the open-ended
discussion – this means that the sequence and exact wording of the questions may not necessarily have followed in this order or been asked in exactly this way.
Changeover to an Independent Evaluation Office? Expectations? Advantages and disadvantages??
Satisfaction with working relations between operations and the OIE from your perspective?
Process of dealing with the PCRs and CCRs? Advantages and limitations?
Quality and credibility of the validation process?
How are the self-evaluation reports used?
Credibility and Quality of OIE’s evaluation reports
Communication of self and OIE independent evaluations? To whom, in what way? Possible improvements?
482
Recommended