ECD Report Ver 3.0 - ECDG

Seminar on Evaluation Capacity Development

(ECD Seminar)

SEMINAR PROCEEDINGS REPORT

Prepared by:

Conference Secretariat MES Conference 2012

MALAYSIAN EVALUATION SOCIETY

MES EVALUATION CONFERENCE 2012 “Integrated Approach for Evaluation & Results Based

Management”

Kuala Lumpur :

10-‐14 Sept., 2012

Centre for Development and Research in Evaluation (CeDRE) International Internet: www.cedre.org.my Email: [email protected]

Abbreviations

ADB -‐ Asian Development Bank ECD -‐ Evaluation Capacity Development ECDG -‐ Evaluation Capacity Development Group IEU -‐ Independent Evaluation Unit INGO -‐ International Non-‐Governmental Organisations IOCE -‐ International Organization for Cooperation in Evaluation IEU -‐ Independent Evaluation Units IRBM -‐ Integrated Results Based Management M&E -‐ Monitoring and Evaluation MES -‐ Malaysian Evaluation Society MfDR -‐ Managing for Development Results NGO -‐ Non-‐Governmental Organisations ODA -‐ Official Development Assistance OECD -‐ Organisation for Economic Cooperation and Development RBM -‐ Results Based Management SEDP -‐ Socio-‐Economic Development Plan TCPR -‐ Triennial Comprehensive Policy Review of the United Nations TNA -‐ Training Need Assessment UN -‐ United Nations UNDP -‐ United Nations Development Programme UNICEF -‐ United Nations International Children’s Fund UNEDAP -‐ United Nations Evaluation Development Group for Asia and the

Pacific (UNEDAP) UNESCO -‐ United Nations Educational Scientific and Cultural Organisation

___________________________________________________________________________ © Sept. 2012 Malaysian Evaluation Society Page 3 of 40

Contents SECTION 1: REPORT ON ECD SEMINAR ........................................................................................................................ 4

1.1 Introduction and Purpose of the Seminar ................................................................................................ 4 1.2 ECD Seminar -‐ Opening ............................................................................................................................. 4 1.3 ECD Concepts, Approach, Application and Implications for Developing Countries – Karen Russon ........ 4 1.4 Issues & Challenges of ECD for Development Partners & Developing Countries: Five Country ECD Study

Findings & Lessons Learnt ........................................................................................................................ 5 1.5 Evaluation Capacity Development: Challenges & Issues from Development Aid Partner Perspective:

UNICEF Experience ................................................................................................................................... 5 1.6 Evaluation Capacity Development: Challenges & Issues from Development Aid Partner Perspective:

UNDP Experience ..................................................................................................................................... 6 1.7 Evaluation Capacity Development: Challenges & Issues from Development Aid Partner Perspective:

World Bank Experience ............................................................................................................................ 6 1.8 Findings of MES-‐ECDG ECD Survey ........................................................................................................... 7 1.9 Group Discussions & Findings ................................................................................................................... 9

SECTION 2: ANNEXURES ............................................................................................................................................ 12

Annexure 1: Participant List ECD Seminar .................................................................................................... 13 Annexure 2: ECD Seminar Agenda ................................................................................................................ 15 Annexure 3: Opening Address Karen Russon ................................................................................................ 16 Annexure 4: Presentation of Findings and lessons learnt: ECD Study in Asia and Pacific by UNEDAP ......... 18 Annexure 5: Evaluation Capacity Development: Challenges & Issues from Development Aid Partner

Perspective: UNICEF Experience ............................................................................................................ 22 Annexure 6: Evaluation Capacity Development: Challenges & Issues from Development Aid Partner

Perspective: UNDP Experience ............................................................................................................... 25 Annexure 7: Evaluation Capacity Development: Challenges & Issues from Development Aid Partner

Perspective: World Bank Experience ..................................................................................................... 31 Annexure 8 ECD Survey Questionnaire ........................................................................................................ 35 Annexure 9: Presentation on MES-‐ECDG ECD Survey ................................................................................... 36

Malaysian Evaluation Society MESConf2012 ECD Seminar Report


SECTION 1: REPORT ON ECD SEMINAR 1.1 Introduction and Purpose of the Seminar This Report presents the proceedings of the ECD Seminar held on 14th September 2012, as a follow-‐up to the MES Conference held in Kuala Lumpur, Malaysia during 10 – 13 September 2012. The Purpose of the Evaluation Capacity Development Seminar (ECD Seminar) was to identify strategic issues and options for Groups and Organisations engaged in Evaluation for their future engagement in national evaluation capacity development in Asia and Pacific Region as well as in other regions such as Africa. The ultimate purpose was to build on the present status of the countries and suggest recommendations to identify key actions aimed at reinforcing national evaluation capacities and country led M&E systems. The ECD Seminar was jointly hosted by the Evaluation Capacity Development Group (ECDG) and the Malaysian Evaluation Society with financial support from ECDG. ECDG worked closely with the MES in designing and organising the seminar as part of their common agenda to review and address key issues surrounding the ECD initiative both in developing countries as well as among major donors working in the area of ECD. The ECD Seminar is seen as an important event as it had representation from about 29 countries and major development partners. 1.2 ECD Seminar -‐ Opening The Evaluation Capacity Building Seminar (ECD Seminar) was held on 14th September 2012 and attended by many evaluation practitioners from Asia, Pacific, Oceania, Europe and Africa. The full list of participants at the ECD Seminar is as in Annexure 1. Dr. A. Rasappan co-‐chaired the Seminar with Ms. Karen Russon from ECDG. He welcomed all the participants for the Seminar and briefed participants on the purpose and agenda of the seminar. The Agenda for the Seminar was as follows.

• Opening and welcome address – Dr. A Rasappan, Co-‐Chairman, ECD Seminar • Presentation on Introduction to ECD: Concept, Approach, Application & Implications for Developing

Countries -‐ Karen Russon, ECDG • Issues & Challenges of ECD for Development Partners & Developing Countries: Five Country ECD

Study Findings & Lessons Learnt -‐ Ada Ocampo (UNEDAP Co-‐Chair)/Aru Rasappan, CeDRE International

• Presentations on Evaluation Capacity Development: Challenges & Issues from Development Aid Partner Perspective -‐ Marco Segone (IOCE), Indran Naidoo (UNDP) and Susan Stout (World Bank)

• Challenges and Issues in Designing & Implementing ECD (country perspectives): Presentation of ECD Survey Findings – P C K Abeykoon, Malaysian Evaluation Society

• Group Breakout Discussions and Presentations (four groups) on Implications & Challenges of ECD for Developing Countries, Alternative Strategies for Developing and Institutionalising ECD for Developing Countries and Recommendations for Next Steps

• Panel Discussion and Closure • Vote of Thanks – Dr. A. Rasappan, Co-‐Chairman, ECD Seminar

A copy of the agenda of the Seminar is as in Annexure 2. After the opening and welcome address, the Chairman invited Karen Russon of ECDG to make the presentation on introduction to ECD: Concept, Approach, Applications and Implications for Developing Countries. 1.3 ECD Concepts, Approach, Application and Implications for Developing Countries – Karen Russon Karen Russon, President of ECDG, made a brief presentation on ECD by way of introduction to its background and relevant aspects. Karen noted important tools that emerged from the International Workshop Agreement on ECD to help guide ECD initiatives. One is a framework for describing an ECD intervention and engaging stakeholders in the process. The other is a set of evaluation principles comprising: ownership, relevance, integration and usefulness. Both are available on the ECDG website. Karen encouraged Seminar participants



to address challenges by looking for positive steps and applications. Karen’s presentation is found in Annexure 3. Karen’s presentation was followed by paper presentations from a number of presenters from various organizations including UNEDAP, World Bank, UNICEF, UNDP, as well as the findings from a brief survey on ECD carried out by the ECDG and MES. The presentations and key issues discussed are presented in the following sections. 1.4 Issues & Challenges of ECD for Development Partners & Developing Countries: Five Country ECD Study Findings & Lessons Learnt The UNEDAP has recently undertaken a study on ECD covering five countries in Asia Pacific (Malaysia, India, Philippines, Vietnam and Sri Lanka). The study findings and lessons learned were presented by Ms. Ada Ocampo, UNEDAP Co-‐Chair. The presentation discussed the following findings and lessons learnt: Findings

• M&E is present in all countries with varied capacities, and mandates and levels of influence; • National policies (except Malaysia) inexistent or under discussion; • Focus on monitoring rather than on evaluation; • Weak M&E systems faced with capacity constraints; • Weak enabling environment with limited leadership allocation of funds and incentives, inadequate or

less effective legislative and regulatory requirements to support the practice and use of evaluation; and

• Resources for evaluation -‐ mostly were from ODA and still project focused. Recommendations

• Strengthening of M&E system -‐ integrated approach to ECD, mainstream evaluation into national, sub-‐national structure and the need of building internal evaluation capacity to improve the institutionalization of evaluation;

• Strategies for improved enabling environment; • Encourage and support of a setting up independent evaluation units; and • Support creation /enhancement of national evaluation plans/agenda: what to evaluate and for what

purposes? Options

• More coordinated efforts; • Synergies should be sought – Partnership strategy; • South –south cooperation as a core strategy for ECD; • Work with Academic Institutions; and • Advocacy

The presentation by Ada Ocampo is found in Annexure 4. 1.5 Evaluation Capacity Development: Challenges & Issues from Development Aid Partner Perspective: UNICEF Experience The UNICEF experience was shared at the Seminar by Mr. Marco Segone from UNICEF. Key ECD challenges and issues discussed included the following: Major Issues – There are three major types of issues; a. Conceptual, b. Technical, and c. Political. Conceptual Issues -‐ Shared principles for a conceptual framework on NECD Technical Issues -‐ UN’s capacity and resources to deal with systemic approaches and Coordination (UNEG) Political Issues – These included: long time frame in a results-‐driven context; despite of favorable political statements, it appears the priority for donors/development agencies is its own accountability; shifts in power relationships; perceived risk by partner countries that independent evaluations of donor support may have political and financial consequences; and perceived risk by donors/development agencies to be evaluated by partners countries.



Way Forward – Four broad suggestions were made. They are:

i. Partners countries need to take a stronger position in the Board of UN agencies; ii. Middle income, transition and developing countries cooperation to share good practices and

lessons learned; iii. National evaluation organizations fostering national demand (and supply) for monitoring &

evaluation; and iv. International organizations strengthening national capacities to design and implement national

M&E systems and facilitating South-‐South Cooperation. The presentation by Marco Segone is found in Annexure 5. 1.6 Evaluation Capacity Development: Challenges & Issues from Development Aid Partner Perspective: UNDP Experience Mr. Indran Naidoo of UNDP evaluation office presented UNDP experience in relation to the ECD challenges and issues as a development partner. The highlights of his presentation are as follows: Using evaluation as a tool for capacity development

• Greater emphasis on collaborating with national governments • Drawing on national professional expertise while conducting ADRs., using national expertise to its

potential. • Specific measures to ensure stakeholder participation during the conduct of evaluations at the

national level.

Evaluation systems and quality of evaluations: Lessons from NEC

• National political process and constitutional mandate are key factors that shaped the national evaluation systems.

• Challenges remain in establishing the operational linkages between planning, monitoring and evaluation.

• There has been progress on monitoring the implementation of development programs and policies: nature of evaluations and linking lessons from evaluations to planning remains weak.

• Independent evaluations are important for adequately ascertaining the rationale and assumptions of public policies

• Independence of agency conducting the evaluations was seen as critical • Concerns remain about potential conflict of interest: need to strengthen independence of evaluations

for accountability purposes. • Need for sound technical capacity and adequate funding in order to conduct evaluations. • Systemic factors contributing to the use evaluation. • Mechanisms in place for linking evaluations to policy and practice. • Technical rigor of the evaluations was seen a key factor. There were varied views on what technical

rigour entailed. • Process of conducting evaluation: participation and method credibility. • Organizational and institutional factors and managerial rigor was seen as important in enhancing use. • Simple ways of communicating evaluation findings and conclusions and usable recommendations. • Countries should build their own evaluation standards, linking to existing international standards and

principles, particularly in consultation with all actors, including civil society and other political parties, and rooting them in the national context.

• Strengthen institutional capacity for national use of evaluation under the umbrella of South-‐South and triangular cooperation.

• Longer-‐term initiatives on national evaluation capacity are needed. A copy of the presentation is found in Annexure 6. 1.7 Evaluation Capacity Development: Challenges & Issues from Development Aid Partner Perspective: World Bank Experience



Ms. Susan Stout, the former head of the results management unit of World Bank made a presentation on Challenges on Evaluation Capacity Building (ECD). She stressed that her knowledge on the new development on the area at the World Bank may be limited but highlighted a few challenges for ECD based on her experience. The challenges highlighted by the presentation are as follows.

i. Demand side issues – The World Bank, and others, tend to see evaluation as something the donor agency needs rather than a key component of country capacity. It tends to be more concerned with ‘reporting to our shareholders’ over ensuring countries reporting to their stakeholders. It also focuses on building monitoring and evaluation for specific interventions rather than as a general capacity. There is also a tendency to focus on inputs and intentions, rather than results. E.g. Bank’s “DNA” tends to focus on volume of lending, rather than results of lending. It is also easier to focus on ‘supply of monitoring and evaluation’ over demand for information at national and local levels. This is mostly driven by tendency to focus on accountability to shareholders, rather than on creating citizen value. Other factor that is visible is the focus on production of plans and strategies more than on actual results. E.G. a. Focus on Poverty Reduction Strategies, Multiyear Expenditure Frameworks, b. Key question is too often ‘how much will it cost’ rather than ‘does it work.’

ii. Who owns the issue? – Is the ECD is an issue of evaluators, Public Sector and Public Financial Management Specialists or Sectoral Specialists? It is not one party. All have particular and unique perspectives, which can lead to conflicting and or overwhelming advice. However, under current development regime, incentives strong for academics and think tanks, weak for actual decision makers

iii. Balancing Need for Specific Evaluation vs. Evaluation Systems – Whole-‐of-‐government approaches offer the greatest consistency, reliability and consistent approach to demands for accountability. However, it requires strong leadership and cooperation from Ministry of Finance/Treasury and collaboration with Ministry of Planning and Line Agencies. It may also require willingness to shift from budgeting by organizational unit to program budgeting.

iv. Making the Perfect the Enemy of the Merely Good -‐ In settings where integrated approach appears infeasible, one needs to start within particular organizational units, identify and build on examples of successful evaluations, use participative methods to generate ownership on ease of defining outcome measures and experience with utilization of evaluative evidence.

The presentation made by Susan Stout is found in Annexure 7.

1.8 Findings of MES-‐ECDG ECD Survey A rapid ECD Survey had been undertaken by MES ad ECDG to assess the ECD status in the counties that are participating at the ECD Seminar. The Survey findings were presented by Mr. P C K Abeykoon, a member of the Malaysian Evaluation Society. The survey covered four areas relating to ECD, namely; areas to be focused under ECD, greatest challenges on institutionalising evaluation capacity, ECD activities and ECD aspects. Each of these was ranked with 1 being the lowest and 4 being the highest. There had been 14 participants responded and results were as follows.

i. Areas to be focused by ECD – The main areas suggested by the respondents to be covered by ECD were:

• Sector, policy, thematic evaluation and institutional evaluations capacity • Both government institutions at national level and sub-‐national level and consultancy capacity of

the private sector also need to be developed. Strengthening the evaluation organization by building capacity among members

• Evaluation capacity of the training institutions needs support and strengthening • Key resource persons and experts need to be trained • Transparency, accountability and value for money • Over emphasis on ODA Projects • Expanding network and partners with government and private organizations, locally & overseas • Lack of ability for generating adequate resources



ii. Greatest challenges on evaluation capacity development – Key challenges presented by the respondents of the Survey are listed below.

• Sensitization and awareness of Policy makers appears to be lacking • Feedback link between evaluation and policy making • Political factors • Implementation capacity. Low awareness, capacity and skills among various sectors on the value

of evaluation • Dissemination of evaluations • Demand for evaluation – supply side constraints • The apathy among members and different sectors, as influenced by local culture • Weak empirical tradition in public sector • Limited ability to negotiate with government and donors on how evaluation should be

conducted • Few institutions providing support to evaluation capacity development in a comprehensive

manner • National guideline for evaluation • Legal framework • Commitment of the senior officials is still not adequate though it has improved over the years • Field of Evaluation is still not established as an academic field of studies

iii. ECD Activities -‐ Ranks provided by the respondents in relation to the status of each of the ECD

activities focused by the Survey are tabulated below.

No. Description Rank / No of Respondents 1 2 3 4

1 Policies and programs 0 3 5 4

2 Program management 0 2 5 5

3 Resource allocations, budgeting 1 6 2 3

4 Government control, coordination 1 3 6 2

5 Accountability and transparency 1 1 6 4

6 Participation by civil society 0 1 7 4

iv. ECD Aspects -‐ Ranks provided by the respondents in relation to the status of each of the ECD aspects

focused by the Survey are tabulated below.

No. Description Rank / No of Respondents 1 2 3 4

1 Evaluation Policy 1 4 5 1

2 Evaluation Standards 3 2 6 2

3 Evaluation Guidelines and Procedures 2 2 5 3

4 Evaluation Training for key personnel 0 2 6 4

5 Performance Management Systems in the Ministries 2 4 6 0

6 Performance Measurement & Reporting System 0 3 8 1

7 Budgeting System 3 3 4 2

8 Personnel Performance Management system 3 3 4 2



The graphical presentation of the above two analysis is given below.

ECD Activities ECD Aspects

Highlights of ECD Survey: As per the Survey, main areas to be focused under ECD included the following:

• Policies and Programs • Resource Allocations and Budget • Government Control, Coordination • Evaluation Policy • Evaluation Standards • Evaluation Guidelines and Procedures • Performance Management Systems in Ministries • Budgeting System • Personal Performance Management System

A copy of the survey questionnaire is found in Annexure 8 while the findings presentation is found in Annexure 9. 1.9 Group Discussions & Findings During the breakout session, the Seminar participants were divided into four groups. Each group was required to review and discuss about ECD implications, challenges, alternative strategies and recommendations for next steps. The groups’ discussion findings and recommendations are presented below under each of the four aspects covered during the discussions. a. Implications of ECD The implications of ECD for Developing Countries that were discussed by the participants at their groups are as follows.

i. Need for institutional structure change, policies and planning frameworks ii. A shift from donor-‐led to country-‐led towards self-‐determination iii. Evaluation is perceived as donor-‐driven. No national emphasis on Evaluation. iv. Institutional constraints due to M&E being done by a single unit – more Monitoring than Evaluation

resulting in monitoring being more developed. Evaluation tends to bring out the bad news which program/project managers tend to avoid

v. Building a culture of evaluation with ethics and values for evaluators vi. Need for national association & evaluators to play role in advocacy of E but lacking skills & knowledge

in ECD vii. Promote a common language and change of behaviours of demand users

b. Challenges of ECD There are many challenges identified in relation to the ECD. The key challenges identified are presented below.

i. Administrators, executives, legislators, and civil society many not accept the importance evaluation ii. Evaluation is not budgeted for from the central government iii. No/weak evaluation policies to promote a culture of evaluation



iv. Reporting and utilisation of evaluation findings v. Lack of commitment/political will by leaders/managers. Lack of government support and need for

government champions vi. Balance supply and demand of evaluation. Demand and supply issues. Limited demand on use of

evaluation vii. Difference between monitoring and evaluation viii. Donors sometimes influence ECD in unhelpful ways ix. Country led M&E systems is more than government and integration of various elements in the

country led systems x. Ownership – Ownership by the civil society xi. Credibility of the Evaluation Profession xii. Implementation challenges -‐ Lack of standardization and at same time open to change. Lack of

harmonization of M&E tools among countries, lack of expertise. xiii. Lack of networks, representativeness & sustainability of networks, working together xiv. Unequal level of needs and capacities in conducting evaluation xv. Need for long term assessment /process xvi. Assessment/comparison of countries to understand context c. Alternative Strategies to Address ECD Issues and Challenges The four groups brainstormed on the issues and challenges related to ECD with special attention to the situation in developing countries. After much discussion, the groups came up with a range of alternative strategies to address the ECD issues and challenges. The alternative strategies identified and proposed by the participants are as follows:

i. Increase public awareness -‐ Build a common understanding and importance of evaluation across all stakeholders (public, civil society, and private sector)

ii. Bring evaluation awareness amongst legislators/politicians iii. Have a policies to allow planning and budgeting for evaluation (e.g. Sri Lanka, South Africa) iv. Identify political (e.g. former Prime of Malaysia) champions in each country to influence development

of evaluation culture. Need for champions v. Mainstreaming of Evaluation into the education system at all levels -‐ formal & informal vi. Creating Evaluation as a profession -‐ incentives by government to promote evaluations (e.g.Use of

Incentives, Evaluation AWARDS), more ECD practices in career progression of public employees; creation of specific posts and an institution of ECD. Specialisation in the field

vii. Multi-‐disciplinary evaluation professionals viii. Developing national-‐level ECD standards and principles rather than the subject of Evaluation itself. ix. Developing a common agenda for ECD between donors and partnering countries x. Evaluation reports should be results-‐oriented and made public xi. Culturally relevant, multi-‐dimensional; responsive (needs of civil societies and stakeholders);

harmonized and participatory approaches, systems and methods for ECD xii. Regional polling of resources, regional strategy xiii. Support within countries and foundations xiv. Evaluation in curriculum; high level government evaluation committee xv. Advocacy and lobby (TESA as an inspiration) xvi. Document benefits of doing evaluation (that it can save cost) and do case studies

d. Recommendations for Next Steps: After intensive discussion in their respective groups, the four groups came up with a set of recommendations for next steps for ECD, with particular reference to developing countries. These recommendations have serious implications both for the countries concerned as well as their relevant development partners. The recommendations by the participants as identified by groups are:

i. Have an appropriate national policy/administrative system for evaluation ii. Develop a rating scale for extent of national commitment and use of evaluation iii. Use international bodies to influence national governments to have evaluation policies iv. Build a policy to support more internal-‐driven v. Planning for evaluation should be a normal part of all policies and program planning vi. Supporting and promoting International Evaluation Day



vii. Evaluation to be more evidence-‐based and the process should be to engage and involve all stakeholders

viii. Documentation and dissemination of country-‐level evaluation practices and studies and continuous knowledge sharing & networking

ix. Awareness raising / Advocacy x. Targeted approach xi. South to south Cooperation xii. Communication – Common Language and United line of thinking xiii. Credentialing Programs (Regional Societies & associations) xiv. Use a rolling action plan-‐countries take turns to host conferences, division of tasks xv. Core groups across countries-‐collaborate and share, partnerships among evaluations. associations xvi. Share training and resources-‐exchange ideas and experiences xvii. Training of young evaluation experts xviii. Knowledge management, develop skills & aptitude and work with various groups: CSOs, academe,

government and private industry xix. Documentation of good evaluation case studies-‐success stories-‐to show cost savings with use of

evaluation It is important for appropriate actions to be taken on the issues and recommendations put forth by the groups in the Seminar as they represent the views of key M&E officials from the various countries. The issues raised reflect the practical challenges faced in the countries as well as by donor entities themselves when dealing with ECD initiatives. The recommendations of the Seminar are also generally consistent with the findings and recommendations of the special ECD studies undertaken by the UNEDAP (2011) and the ADB (2009). Due to the importance of the issues and recommendations, both development partners and even more importantly, the developing countries involved should pay serious attention to them and come up with appropriate remedial action plans for the short, medium, and longer term.



SECTION 2: ANNEXURES Annexure 1: List of ECD Seminar Participants

Annexure 2: ECD Seminar Agenda

Annexure 3: Text of Karen Russon’s Message

Annexure 4: Presentation of Findings and lessons learnt: ECD Study in Asia and Pacific by UNEDAP

Annexure 5: Evaluation Capacity Development: Challenges & Issues from Development Aid Partner

Perspective: UNICEF Experience


Perspective: UNDP Experience


Perspective: World Bank Experience

Annexure 8: MES-‐ECDG ECD Survey Questionnaire

Annexure 9: Presentation of MES-‐ECDG ECD Survey findings



Annexure 1: Participant List ECD Seminar Ref Gend Name Country Organization Position

1. M AbdolReza Rajaien Iran Research Center Senior Researcher 2. F Ada Ocampo Thailand UNEDAP/EvalPartners EvalPartners Co-‐Chair 3. M Arunaselam Rasappan Malaysia MES Secretary 4. F Ayse Oguz Turkey DPT Deputy Director 5. F Bonolo Oratile Sefhomo Botswana Botswana National Productivity

Center Research Consultant

6. M Champak Pokhrel Nepal NEN President 7. M Chang Yii Tan Malaysia MES President 8. M Chunly Serey Vichet Cambodia Cambodian Evaluation Network Interim President

9. F Constance Chigwamba Zimbabwe Ministry of Public Service Secretary 10. M Craig Russon Geneva ILO Senior Evaluation Officer 11. M David Kolitagane Fiji MoF Dept Secretary Finance 12. M Dev Ruhee Mauritius Gov't Informatics Center Chairman 13. F Ellyna Chairani Indonesia Min. of Nat' Dev't Planning Consultant 14. M Fatih Cinoğlu Turkey DPT Dep. Director 15. F Florence Tan Malaysia MES Committee Member 16. M Hay Sovuthea Cambodia CEN Secretary 17. M Imarciana Cunamiana Mozambique Ministry of Planning

&Development Senior Official

18. M Indran Naidoo New York Evaluation Office, UNDP Director 19. M Indrasathi Muniandy Malaysia MES Treasurer 20. M Jerome Winston Australia PPSEI Director 21. M John Samy New Zealand Advisor Fiji Director 22. M Kabir Hashim Sri Lanka SLEvA Member of Parliament 23. M Karen Russon Geneva ECDG President 24. M Khairul Islam Bangladesh BEN Independent Consultant 25. F Kunnavathy Murugan Malaysia MES Member 26. F Laura Lou China MoSTE Dep Director 27. M Lim Kheng Joo Malaysia MES Vice President 28. M Marco Segone USA UNICEF/EvalPartners EvalPartners Co-‐Chair 29. M Mbako R. Ramashaba Botswana Botswana National Productivity

Center Public Service Program Officer

30. M Mohammad Ismail Rahimi Afghan Ministry of Economy Director General of Policy and ANDS M&E

31. M Mohd Sakeri Abdul Kadir Msia MoF Deputy Director 32. M Nghiem Ba Hung Vietnam Vietnam M&E Network Interim President

33. F Nguyen Thu Thi Que Vietnam Vietnam M&E Network Interim Secretary

34. F Nilanthi De Silva Bandara Sri Lanka SL Evaluation Association President

35. F Nuno Arnaldo Dos Santos Mozambique Ministry of Planning &Development

Senior Official

36. F Pamornrat Pringsulaka Thailand UNEDAP Regional Monitoring and Evaluation Officer

37. M Pamornrat Pringsulaka Thailand UNEDAP Regional Monitoring and Evaluation Officer

38. M Pindai Sithole Zimbabwe Zimbabwe Evaluation Society Treasurer 39. M Prasantha Abeykoon Sri Lanka MFC Director 40. M Rajagopalan Pillay Malaysia MES Auditor 41. M Richard Barahim Malaysia MoF Senior Assistant Director 42. M Romeo Santos Philippines Philippines M&E Society President 43. M Romeo Santos Philippines PMES President 44. M Ryokichi Hirono Japan Seikei University Professor 45. F Sonal Zaveri India COE Independent Consultant 46. M Subarna Lal Shrestha Nepal NEN Secretary 47. M Subrato Kumar Mondal India Pop Foundation India Independent Consultant 48. F Susan Stout Washington WB Independent Consultant

49. F Susan Tamondong Tunisia IDEAS Vice President



50. F Umi Hanik Indonesia Indonesia Dev’t Eval. Community

Dep. Chair, Communication, Networking & External Affairs

51. F Vijayalaksmi Vadivelu New York Evaluation Office, UNDP Evaluation Specialist Secretariat

52. F Lavania A. Malaysia CeDRE 53. M Dheva M. Malaysia CeDRE 54. F Mahaletchmi S. Malaysia CeDRE 55. M Prasad M. Malaysia CeDRE



Annexure 2: ECD Seminar Agenda

ECD SEMINAR (by invitation)

14 September, 2012

Agenda

TIME ACTIVITY

08:00 - 08:30 Registration, refreshments, networking

08:30 – 08:40 Welcome and Introductions (MES)

08:40 - 08:55 • Introduction to ECD: Concept, Approach, Application & Implications for Developing Countries

Karen Russon, ECDG

09:00 – 09:15

Panel Session (Chair Person: Jerry Winston)

• Issues & Challenges of ECD for Development Partners & Developing Countries: Five Country ECD Study Findings & Lessons Learnt Ada Ocampo (UNEDAP Co-Chair)/Aru Rasappan, CeDRE International

09:15 – 0945

0945 – 1000

1000 – 1015

• Evaluation Capacity Development: Challenges & Issues from Development Aid Partner Perspective

a. Marco Segone (IOCE) b. Indran Naidoo (UNDP) c. Susan Stout (World Bank)

• Challenges and Issues in Designing & Implementing ECD - Country Perspectives (MES Member, P C

K Abeykoon)

• Q&A

10:15 – 10:45 Refreshments

10:45- 11:45 Group Breakout & Discussion (four groups)

• Implications & Challenges of ECD for Developing Countries

• Alternative Strategies for Developing & Institutionalising ECD for Developing Countries

• Recommendations for Next Steps

11:50 – 12:45

Plenary Session (Chair Person: Jerry Winston)

• Report Back by Groups

• Discussion & Recommendations: ECD and the Way Forward

12:45 – 13:00 • Closing Remarks (Karen Russon, ECDG & Aru Rasappan, MES/CeDRE International)

13:00 – 14:00 Lunch



Annexure 3: Opening Address Karen Russon Thanks to Dr. Aru for allowing ECDG to co-‐sponsor this important event. Thanks to all of you for recognizing its importance and being here on a Friday morning following a long week of conferencing and workshops. I would like to provide a little history on how ECDG came to sponsor this seminar on ECD. It goes back one year. Last September, Dr. Aru facilitated a month-‐long discussion on the topic of ECD on XCeval -‐ a cross-‐cultural evaluation listserv. The topics for this rich and informative discussion included ECD dimensions and interpretations; the need for ECD & its standardization; strategies (approaches & interventions) for ECD; and responsibilities for ECD. Following this online discussion, an international workshop on ECD took place in Geneva, Switzerland in October 2011. The event was open to the public and brought together two dozen ECD experts and members of standardization organizations from around the globe. Participants came with their own ECD perspectives, those of their constituents, and the input of the greater evaluation community thanks to the XCeval discussion. Several who are attending the MES conference were present. It was an intense 5 days. Staying on the campus of a quiet retreat center with meals taken together, the atmosphere was conducive to discussions going beyond the meeting room. Some participants met in the evening to continue deliberations. Some lingered after tea breaks, caught up in interesting conversations and had to be coaxed back to the next session. Over the ensuing three months following the workshop, participants worked online and in work groups through a designated listserv to produce a voluntary, consensus-‐driven agreement known as the International Workshop Agreement (IWA) on Evaluation Capacity Development. The IWA, including a transcript of the ECD discussion on XCeval facilitated by Dr. Aru, can be downloaded on the ECDG website -‐ ecdg.net. The document is a transparent accounting of the group process during the workshop. And it lists some of the wide range of ECD activities and processes such as: training, mentoring, coaching, learning by doing, technical assistance, sharing practice guidelines, developing organizational policies and infrastructure, supporting communities of practice, preparing legislation, and promoting a favorable political environment to sustain evaluation. I want to briefly mention two fundamentally important and key components of this International Workshop Agreement. One is a framework for describing an ECD intervention. It provides a new approach to engaging stakeholders in the ECD process by encouraging them to view ECD situations in different ways, to work out some of the logical consequences of each perspective, and to compare them with the messiness of reality. The second important component is a set of criteria that was created for the successful development of evaluation capacity and the attributes of quality evaluation practice. These four principles are: ownership, relevance, integration, and usefulness. Requirements to fulfill each principle are outlined in the IWA. The process the group went through to create this document had a profound impact on ECDG’s mission. It became apparent to us that a paradigm shift had to take place. Research in SE Asia, discussed during the workshop, pointed to the fact that the donor community is driving ECD and their efforts to develop evaluation capacity are often severely under-‐resourced. Ada Ocampo will be sharing the findings from these regional studies in a few moments. It was evident to ECDG that a need existed for an independent organization to act as an “honest broker” between donors, ECD experts and practitioners in the field. As a result of this understanding, our organizational vision was refocused to establish our NGO as an intermediary organization for re-‐granting purposes. ECDG’s first pass-‐through grant is on behalf of the Canadian International Development Research Centre (IDRC) to sponsor this MES Day-‐5 Seminar on ECD. The American linguist and philosopher, Noam Chomsky said, “Optimism is a strategy for making a better future. Because unless you believe that the future can be better, you are unlikely to step up and take responsibility for making it so.”



This morning, I encourage you to address issues, challenges, problems and conflict. But do so by shifting the focus and language from one of deficits to one of hope and possibilities based on what has worked, what is working, and what could work. Rather than being reactive, be generative. By this I mean to look for positive steps and applications to develop this critical institutional capacity. Discussions that take place this morning will reveal a glimpse of the regional vision for ECD. ECDG will be looking for opportunities to support your efforts to realize this vision. Mahatma Ghandi said, “The future depends on what we do in the present.” The future of ECD will be shaped by what you learn in the seminar this morning, take back to your government agencies, civil service organizations or community groups, and then strive to implement. As they say in French, “Bon courage.”



Annexure 4: Presentation of Findings and lessons learnt: ECD Study in Asia and Pacific by UNEDAP










Perspective: UNICEF Experience








Perspective: UNDP Experience













Annexure 7: Evaluation Capacity Development: Challenges & Issues from Development Aid Partner Perspective: World Bank Experience









Annexure 8 ECD Survey Questionnaire

MES-‐ECDG EVALUATION CAPACITY DEVELOPMENT SURVEY Evaluation Capacity Development (ECD) involves activities and processes that help create, strengthen, and sustain evaluation for individuals, teams, groups, networks, organizations, communities, sectors, and countries. Such activities and processes may include training, mentoring, coaching, learning by doing, technical assistance, sharing practice guidelines, developing organizational policies and infrastructure, supporting communities of practice, setting up systems, preparing legislation, and promoting a favorable political environment to sustain evaluation, among others. Please provide information for the following set of questions based on your own country experiences. Question #1: In what areas are you (or your country) focusing your efforts to develop evaluation capacity? Question #2: What are the greatest challenges to institutionalizing evaluation capacity that you/your country faces? Question #3: To what extent are capacity development activities supporting improvements to:

Question #3: To what extent does evaluation capacity development in your country cover, or is a part of, the following aspects:

We thank you kindly for your cooperation in completing and returning this survey form to us by 24th August, 2012

ECD Activities RATING Low High

1. Policies and programs 1 2 3 4

2. Program management 1 2 3 4

3. Resource allocations, budgeting 1 2 3 4

4. Government control, coordination 1 2 3 4

5. Accountability and transparency 1 2 3 4

6. Participation by civil society 1 2 3 4

ECD Aspects RATING Low High

1. Evaluation Policy 1 2 3 4

2. Evaluation Standards 1 2 3 4

3. Evaluation Guidelines and Procedures 1 2 3 4

4. Evaluation Training for key personnel 1 2 3 4

5. Performance Management Systems in the Ministries 1 2 3 4

6. Performance Measurement & Reporting System 1 2 3 4

7. Budgeting System 1 2 3 4

8. Personnel Performance Management system 1 2 3 4



Annexure 9: Presentation on MES-‐ECDG ECD Survey









Documents

ECD Report Ver 3.0 - ECDG