Experiences in Impact Evaluation: The PEMA Perspective

Mandate PEMA (Centre for Project Evaluation &

Macroeconomic Analysis) was established in January 2003 as an operating unit of the Ministry of International Cooperation

While a basic mandate is impact evaluation, this mandate

has been extended during the past three years to:

a) Help upgrade the M&E capacities in key ministries

b) Coordinate the M&E reports prepared by various U.N. agencies in Egypt to assess progress made annually in achieving the objectives under the United Nations Development Assistance Framework (UNDAF)

c) Contribute to OECD efforts to survey progress towards better aid effectiveness

PEMA undertakes evaluations of development projects in order to:

Ascertain their impact on target beneficiaries and the sustainability of such impact

Draw lessons learnt from evaluation and put these at the disposal of policy and decision makers to assist them in taking measures to improve the efficiency and effectiveness of assistance received

PEMA has completed the evaluation of 31 projects in various sectors

Current staff of the Centre compromises twelve researchers and the executive director

Steps for undertaking impact evaluation

Once a project is selected, the following steps are undertaken:

Desk study of documents Propose evaluation questions and methodology Data gathering (fieldwork) Content analysis Draft report writing Roundtable meeting is held to receive stakeholders’

comments Final report is sent to stakeholders and to Minister of

International Cooperation, whereby follow-up actions take place

Findings on impact evaluations in various sectors

Educational Sector

Girls’ access to primary (basic) education is enhanced by providing comprehensive packages of community services

Projects implemented for improving teaching methods in primary schools are not sustainable due to weak monitoring mechanisms

Vocational Education: Businessmen (private sector) support is crucial for the success

and sustainability of vocational education programs, particularly for funding training in specialized areas.

Many of the graduates of vocational schools do not pursue a vocational career as they enter universities to obtain academic degrees

Small and micro-enterprise development Building the capacities of NGOs and CDAs is crucial for

creating efficient systems for operating micro-credit loans

High charges of loans and disregard of the grace period for loan repayment impede the sustainability of SMEs

Insufficient follow-up surveys to determine the impact of loans on beneficiaries, and to identify measures that could further enhance sustainability and effectiveness of loans

Insufficient coordination between funding agencies results in the implementation of several SME projects in the same areas, thus reducing demand for loans

Health Sector Embedding project activities into the existing

operational structure of health system is essential for ensuring sustainability

Systems for collection and analysis of data on health services are required for effective supervision and delivery of services

Highly centralized system of distribution of medical resources (medical staff and medical supplies) results in ineffective distribution

Low salaries and inadequate financial incentives lead to shortage of medical staff and health workers

Agricultural Sector Insufficient public facilities discourages

beneficiaries from settling in reclaimed areas

Insufficient irrigation water impedes application of modern farming practices in some of the reclaimed areas

Limited financial and technical capabilities of

farmers associations after project termination

Lessons learned from impact evaluations

More emphasis should be given to project design: project objectives; outcomes; means of verification and implementation strategy

Some projects do not set technical and financial mechanisms required for sustaining project outcomes

In some projects, responsible bodies do not establish new units or appoint permanent staff to implement and monitor project activities

Capacity building activities should be relevant and based on the needs of trainees

Setting criteria for selecting trainees and ensuring application of criteria are important for ensuring effectiveness of capacity building activities

Executing bodies should have more flexible systems of work by encouraging trained personnel to apply what they learned from the project

It is crucial to establish budget lines within ministries for capacity building activities

Bad management of project's time leads to implementation of project activities in the remaining time of the project. Therefore, activities are implemented at "less than acceptable" level.

A correct budget estimate for project activities is essential to avoid deficit or loss of funds

How do target beneficiaries perceive of the project and its outcomes is an important indicator of project effectiveness