Policy and Operations Evaluation Department (IOB) Evaluability Asessment: Preparatory Steps before...

Policy and Operations Evaluation Department

(IOB)

Evaluability Asessment:

Preparatory Steps before

Starting an EvaluationProf. dr. Ruerd Ruben

Director Policy & Operations Evaluation Department (IOB)

2

Rationale

Growing number of pseudo evaluations

Repositioning Evaluation in PM&E cycle

Concept of ‘evaluability’ (Whiley 1979; Smith 1989)

Procedures for Evaluability Assessment (EA)

Further steps

3

Pseudo evaluation • Growing number of aid projects/ programs.

• DAC Evaluation Resource Centre (DeRec): > 1000 evaluations

• MinBuza: substantial decentralized & local (embassy) evaluations

• Broad definition of evaluation: systematic assessment if and how activities contributed to stated goals

4

Loss of Evaluability

• Review of Dutch Private Sector Programs 2/3 of executed ‘evaluations’ cannot be used

• Major reasons:- evaluation agency not fully independent- stated objectives too broad/vague- no clear indicators defined- data at too aggregate level- absence of baseline data- no representative sampling

- too general intervention theory

5

Evaluation in the PME Cycle

• Ex-post assessment of program performance

• Separation of ‘monitoring’ (internal) and ‘evaluation’ (external)

• Tracking of evaluation results

• Ex-ante guarantees for effective monitoring/evaluation

Ex post

Ex ante

EA

6

Theory of Evaluability

• Systematic process to determine whether program evaluation is feasible. • Decide whether an evaluation is useful to provide timely, relevant findings for decision makers.

• Wholey (1979): initial steps to assess if a program can be evaluated (a) set of clear objectives (+ side effects) (b) measurable performance indicators • Smith (1989): ten steps procedure for Evaluability Assessment initial diagnostics

7

Evaluability Assessment

1. Identify relevant stakeholders2. Define boundaries of the program3. Analyze available program documents4. Clarify intervention theory (goals, resources, activities, outcomes)

5. Analyze stakeholders perceptions of the program6. Assess target population(s)7. Discuss differences in outcome perceptions8. Determine plausibility of intervention model

9. Discuss validity of the program10.Decide about continuation (= full evaluation)

8

Key EA Questions

1. Does the program serve the population for whom it was designed?

2. Does the program have the resources (available/used) as scheduled in the program design?

3. Are the program activities being implemented as designed?

4. Does the program have the capacity to provide data for an evaluation?

9

Crucial Steps

EA has five crucial tasks that an evaluator must successfully complete:

• Task 1. Study the program history, design, and operation;• Task 2. Watch the program in action (participants)• Task 3. Determine the program’s capacity for data collection, management and analysis;• Task 4. Assess the likelihood that the program will reach its goals and objectives (consistency)• Task 5. Show whether an evaluation is feasibly (will help the

program and its stakeholders).

10

Towards EA operationalization

• Screening of programs with expenditures > € 5 million

• Develop EA Protocol (e.g. Unifem’s Checklist for Programme Evaluability UNEF Delivery as One United Nations)

• Systematic review by sector, by channel, by country, etc

• Ex-ante assessment (part of project approval procedure)

• Yearly reporting (feedback to program design & operations)