Suggestions for the Evaluation of Transitional Justice

Suggestions for the Suggestions for the EvaluationEvaluation

of of Transitional JusticeTransitional Justice

AngleAngle Taken in this Taken in this PresentationPresentation

Perspective of an evaluator of conflict related Perspective of an evaluator of conflict related programmes, of which TJ is one field. The programmes, of which TJ is one field. The experience of TJ is drawn from:experience of TJ is drawn from:

Evaluation of EC support to human rights Evaluation of EC support to human rights and the rule of law, thematic 2005, and and the rule of law, thematic 2005, and support to ICC ratificationsupport to ICC ratification

Evaluation of USAID peace-building and Evaluation of USAID peace-building and justice support programmes in Burundi, justice support programmes in Burundi, Senegal (Casamance) and Uganda (north).Senegal (Casamance) and Uganda (north).

Evaluation of a World Bank and of a Danida Evaluation of a World Bank and of a Danida funded project in Burundi concerning the funded project in Burundi concerning the repatriation of refugees with a focus on repatriation of refugees with a focus on restorative and traditional justice.restorative and traditional justice.

WhatWhat is Evaluation ? is Evaluation ?

Evaluation is…Evaluation is… To assess or judge the value or worth of To assess or judge the value or worth of

something (its relevance, effectiveness, something (its relevance, effectiveness, efficiency, sustainability)efficiency, sustainability)

It is Not MonitoringIt is Not Monitoring::Continual self-evaluation of changes to generate Continual self-evaluation of changes to generate

a lateral view, through formal and informal a lateral view, through formal and informal processesprocesses

It is Not Impact AssessmentIt is Not Impact Assessment: : Tracking of change in a society as it relates to an Tracking of change in a society as it relates to an

identifiable source or set of factors identifiable source or set of factors

Why Why Evaluation?Evaluation?

WhyWhy Evaluation ? Evaluation ?

AccountabilityAccountability LearningLearning

… … However evaluation comes at a However evaluation comes at a greater cost (financial or not) greater cost (financial or not) than is acknowledged, and are than is acknowledged, and are used sparingly.used sparingly.

WhyWhy Evaluation? Evaluation?

Evaluation is part of the concept of results Evaluation is part of the concept of results based management: a verifiable argument based management: a verifiable argument about what worked and what did not, and about what worked and what did not, and why is it so.why is it so.

It guides the interface between a programme and a It guides the interface between a programme and a society- all too often this information is restrictedsociety- all too often this information is restricted

Creating an information and analysis Creating an information and analysis environmentenvironment

Creating a flow of decision-oriented informationCreating a flow of decision-oriented information Obtaining reliable information about the progress of Obtaining reliable information about the progress of

activites and their outcomes, the reasons for success activites and their outcomes, the reasons for success or failure, and the contexts in which activities are or failure, and the contexts in which activities are taking place. taking place.

HowHow to Evaluate: to Evaluate:

A Rule of Tumb: Keep Definitions Simple:A Rule of Tumb: Keep Definitions Simple: Activity: the process to deliver an output Activity: the process to deliver an output

(use made of an input or resources)(use made of an input or resources) Output: a product, usually a definable Output: a product, usually a definable

quantity (most easily monitored over time quantity (most easily monitored over time and space)and space)

Outcome: use made by the beneficiaries of Outcome: use made by the beneficiaries of an outputan output

Impact: the consequence of the outcome, Impact: the consequence of the outcome, or change triggered by the use of an or change triggered by the use of an output.output.


Outcome MappingOutcome Mapping focuses in focuses in particular on changes in behaviour particular on changes in behaviour and relationships, and seeks to be and relationships, and seeks to be specific about changespecific about change

My preferred definition of outcomes My preferred definition of outcomes fits this concept, although in some fits this concept, although in some ways it is more specific even: the use ways it is more specific even: the use made of an output is a particular made of an output is a particular form of behavioural change.form of behavioural change.


HowHow: Spheres of : Spheres of InfluenceInfluenceChange operates as a ripple of change around an Change operates as a ripple of change around an

actoractor

Objectives: Impact: general

population = sphere of indirect

influence

Results: Outcomes: level

of beneficiaries = sphere of direct

programme influence

Activities: Outputs: level of

implementing partners, = sphere of primary influence


Change should ideally be mechanical like a spring. The Change should ideally be mechanical like a spring. The combination of different changes should be necessary combination of different changes should be necessary and sufficient to lead to a higher change. This calls for and sufficient to lead to a higher change. This calls for projects and programmes and sector-wide approaches projects and programmes and sector-wide approaches to be modest in the claims they make about impact.to be modest in the claims they make about impact.

ImpactImpact

OutcomeOutcome

OutputOutput

Activity Activity

Evaluation

HowHow to Evaluate: Plan & to Evaluate: Plan & FeedbackFeedback

Resources

Activity & Process (100% control)

Output (75% control)

Outcome (50% control)

Impact (25% control)

Detailed Operational Plan

Implementation

Results, Specific & General Objective

Improvements through M&E

HowHow to Evaluate: to Evaluate: Intentional DesignIntentional DesignThere are no set objectives, outcomes and indicators. These There are no set objectives, outcomes and indicators. These depend on depend on

the casethe case::For example: “Progress in adherence to the Rome Statute, a For example: “Progress in adherence to the Rome Statute, a

better public perception of the court’s role, and toward better public perception of the court’s role, and toward implementation of the Rome Statute into national implementation of the Rome Statute into national legislation”:legislation”:

Emergence of national lobby groups and coalitions to support Emergence of national lobby groups and coalitions to support ratification by a given state of the Statutes of international criminal ratification by a given state of the Statutes of international criminal courtscourts

Evidence of trust and active relations within a network of Evidence of trust and active relations within a network of professional experts in areas relevant to the Courts; professional experts in areas relevant to the Courts;

evidence of a strong community of purpose, in particular in the evidence of a strong community of purpose, in particular in the number of subscriptions to specialised newsletters, number of “hits” number of subscriptions to specialised newsletters, number of “hits” to a specialised website, participation in meetings, participation in to a specialised website, participation in meetings, participation in demonstrations.demonstrations.

Recurrence of key words in the national press, particularly the most Recurrence of key words in the national press, particularly the most popular media, and in specialised legal literature.popular media, and in specialised legal literature.

Judges and magistrates are selected in a public and transparent Judges and magistrates are selected in a public and transparent manner which is commented on in national communities.manner which is commented on in national communities.

WhatWhat to Evaluate: to Evaluate: Intentional Design Intentional Design

Set objectives, and detail changes which could be Set objectives, and detail changes which could be observed as a result of success (here based on F. observed as a result of success (here based on F. Oduro):Oduro):

Trust Trust restoredrestored

Forward Forward relationshrelationshipip

CooperatiCooperative ve endeavouendeavoursrs

BehaviourBehaviours and s and attitudesattitudes

StructuraStructural l transformtransformat°at°

Process Process (still (still outcome!)outcome!)

…… …… …… …… …… ……

WhatWhat to Evaluate: The to Evaluate: The Constraint of TransitionConstraint of TransitionThe The challengeschallenges of programmes in of programmes in situations of situations of

transitiontransition are unique and different from are unique and different from evaluations in other fields. This naturally evaluations in other fields. This naturally applies to TJ evaluations:applies to TJ evaluations:

The balance between reconciliation The balance between reconciliation and other objectives is rarely clearand other objectives is rarely clear

The constraint of The constraint of TransitionTransition

The balance between reconciliation The balance between reconciliation and other objectives is rarely clearand other objectives is rarely clear

Objectives as benchmarks is denied Objectives as benchmarks is denied by the fact that conflict by the fact that conflict programming requires above all a programming requires above all a response to opportunitiesresponse to opportunities


The balance between reconciliation and The balance between reconciliation and other objectives is rarely clearother objectives is rarely clear

Objectives as benchmarks is denied by Objectives as benchmarks is denied by the fact that conflict programming the fact that conflict programming requires above all a response to requires above all a response to opportunitiesopportunities

The justice and reconciliation objective The justice and reconciliation objective is not well articulated to the objectives is not well articulated to the objectives of a programme, often based on of a programme, often based on assumptions.assumptions.


The balance between reconciliation and The balance between reconciliation and other objectives is rarely clearother objectives is rarely clear

Objectives as benchmarks is denied by the Objectives as benchmarks is denied by the fact that conflict programming requires fact that conflict programming requires above all a response to opportunitiesabove all a response to opportunities

The justice and reconciliation objective is The justice and reconciliation objective is not well articulated to the objectives of a not well articulated to the objectives of a programme, often based on assumptions.programme, often based on assumptions.

Selecting indicators demands a strong Selecting indicators demands a strong understanding of the conflict and culture, understanding of the conflict and culture, and verification methods applicable to a and verification methods applicable to a conflict situation.conflict situation.


The balance between reconciliation and other The balance between reconciliation and other objectives is rarely clearobjectives is rarely clear

Objectives as benchmarks is denied by the fact that Objectives as benchmarks is denied by the fact that conflict programming requires above all a response conflict programming requires above all a response to opportunitiesto opportunities

The justice and reconciliation objective is not well The justice and reconciliation objective is not well articulated to the objectives of a programme, often articulated to the objectives of a programme, often based on assumptions.based on assumptions.

Selecting indicators demands a strong understanding Selecting indicators demands a strong understanding of the conflict and culture, and verification methods of the conflict and culture, and verification methods applicable to a conflict situation.applicable to a conflict situation.

Justice and reconciliation usually depends on the Justice and reconciliation usually depends on the identification of an end-state, an overall objective. identification of an end-state, an overall objective. The value given to this may not be shared by all The value given to this may not be shared by all stakeholders, even those within a single programme stakeholders, even those within a single programme or organisation.or organisation.

WhatWhat to Evaluate: to Evaluate: Opportunistic Design or Opportunistic Design or Evaluation Without Evaluation Without ObjectivesObjectives

Individual Individual Project Project OutcomesOutcomes

Analysis Analysis of of Societal Societal IssuesIssues

Testing Testing the the connectioconnection n outc./issuoutc./issueses

ReasonsReasons

…… …… …… ……

Outcome LevelOutcome Level: For Opportunistic : For Opportunistic Strategy or Sector-Wide ApproachesStrategy or Sector-Wide Approaches

E T

E

E

E

TT

T

Conflict Drivers Issue 1

Issue 2

Issue 3

Outcomes Outcomes Outcomes

Relevance x Extent x Duration

WhoWho Evaluates? Users Evaluates? Users and Rolesand Roles All evaluations of TJ I have done were All evaluations of TJ I have done were

commissioned by donors, and were not commissioned by donors, and were not well disseminated to the beneficiaries.well disseminated to the beneficiaries.

The link between general public The link between general public perception, research on one side, and perception, research on one side, and evaluation on the other, is very tenuous.evaluation on the other, is very tenuous.

Victims, support groups, civil society, Victims, support groups, civil society, need to intervene in what questions are need to intervene in what questions are asked at the outset, for evaluations to be asked at the outset, for evaluations to be relevant to them.relevant to them.

WhoWho Evaluates? Evaluates? Participatory EvaluationParticipatory EvaluationOnce stakeholders have been identified, Once stakeholders have been identified,

agenda, objectives and methodology agenda, objectives and methodology should be agreed with them. The skills are should be agreed with them. The skills are the ability to ask critical questions, and the ability to ask critical questions, and suspend personal judgement. The following suspend personal judgement. The following questions could guide preparation:questions could guide preparation:

What are the questions to be answered?What are the questions to be answered? What will the evaluation be used for?What will the evaluation be used for? Who must be heard and how to create the Who must be heard and how to create the

conditions for them to speak freely and conditions for them to speak freely and openly?openly?

Evaluation of Evaluation of Transitional JusticeTransitional Justice

When the programmes are closer to When the programmes are closer to a classical development programme a classical development programme I would argue that evaluation is not I would argue that evaluation is not very different from a general form.very different from a general form.

When on the other hand the When on the other hand the objectives are not clearly defined, it objectives are not clearly defined, it is best to approach evaluation by is best to approach evaluation by focusing on outcomes and on their focusing on outcomes and on their connection to the context and needs.connection to the context and needs.

Documents

Suggestions for the Evaluation of Transitional Justice