The SPADA/MCLE impact evaluation:

work in progress

Markus Goldstein

AFTPM/DECRG

The acronyms

• This is an impact evaluation of the Mediation and Community Legal Empowerment of the Support for Poor and Disadvantaged Areas Project in Indonesia

• It’s a big project, and the impact evaluation will only be for one component (other impact evaluations for other components)

The big idea

• We have (almost) no rigorous evidence of the effects of governance reform

• Why?– It’s hard to measure– It’s hard to set up

• Setting up control groups• Getting sufficient sample size

– It’s political science & IE has tended to come from economists

• We know very little about poverty-governance-justice nexus

MCLE: project objectives

1. Strengthen the capacity of institutions and individuals to resolve disputes in an open, independent, and just manner

2. Assist and facilitate the resolution of disputes involving community wide interests

3. Increase public trust in the formal legal system by enhancing community legal awareness and access to the system

MCLE: project activities

• Community legal education

• Capacity building for informal justice actors

• Legal aid and complaint handling network

• Awareness raising of community legal needs and challenges

NGO implemented under government contract

MCLE activities - 2

• Support network for providing legal aid and empowerment activities

• Focus at community level• Sub-district: establish posko (community legal

aid posts), staffed by facilitator and providing legal information, mediation and complaints handling

• Village level: paralegals (one man/one woman) – work with local leaders and community figures

• District level: community lawyer – part time technical legal support

Why evaluate this project?

• Global– What does access to law get you?– What works to get people access to justice?– Top down vs bottom up access to legal system &

formal vs informal dispute resolution• Local

– Gov’t is preparing first ever access to justice strategy for inclusion in the 2009-14 medium-term development plan

– Inform nation-wide poverty and empowerment program

– Scale up or not?

Evaluation questions: primary

• Does MCLE increase beneficiaries ability to claim rights, enforce rights, and resolve disputes through non-violent means?

• Does MCLE reduce violent conflict and increase safety and security?

• Does MCLE improve HH welfare in the villages that receive paralegals?

Note on analyzing the causal chain…

• Idea here is to look at different points of the causal chain:

Knowledge of rights (ask about legal K)

Enforcement of rights (e.g. get a land cert.)

Better dispute resolution (e.g. mode used, hypothetical dispute resolution)

Investment in property (e.g. in land improvements)

Improvements in household consumption

Evaluation questions: secondary

• Does MCLE increase the reporting of corruption and bribery and decrease these events in the long term?

• Does MCLE improve individuals’ capacity to engage with the state and the state capacity to respond, particularly for dispute resolution?

• Does MCLE improve gender relations at the household and community level?

Evaluation design

• Two provinces: selected for level of conflict (Aceh and Maluku)

• Districts chosen by gov’t based on need and demand: issues of generalizability

• Sub-districts: random selection agreed by gov’t

• Villages: random selection• Randomization done in Washington with

evaluation team

Evaluation design:quantitative core

• Method is a randomized difference in difference

• Baseline quant survey to run right as contracting for implementation goes out

• Mid-point to run 2 years after implementation (early results for policy)

• End-line survey to run 4 years after implementation

Evaluation design: survey structure

• Innovative household survey (“best of” other governance/justice surveys)– Pretest and adjust– Links with SUSENAS– Rotating panel

• community survey (capture major disputes, village level variables)

• Key informant survey (capture village/district authority attributes, perceptions of safety, corruption, legal claims)

• Program facilitator survey (get at heterogenity of treatment – ed levels, experience, social connections)

Evaluation design:other data sources

• Newspaper monitoring: conflict levels, security and safety, reported corruption

• Project MIS – implementation and also dispute tracking and other case handling information

Evaluation design:Qualitative core

• Quant answers: does MCLE have an impact while Qual answers: How does it have an impact

• Village mapping (understanding village dynamics)– Village profile (demographics, livelihoods, etc)– Dispute timeline (types, sequence, actors,etc)– Claims and collection timeline (indiv vs community)– Power relations and networks – map these

Evaluation design:Qualitative core

• Comparative case studies– In-depth studies in treatment and comparison

locations – similar to quant in structure

• Community ethnographies– Two per province– How does program work, how does it interact w/local

power structures, experiences of key individuals

• Cooperation across qual-quant data collection, not just iterative data collection

Complication: CRT

• Add a conflict resolution training component in a select number of villages

• Creates sampling headache• Power issues: the effect of treatment A vs

treatment B requires (in general) a larger sample than treatment A vs control

• Solution: lay it on top of MCLE (still have avg effect, control for CRT), separate CRT only villages. CRT contributes to surveys

• Danger here is that things get too complicated and we end up with nothing done well…

Evaluation team

• East Asia SDV: project design team, evaluation/survey expertise

• PRMPR/DECRG: evaluation design/analysis team (quant/qual)

• Gov’t: project design/M&E staff• Legal: eval design, guidance, dissemination,

funding• DECRG: survey design specialist• Consultants: sampling, local qualitative work• Survey design firm

Dissemination plan

• Core team builds design• Disseminate evaluation design w/ wider gov’t

audience, implementing NGOs• Baseline analysis – informs program design,

gov’t agencies• IE results:

– Workshop w/ gov’t early – compare treatment arms– Disseminate more widely – other relevant programs

(national pov/empowerment program), NGOs, Bank, donors, academics, regional J4P program

Budget and funding

• Survey costs: still unknown, problem of Maluku islands

• Qual contract out

• Initial seed funding from BNPP grant, Aceh TFs, project $. More grant applications on the way

• Donated time helps a lot (save on BB)

• Missions at end of FY…

A couple of points of reflection

• The process was important, esp initial workshop with gov’t & project team

• Project & IE team really, really have to work together (lots of early am VCs)

• Be prepared for transitions in all teams• Doing something new? Early survey pretest

helps for sample size calculations• Timing of “end-line” survey depends on policy

cycle – but also time for effects to manifest• Keep it as simple as you can