Valuing Voices Cekan Wageningen Conference 0315

Jindra Cekan, PhDValuing Voices, Sustainable Solutions for Excellent Impact

Wageningen University M&E for Responsible Innovation Conference, March 2015

Who’s Listening? Community-led Post-Project Sustainability Evaluation

Great Evaluation and Learning work

© 2015 Valuing Voices, Cekan Consulting LLC

•Our industry is evaluates thousands of multi-sectoral projects each year

•Donors have invested millions of dollars/Euros in learning ventures looking for real impact, such as:• Grand (Health) Challenges• Making All Voices Count• Aid Transparency (IATI)• Impact Evaluation by 3ie

•100 national national evaluator associations in 93 countries support country-led evaluation

Overlap of M&E for Responsible Innovation + Valuing Voices

• How can M&E responsibly support the management and governance of innovation … and contribute to deeper reflexivity and transparent decision making? Are Institutional changes needed?

• How do we take responsibility for systemic change in: – M&E professional’s roles & responsibilities; – M&E process design, focus and approach

Three questions:1. What is in our way?2. What can be done?3. What methods are best?


Governance: Global Aid Effectiveness


Paris Declaration (2005), Accra Agenda for Action (2008) and the Busan Partnership (2011) created shared principles to achieve common goals:

• Ownership of development priorities by developing counties

• A focus on results

• Partnerships for development

• Transparency and shared responsibility

Development becoming more transparent, focused on sustainability

• Local Accountability has been proposed as a core feature of the new post-2015 development agenda, according to UNICEF.

• Donors talk about Country-led Development, M&E Capacity (OECD), Local Systems (USAID), as do non-profits like Local First- wonderful. Let’s support community-owned and driven ones!


• MDGs are becoming Sustainable Development Goals post-2015

http://post2015.org/tag/unicef/

http://www.mymande.org/content/country-led-development-evaluation-donor-role-supporting-partner

http://www.mymande.org/content/country-led-development-evaluation-donor-role-supporting-partner

http://valuingvoices.com/sustainability-springing-out-all-over-the-place-and-disrupting/

http://actlocalfirst.org/

http://sustainabledevelopment.un.org/?menu=1300

http://sustainabledevelopment.un.org/?menu=1300

Changes in Responsibilities, Approach, Focus


For them to evaluate our efforts = how must we change?

1. What’s in our way?


Definitions: ‘Sustainable’ Development?


How do we know it’s sustainable if we do not go back and ask?o $1.6 trillion spent on international development by EU and US alone since 2000o $100 billion spent in 2014o Trillions more spent by private donors ALL UNEVALUATEDo National evaluations nonexistent o Data + tracking systems discarded<1% of projects evaluated post- closeout for sustainability We don’t know, we haven’t asked

about the other 99% projects

Scant Design for self-sustainability

Communities’ voices are rarely elicited to design what activities they will make last… for RFPs are not designed in communities

Yet to not evaluate sustainability for future design short-changes the communities which have entrusted us to improve their lives


What Exists Now in project sustainability

2-5 year duration, typically with great M&E

…The continuity of people’s lives in communities…

? ? ? ? ?Earlier or concurrentDevelopmentprojects

? ? ? ? ?

? ? ? ? ?

STOP?

baseline midterm finalE V A L U A T I O N S

Feedback loops?!


Follow –on or new

Project

ProjectProject

GAP

“Responsible Development” requires Innovation


Why We Value Voices


Designing for sustained exit should be our first priority

We ALL deserve self-sustaining projects: Communities/ countries, we development workers, taxpayers… it’s true respect.

‘They’ should evaluate our assistance for effectiveness, not the other way around

A business model works – looking at Return on Investment through their eyes illuminates what should be repeated – or not.

Once we know what communities could self-sustain, let’s design and fund future projects accordingly!

‘They’ are ‘Us’

Views from outside on our DIME*


"You cannot set your own examination, take the examination and mark it. Then you cry out success or failure. The community will just 'look at you' and wonder what is the issue.... It will be sustainable development if the people at community level are involved in designing and delivering their own dreams of development”

Peter Kimeu, Catholic Relief Services’ Senior Technical Advisor for Partnership/ Solidarity, East Africa (30 years expertise)

* Design, Implementation, Monitoring and Evaluations

Views from outside on our DIME*


• “We push new ideas, rather than building on what communities have already created…. we must be able to look back and see what they've learned and can sustain themselves.… to show what they've done themselves, and where they think they're taking themselves. [Sustainability] is an ignored area. The challenge is, it's just been gathering numbers for 5 years, 10 years….

• We don't have the right feedback system from communities to the NGO world. It’s the tyranny of deadlines so they "just get their concerns" when staff go out [to the field]…”R.K. Kenyan Independent Evaluation Professional (20 yrs)* Design, Implementation, M&E

VV Data Sustainability Approach: Open Data

• Design data captured using open data international standards (such as IATI)– Aggregate with other data

sets (project, demographic, maps)

– Sharable beyond project staff/activities

– Available after project period of performance

– Allows for public accountability /transparency

– Historical marker of “what happened back then” for learning from measurement


Mapping malnutrition in Uganda

2. What Can Be Done?


Discussion: what is in our way, what can be done?


World Bank Civil Society Forum Meeting 2014

1. ARE WE RIGHT ABOUT WHAT’S IN OUR WAY?2. WHAT CAN WE DO TO HELP DONORS LISTEN

AND HELP IMPLEMENTERS COME ON BOARD?

LISTENING TO YOUR VOICES!

3. What methods are best?



Right Design of Post-Project Sustainability Evaluation and Objectives

DEVELOPMENT INDUSTRY LEARNING

ACCOUNTABILITY TO PARTICIPANTS

LEARN FROM UNEXPECTED IMAPCTS

EMPOWER LOCAL NGOS, COMMUNITIES, AND NATIONAL EVALUATORS

LOCAL ROI SHAPES WISER FUNDING DECISIONS AND BETTER PROGRAMMING

PROMOTE COMMUNITY-INFORMED AND JOINTLY LED LEARNING FROM SUSTAINED OUTCOMES

Moving from Forest to Weeds… How


“Because… the devil is in the detail”

Site/ Partner Selection• Project sites closed within the last 2-5 years• Communities, partners available for consultations, final

evaluations accessible• National evaluators are available and interested• Sampled sites comparable in livelihood, diversity• No similar work done by NGOs in gap years• Ideally web-savvy partners to handle data/ analysis/ curation/ storage

• Analysis, presentation, sharing and curation of findings are clear (by whom and for whom, how, where, for how long)



Mixed-Methodologies• Qualitative: Participatory RRA methods/ tools (based on Appreciative Inquiry and Empowerment Evaluation, potentially also Outcome Harvesting) of past participants in projects

• Quantitative (purposive, then random-sampled HH survey, possibly including mobile surveys for participants who’ve left

• Small control group: non-participants in same communities• Use mix of internal and external evaluation teams covering all

sectors • Findings shared widely and stakeholders invited to discuss

Qualitative process


Research:1. Update sectoral evaluator team and translators in RRA

approach2. Discuss best RRA tools (maps/calendars, rankings, focus groups)3. Visits: Two to three days per site in 3-4 sites, with meetings in

communities reminding them of project and asking:A. What activities/outputs were most self-sustainable and why?B. Which activities should be repeated? Which shouldn’t?C. What else would participants like to evaluate of the project?Distillation:

1. Daily team analysis2. Each site summary analysis is

presented to the community for input

3. Community invited to final presentation in region

4. Findings inform survey + report Later, compare across projects/sectors

Quantitative process


Research:1. Survey questionnaires of a random sample subset of the

population (past participants and some communities’ non-participants)

2. Stratification of intervention area by criteria; the strata would include a homogeneous stratum (same behaviors in project, e.g. all sustaining activity) and heterogeneous stratum (those not) but who are judged on the same key variables or indicators.

3. Randomly choose a fixed number of villages for interviews

Distillation:1. SPSS for analysis2. Community invited to final

presentation in region 3. Findings for integrated report

Later, compare across projects/sectors

Looking for Unexpected Results


LWR Niger post-project unexpected impactsEvaluated Food security/ Livelihood/ Resilience

project: sheep distribution, fodder, wells, peaceful transhumance for 500 women post-drought

Expected findings:• Variable impacts on women’s economic benefits• Great impact on time gain from wells for income generation.

Unexpected wonder:• Decreased domestic violence from presence of livelihood + water


3. Methods: Discussion questions


1.COMMENTS ON METHODS FOR ANY ONE SITE?

2. THOUGHTS ON COMPARABILITY ACROSS SITES, ACTIVITIES AND PROJECTS?

Let’s change how international development works… together


Thank you for your [email protected]

mailto:[email protected]

Presentations & Public Speaking

Valuing Voices Cekan Wageningen Conference 0315