Dr Robin Vincent robvconsult@gmail.com (Independent Consultant and Visiting Fellow, Durham

School of Applied Social Sciences)CARES Conference Liverpool October 29th

Using a realist evaluation framework for a learning-focused evaluation of the Cambodian Initiative for

Disability Inclusion: lessons and challenges.

Overview Why a realist evaluation framework? – challenging dominant

evaluation practice in international development

Cambodian Initiative for Disability Inclusion – background

Components of the evaluation

Theory of change of the CIDI programme

Key thematic findings

Reflections on the method and process – value for ‘learning’ and ability to tackle complexity

Reflections on presentation of findings

Reflections on theory - complex causality and mechanisms

Why realist evaluation?Engagements with complexity theory and complex social change - attempts to put into practice insights for evaluation of development practice

Panos Institute, evaluation of ‘Communication for Social Change’ in HIV, health and development using insights from complexity (Vincent 2011)

UNAIDS, HIV prevention Think Tank 2009 – assessing the ‘social drivers’ of HIV infection (Vincent 2009)

DFID Uganda HIV prevention strategy using Realist Evaluation and Qualitative Comparative Analysis (late 00’s)

Collaborations with Durham School of Applied Social Sciences, Liverpool School of Tropical Medicine.

Intransigence around ‘traditional’ experimental methods in range of proposals reviews processes

Evaluation of community/public engagement with international health research (programmes supported by Wellcome Trust)

Cambodian Initiative for Disability Inclusion - CIDI

Complex programme with national reach 38 partner organisations, 55 projects over 2.5 years aiming ‘to improve the quality of life of people living with disabilities

by supporting national efforts to address the risks, causes and consequences of disability’

Variety of projects - disability awareness, physical rehabilitation, inclusive education, psychosocial support, accessibility, livelihood security including self-help and savings groups, mine risk reduction, road safety….

Focus on ‘influencing strategies’ rather than more direct ‘outputs’ facilitating networking among organisations in disability sector

(including disabled people’s organisations and ‘mainstream’ organisations)

Capacity development of organisations in disability sector

Wanted a learning focused evaluation to address complexity and link to participatory methods of the project

Evaluation elementsNegotiating with commissioning team and advisory

board - adapted Realist Evaluation framework

Evaluation team of 2 Cambodian based programme managers, one Australian Red Cross quality team member and lead evaluator

25 days of lead evaluator overall

10 days in Cambodia visiting projects and doing interviews, and 3 day participatory workshop

Range of existing sources of data

Additional data gathering

Existing data Existing eclectic mix of programme monitoring data, including a

summary report of outputs across all projects

Review of role of partnerships in catalysing change in disability inclusion by ‘Enable’, organisation of people with disabilities conducted just before the evaluation (seen as part of accountability)

CIDI on-line forum for organisations involved in the programme.

Documentation of regular all-partner participatory networking and learning workshops

Existing participatory self-assessment/evaluation exercises undertaken by CIDI participants (capacity development/empowerment dimension): ‘peer to peer’ evaluation and exchange visits ‘most significant change’ stories regular review meetings with CIDI programme team video accounts of where projects made a difference

additional elements Distilling ‘theory of change’ from secondary research and programme

documentation

‘Realist’ interviews and group discussions with range of stakeholders with different relationships to programme and ‘division of expertise’ – (Pawson & Tilly 2006, Pawson 2013)

Quantitative survey on networking and capacity development related outcomes and ‘how’ programme had helped (both as a data source and to focus participatory evaluation exercises).

Participatory evaluation exercises at 3 day workshop with 32 out of 38 organisations (drawing ‘journey’s of change’, network outcomes mapping, group discussion/analysis)

Visits to projects, observation at events

Ongoing reflection with key programme managers on emerging findings

Integration and analysis of data sources along realist evaluation lines and writing report

In search of the theory of change…

CIDI programme documentation a ‘people- centred programme logic’ drawing on

network perspectives in project/programme design Unelaborated theory of change on paper

Review of secondary literature on networking and capacity development in development contexts

Key informant interviews with programme managers during planning/inception phase of the evaluation

Limitations of evaluationVolume of documentation – review was selective

and focused by theory of change and pragmatics of time constraints

Language: some interviews remained at the ‘general’ level Translation in participatory workshop exercises – time

and comprehension

Some partner organisation ‘peer’ evaluation exercises had a different focus from the current evaluation

Focus on ‘what worked’, less on where things didn’t work

Evaluation findingsEvaluation report available (in English and Khmer)

Key concrete things that make for effective networking

Consistent use of participatory methods underpinning programme partner meetings AND personal style of facilitators (former less recognised before).

Multiple and overlapping methods for communication among partners and bringing people together to share experience and reflect

Encouragement of horizontal connections, exchanges and collaboration among partners

Involvement of ‘local champions’ with recognised commitment to the issue

Evaluation findingsKey concrete things that make for effective capacity development

Focus on identified needs of partner organisations

Practical action learning focus of training

Promotion of ongoing reflection on practice and action learning

development of personal relationships, trust and respect (underpinned ownership and responsibility for learning and change)

flexibility in project management

Evaluation findingsProgramme design and management

Variety of ‘ways of working’ in the programme:

flexibility with partner organisations fostered respect regular review and reflection Creativity, having fun and practical at meetings pragmatic orientation of programme staff on helping partner

organisations to achieve ‘their’ objectives layering of networking, capacity development and

communication activities reinforcing each other

Open, flexible orientation consistent across layers of programme management and staff

Programme model emphasising relationships

Evaluation findingsChallenges identified

Language – working more consistently across English and Khmer demands more resources and facilitators

Tools for capacity assessment over time may help for more systematic attention in identified areas of need (with caveats)

Consistent long-term funding (experience of various changes, relatively short programme, being wound up at time of evaluation)

Reflections on processProject staff had a clearer idea of how and why

the programme was working (where it was). Sense of being successful made more concrete and ‘tangible’ – validated.

Partner organisations keen to feed into their learning and future work – specific meeting to discuss and unpack the findings (consistent with learning focus of work and evaluation)

Australian Red Cross used evaluation as a basis for regional reflections on processes of supporting networking and capacity development

Reflections on processPersonal reflection that core elements of the Realist evaluation framework valuable for:

Integrating data of a variety of types Integrating input from varied stakeholders the focus on theory of change as the ‘adjudicator’

(Pawson 2013) helping to navigate complex and social processes that are often poorly understood/articulated

Good for learning AND accountability and integrating quantitative and qualitative, participatory and ‘outsider’ perspectives

Perceived ‘authority’ of method as scientific AND learning focus

Reflections on process Importance of advocates within organisation for evaluation

approach and willingness to engage with complexity

Time constraints and data overload – here TOC helps, providing some ‘boundaries’ of the inquiry.

lack of detailed engagement with context both as ‘setting’ and ‘type of participant’

First articulation and refining of the core theory (s) of change around networking and capacity development that fitted the data gathered.

Humility Realist evaluation really a temporary full stop - needs to be positioned within changing realities of programme AND wider body of cumulative learning.

Need to make the case for the necessary secondary review of literature/practice – relied on existing knowledge of evaluator

Presentation of findingsFindings of the evaluation were presented as a set of short mutually reinforcing sentences aiming to explicitly articulate links between aspects of interventions and the ‘conditions’ that made for success

Networking:

“A combination of platforms for sharing information and knowledge among CIDI partners, including face-to-face and on-line forums mutually reinforced each other to build communication and relationships among CIDI partners”

“Quality and intensity of interactions facilitated the establishment of trust among CIDI partners, providing a foundation for more substantive relationships, joint working and mutual learning among a core of CIDI partner organisations”

Presentation of findings

Capacity development:

“training in core areas of organisational functioning and technical areas of disability inclusion programming responded to the identified needs of partners, which increased its perceived relevance and uptake”

“trained was delivered using participatory learning and practice oriented approaches so that it could be applied to the immediate needs of organisations to embed the learning”

Context, mechanism, outcome configurations..

Realist evaluations often present tables of context, mechanism, outcome configurations

Importance of accounting for articulation of these (context, mechanism and outcomes) in any particular instance (‘Realist Diagnostic workshop’, Pawson 2013 - risk of lack of articulation).

Presentation of CIDI findings showing combination of conditions or ‘active ingredients’(though less attention to context)

Resonance with insights from complexity theory - parallels with notion of ‘configuration of conditions’ and causal factors that lead to outcome patterns in case-based comparative work (Ragin and Rihoux 2009)

Notion of ‘conditions’ rather than ‘context’ (Tilly 1996)

Reflections on causality Causality in social practice as ‘complex and contingent’ (Byrne 1998, 2002, 2005)

and emergent - in nested levels of social reality - with causality flowing in many directions (‘up’ and ‘down’, ‘across’)

Challenge of range of different levels and processes that need to be simultaneously taken account of in understanding social practice

Complexity consistent theory can usefully help to understand and evaluate the nested levels of reality in social change (Westhorpe 2012, 2013, Byrne and Callaghan 2014)

Granularity or level of focus is key for what bodies of theory are relevant, which data to gather and which mechanisms, contexts and outcomes are relevant

Theory (s) of change under primary focus is key for ‘managing’ this, putting boundaries around the inquiry.

‘boundaries of inquiry’ for systems and complexity theorists (Williams and Imam, 2007, Byrne and Vincent 2011, Byrne and Callaghan 2014)

One persons outcome is another’s context….or do I mean mechanism…

Mechanisms of ‘critical realist’ philosophy may operate at a range of different levels

Some generate outcomes that become the ‘context’ for other mechanisms operating at different scales (gender difference generated as an emergent outcome from a range of mechanisms but also acts as a contextual factor)

In the CIDI case the establishing of trust through respectful interactions, was both

an important outcome

and a key condition or contextual factor for practitioners taking ownership of learning and responsibility for organisational change

also in a sense, a key part of how the mechanism of collaboration and mutual learning worked.

Where causality is complex and emergent, distinctions between context, mechanism, outcome may depend on focus of inquiry (or action) – valuable heuristic for analysis

Layers and combinations of cause also important, with feedback and circularity among them in social practice (theories of practice - Bourdieu 1977, Crossley 2011)

Conclusions Realist Evaluation has clear core principles of inquiry useful in time/resource

constrained settings

useful for learning AND summative focused evaluation (if by summative mean interpretation of what the evidence is telling us about outcomes)

Humility about partial picture, but confidence about findings within the focus of the evaluation

Reminder of the need for cumulative learning among body of evaluations – steps towards ‘portable programme theories/reusable conceptual platforms’ useful

Link between Realist informed systematic reviews and evaluations clear (with caveats)

Complementary with complexity theory worth developing further understandings of emergence, recursive causality and complexity consistent theory Complementarity with complex case based methods and ‘configurational’

approach of Qualitative Comparative Analysis QCA, (Ragin and Rihoux 2009, Byrne and Ragin 2009).

Thanks!robvconsult@gmail.com

Input/review from David Curtis, Australian Red Cross

CIDI Evaluation report: http://www.redcross.org.au/files/CIDI_REFLECTION_REPORT-%28En%29_indd.pdf

Vincent (2011) Insights from Complexity theory for the evaluation of development practice http://wiki.ikmemergent.net/files/1203-IKM_Emergent_Working_Paper_14-Complexity_Theory-March_2012.pdf

Vincent (2009) Assessing social and structural change for HIV Prevention: http://e-mops.ning.com/page/insights-from-complexity-theory-for- evaluation

Byrne and Vincent (2011) Evaluating Social Change Communication for HIV/AIDS: New directions. UNAIDS discussion paperhttp://www.unaids.org/en/media/unaids/contentassets/documents/unaidspublication/2009/20111101_JC2250_evaluating-social-change_en.pdf