Upload
chars
View
33
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Thinking systemically: Seeing from simple to complex in impact evaluation. Expert lecture for AfREA Conference March 30 – April 2, 2009Cairo, Egypt. Professor Patricia Rogers Royal Melbourne Institute of Technology, Australia Dr. Irene Guijt Learning by Design, the Netherlands - PowerPoint PPT Presentation
Citation preview
Thinking systemically: Thinking systemically: Seeing from simple to complex Seeing from simple to complex
in in impact evaluationimpact evaluation
Professor Patricia RogersRoyal Melbourne Institute of Technology, Australia
Dr. Irene GuijtLearning by Design, the Netherlands
Bob WilliamsIndependent consultant, New Zealand
(with thanks to Dr. Jim Woodhill, Wageningen International, the Netherlands)
Expert lecture for AfREA Conference Expert lecture for AfREA Conference March 30 – April 2, 2009March 30 – April 2, 2009 Cairo, EgyptCairo, Egypt
TODAY’S SESSIONTODAY’S SESSION• Explore what thinking systemically
is and how it relates to evaluation
• Introduce a systems approach that we think has the potential to move IDE forward, plus give you something you can use in your own practice
• Give you an opportunity to reflect and play with systems ideas and this method.
SYSTEMS CONCEPTS IN EVALUATIONAN EXPERT ANTHOLOGY
Eds. Iraj Imam & Bob Williams
http://www.iigss.net/gPICT.pdf
INTER-RELATIONSHIPS
PERSPECTIVES
BOUNDARIES
THREE ELEMENTS OF THINKING SYSTEMICALLY
INTER-RELATIONSHIPS
Being deeply aware of their significance
Some inter-relationships matter more than most
Some only matter over time
Some are slower in their impact than others
Some are linear (A affects B), some are non-linear and recursive (A affects B which affects A)
Most critically, systems thinking focuses on the inter-relationship of ideas, assumptions, beliefs as well as actions in the traditional cause and effect.
PERSPECTIVES
Thinking systemically about perspectives is not the same as stakeholder analysis
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
PERSPECTIVES
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
PERSPECTIVES We all bring different perspectives to
bear on anything we do. In this workshop I am handling four different perspectives
1. A session where people learn something
2. Something that allows me to communicate my knowledge
3. A means of expressing friendship and support to colleagues,
4. A way of enjoying myself.
You cannot understand how I behave at this session unless you understand how I juggle these perspectives.
QuickTime™ and a decompressor
are needed to see this picture.
BOUNDARIES
BOUNDARIES
Who or what is “in” and who or what is “out”
1. Purpose of evaluation; how will you judge “success”?
2. Resources for evaluation; what is not in your control?
3. What evidence is considered credible, whose expertise is acknowledged, or ignored?
4. Whose or what interests are not being served by an evaluation ?
BOUNDARIES
Thinking systemically requires you to do two things around those boundary decisions;
1.to identify the consequences of boundary decisions
2.consider how to mitigate any negative consequences of boundary decisions.
Tool for thought - the Cynefin FrameworkTool for thought - the Cynefin Framework
Facilitates seeing situational diversity Based on recognizing different types of cause
and effect relations – a given situation will contain aspects of all
Draws on theories of: Complexity Cognitive Systems Narrative Networks
Developed by Dave Snowden – ex-IBM knowledge management researcher
The Cynefin Framework – knowing The Cynefin Framework – knowing what you are dealing withwhat you are dealing with
ComplicatedComplicatedComplexComplex
ChaoticChaotic SimpleSimple
DisorderDisorder
UnorderedUnordered OrderedOrdered
Source: Cognitive Edge (www.cognitive-edge.com)Source: Cognitive Edge (www.cognitive-edge.com)
Ordered Domain – Simple (known)Ordered Domain – Simple (known)
Cause and Effect: repeatable, perceivable, and predictable
Approach: Sense – Categorise - Respond
Methods: Standard operating procedures Best practices Process reengineering
Source: Cognitive Edge (www.cognitive-edge.com)Source: Cognitive Edge (www.cognitive-edge.com)
Evaluating the ‘simple’Evaluating the ‘simple’
Simple aspects of a situation Causal links are tight, clearly observed and
understood Key variables to assess can be determined
For evaluation Need to know activities and some context If activity takes place, outcomes are known (e.g. polio
vaccination once in the person is guaranteed) Monitoring is important
“sense, categorise, respond” But need to guard for slipping into chaos
Ordered domain – Complicated Ordered domain – Complicated (knowable)(knowable)
Cause and Effect Cause-effect knowable with ‘expert’ input
Approach Sense – Analyse - Respond
Methods Analytical/reductionist Results-based thinking Scenario planning Good practices
Source: Cognitive Edge (www.cognitive-edge.com)Source: Cognitive Edge (www.cognitive-edge.com)
Evaluating the ‘complicated’Evaluating the ‘complicated’ Complicated aspects
less predictable, less self-evident, subject to some debate and discussion
usually an evidence base: ‘if A in relation to X under Y conditions, then Z likely’
For evaluation outcome hierarchy/results chain to identify information
needed to understand impact activities, chain(s) of results and assumptions that link them contextual information to explain results consulting ‘experts’ is key - “sense, analyse, respond”
Unordered Domain - ComplexUnordered Domain - Complex Cause and Effect:Cause and Effect:
Coherent in retrospect and do not repeatCoherent in retrospect and do not repeat
Approach:Approach: Probe – Probe – Sense Sense - Respond- Respond
Methods:Methods: Pattern managementPattern management Perspective filtersPerspective filters Circular dialogueCircular dialogue Emergent practiceEmergent practice Source: Cognitive Edge (www.cognitive-edge.com)Source: Cognitive Edge (www.cognitive-edge.com)
Evaluating the ‘complex’Evaluating the ‘complex’ Complex aspects of a situation
unpredictable in advance; no clear understanding of chain of results; highly dependent on context and starting conditions
View as (a set of) experiments and figure out what ‘sticks’: “probe, sense, respond”
For evaluation: observe activities and possible results of those activities,
context fundamental, pathway of decisions to change construct sense-making by drawing people into a dialectic
Unordered Domain - ChaoticUnordered Domain - Chaotic
Cause and Effect: not perceivable
Approach: Act – Sense - Respond
Methods: Stability-focused intervention Crisis management Novel practice
Source: Cognitive Edge (www.cognitive-edge.com)Source: Cognitive Edge (www.cognitive-edge.com)
Evaluating the ‘chaotic’Evaluating the ‘chaotic’
Chaos totally unpredictable; no clear understanding of chain
of results
For evaluation: Observe context, prioritise needs, act, observe again Afterwards (if/when situation stabilises), evaluate if
best possible action under the circumstances was taken (real time evaluation)
“act, sense, respond” - assess worth of “act” and subsequent effects and follow-up actions
Implications for IE PracticeImplications for IE Practice
How do we understand what is happening?
How do we change what is happening?
Intervention is both necessary and sufficient to produce Intervention is both necessary and sufficient to produce the impactthe impact
Impact
Intervention
‘‘Silver bullet’ simple impactsSilver bullet’ simple impacts
No impact
No intervention
Intervention is necessary but not sufficient to produce Intervention is necessary but not sufficient to produce the impactthe impact
Impact
Intervention
‘‘Jigsaw’ complicated impactsJigsaw’ complicated impacts
Favourable context Intervention Unfavourable
context
No impact
Intervention is sufficient but not necessary to produce Intervention is sufficient but not necessary to produce the impactthe impact
Impact
Intervention
‘‘Parallel’ complicated impactsParallel’ complicated impacts
No intervention
Impact
Alternative activity
Plan B3Plan B2
Impact / Vision
Plan B
‘‘Life is a path you beat by walking’ Life is a path you beat by walking’ complex impactscomplex impacts
Intermediate Results (at 1, t+1, t+2)
Plan A
Plan C Plan D Plan D2 Plan E Plan E2
Plan D2
Plan F
ConceptualisingConceptualising the Intervention the Intervention
‘Silver bullet’ impactsintervention is necessary and sufficient to produce impact
Counterfactual approaches comparing similar groups with/without the intervention
‘Ducks lined up’ impactsintervention only produces impact with other things in place
Need to compare with and without full causal package
‘Multiple path’ impactsimpacts achievable by different means
Need to compare alternative ways of achieving impacts
‘Complex’ impacts –achieved by right combination for that particular situation
Need to assess quality and impact of non-standardised interventions
Differentiated study of causality
So what is ‘impact evaluation’?
SIMPLE COMPLICATED COMPLEX
Question answered
What works? What works for whom in what contexts?
What’s working?
Process needed Knowledge transfer
Knowledge translation Knowledge generation
Nature of direction
Single way to do it
Contingent Dynamic and emergent
Metaphor for direction
Written directions
Map and timetable Compass
Take home messagesTake home messages
1. Seeing ‘ontological diversity’ in situations (and interventions) enables a more conscious, appropriate methodologically mixed approach – it’s about being a good professional.
2. Cynefin is just a heuristic - a tool for thinking systemically
3. Thinking systemically is about a deep understanding inter-relationships, perspectives and boundaries. Boundary critique is the area where evaluation can learn most by drawing on the experience of the systems field.
ReferencesReferencesEoyang, Glenda. (2008) So, what about accountability?
http://www.cognitive-edge.com/blogs/guest/2008/12/so_what_about_accountability_1.php Glouberman, S. and Zimmerman, B. (2002) Complicated and Complex Systems:What Would Successful Reform of
Medicare Look Like? Commission on the Future of Health Care in Canada. Discussion Paper 8. Available at http://www.healthandeverything.org/pubs/Glouberman_E.pdf
Guijt, I. (2008). Navigating Complexity. Report of an Innovation Dialogue, May 2008. http://portals.wi.wur.nl/files/docs/Innovation%20Dialogue%20on%20Navigating%20Complexity%20-%20Full
%20Report.pdf Guijt, I. and P. Engel. (2009). ‘Nine Hot Potatoes. Current Debates and Issues in Results-Oriented Practice.’
Presentation for Hivos In-house Training on Reflection-oriented Practice. Mackie, J. (1974). The Cement of the Universe. Oxford University Press, Oxford.Mark MR. 2001. What works and how can we tell? Evaluation Seminar 2. Victoria Department of Natural Resources
and Environment.Rogers, P.J. (2008) ‘Using programme theory for complicated and complex programmes’ Evaluation: the international
jourmal of theory, research and practice. 14 (1): 29-48.Rogers, P.J. (2008) ‘Impact Evaluation Guidance. Subgroup 2’. Meeting of NONIE (Network of Networks on Impact
Evaluation), Washington, DC.Rogers, P.J. (2001) Impact Evaluation Research Report Department of Natural Resources and Environment, Victoria. Ross, H. L., Campbell, D. T., & Glass, G. V (1970). Determining the social effects of a legal reform. In S. S. Nagel
(Ed.), Law and social change (pp. 15-32). Beverly Hills, CA: SAGE.Williams, B. and I. Imam. (2007). Systems Concepts in Evaluation: An Expert Anthology. American Evaluation
Association.Woodhill, J. (2008). The Cynefin Framework: What to do about complexity? Implications for Learning, Participation,
Strategy and Leadership. Presentation for the ‘Navigating Complexity Workshop’, Wageningen International. http://portals.wi.wur.nl/files/docs/File/navigatingcomplexity/CynefinFramework%20final%20%5BRead-Only%5D.pdf