7/28/2019 20090601 Quinn Patton Michael A
1/57
Michael Quinn PattonCES, 2009
Dev elo p m en tal Ev alu at ion with
Michael Quinn Patton
CESJune 1, 2009
1
7/28/2019 20090601 Quinn Patton Michael A
2/57
Michael Quinn PattonCES, 2009
Evolving Understandings
I keep changing what I said. Any person
who is intellectually alive changes hisideas. If anyone at a university isteaching the same thing they wereteaching five years ago, either the fieldis dead, or they havent been thinking.
Noam Chomsky
The Professor Provaocateur, The New York Times Magazine, Nov. 2, 2003: 13.2
7/28/2019 20090601 Quinn Patton Michael A
3/57
Michael Quinn PattonCES, 2009
Interpretive Frameworks
May 2003 Harvard Business Review "The
High Cost of Accuracy." Kathleen Sutcliffeand Klaus Weber.
They concluded that " the way senior executives interpret their business
environment is more important for performance than how accurately theyknow their environment ."
3
7/28/2019 20090601 Quinn Patton Michael A
4/57
Michael Quinn PattonCES, 2009
They further concluded that it is a waste of
resources to spend a lot of moneyincreasing the marginal accuracy of dataavailable to senior executives compared tothe value of enhancing their capacity tointerpret whatever data they have .
Executives were more limited by a lack of capacity to make sense of data than byinadequate or inaccurate data.In essence, they found that interpretivecapacity, or "mind-sets," distinguish high-
performance more than data quality andaccuracy. 4
7/28/2019 20090601 Quinn Patton Michael A
5/57
Michael Quinn PattonCES, 2009
Original Primary Options
Formativeand
SummativeEvaluation
(Mid-term and End-of-Project Reviews)
5
7/28/2019 20090601 Quinn Patton Michael A
6/57
Michael Quinn PattonCES, 2009
ImprovementversusDevelopment
6
7/28/2019 20090601 Quinn Patton Michael A
7/57
Michael Quinn PattonCES, 2009
Evidence-based Practice
Evaluation grew up in the projects
testing models under a theory of change that pilot testing would lead toproven models that could bedisseminated and taken to scale :
The search for b es t p rac t ices and ev id en c ed -b as ed p rac t ices
7
7/28/2019 20090601 Quinn Patton Michael A
8/57
Michael Quinn PattonCES, 2009
Fundamental Issue:How the World Is Changed
Top-down dissemination of
proven modelsversus
Bottoms-up adaptivemanagement
8
7/28/2019 20090601 Quinn Patton Michael A
9/57
Michael Quinn PattonCES, 2009
Models vs. Principles
Identifying proven principles for adaptive management
(bottoms-up approach)versus
Identifying and disseminatingproven models
(top down approach)9
7/28/2019 20090601 Quinn Patton Michael A
10/57
Michael Quinn PattonCES, 2009
Conditions that challenge traditional
model-testing evaluation
High innovation Development
High uncertainty Dynamic
Emergent Systems Change
AdaptiveManagement
10
7/28/2019 20090601 Quinn Patton Michael A
11/57
Michael Quinn PattonCES, 2009
Mintzberg on StrategyTwo types of strategy: Intended & Emergent
Unrealized
StrategyIntended
Strategy DeliberateStrategyRealized
Emergent StrategyStrategy
11
7/28/2019 20090601 Quinn Patton Michael A
12/57
Michael Quinn PattonCES, 2009
Re-conceptualizing Use
Use is a process not a event Use involves an interaction not just
a report Use involves training for use not just delivery of results
Use is a leadership function
12
7/28/2019 20090601 Quinn Patton Michael A
13/57
Michael Quinn PattonCES, 2009
Some Evaluation Premises : Evaluation is part of initial program design, including
conceptualizing the theory of change Evaluators role is to help users clarify their purpose,
hoped-for results, and change model. Evaluators can/should offer conceptual and
methodological options. Evaluators can help by questioning assumptions. Evaluators can play a key role in facilitating
evaluative thinking all along the way. Interpretative dialogue is critical. Designs can be emergent and flexible.
13
7/28/2019 20090601 Quinn Patton Michael A
14/57
Michael Quinn PattonCES, 2009
Contingency-based
Evaluation Situational analysis & responsiveness
Context sensitivity Clarify and focus on intended users:
stakeholder analysis Clarify and focus on intended uses Methodological appropriateness Criteria for evaluating the evaluation:
credibility, meaningfulness
14
7/28/2019 20090601 Quinn Patton Michael A
15/57
Michael Quinn PattonCES, 2009
Seeing Through AComplexity Lens
You don't see something until you have the right metaphor to let you perceive it. Thomas Kuhn 15
7/28/2019 20090601 Quinn Patton Michael A
16/57
Michael Quinn PattonCES, 2009
Get t in g to May b e:
How th e World Is Changed? 2006
Frances Westley, BrendaZimmerman, Michael Q. Patton
Random House Canada ,
16
7/28/2019 20090601 Quinn Patton Michael A
17/57
Michael Quinn PattonCES, 2009
Conceptual Options
Simple
Complicated
Complex
17
7/28/2019 20090601 Quinn Patton Michael A
18/57
Michael Quinn PattonCES, 2009
Types of Community IssuesThe Certainty/Agreement Matrix
CertaintyClose to Far from C l o s e
t o
18
7/28/2019 20090601 Quinn Patton Michael A
19/57
7/28/2019 20090601 Quinn Patton Michael A
20/57
Michael Quinn PattonCES, 2009
Michael Quinn Patton
Simple (Known arena of action) Tight, centralized connections. Can identify and make sense of patterns. Linear cause and effect.
Best practices identifiable within the currentcontext (which of course may not beself-evident or known to others henceimportance of context).
20
7/28/2019 20090601 Quinn Patton Michael A
21/57
Michael Quinn PattonCES, 2009
Following a Recipe A Rocket to the Moon Raising a Child
Complicated Complex
] The recipe is essential
]Recipes are tested toassure replicability of
later efforts
] No particular
expertise; knowing howto cook increasessuccess
] Recipes producestandard products
] Certainty of sameresults every time
Simple
21
7/28/2019 20090601 Quinn Patton Michael A
22/57
Michael Quinn PattonCES, 2009
Michael Quinn Patton
Complicated (Knowable arena)Relationships are looser but still
clustered around a central core.Cause and effect is dynamic, multi-
dimensional, and enmeshed in
system relationships. System relationships can be
modelled and understood.Expertise and coordination needed.
22
7/28/2019 20090601 Quinn Patton Michael A
23/57
Michael Quinn PattonCES, 2009
Following a Recipe A Rocket to the Moon Raising a
Child Formulae arecritical andnecessary
Sending one rocketincreasesassurance that nextwill be ok
High level of expertise in manyspecialized fields +coordination
Rockets similar incritical ways
High degree of certainty of outcome
Complicated Complex
] The recipe is essential
] Recipes are tested toassure replicability of later efforts
] No particularexpertise; knowing howto cook increasessuccess
] Recipes producestandard products
] Certainty of sameresults every time
Simple
23
7/28/2019 20090601 Quinn Patton Michael A
24/57
Michael Quinn PattonCES, 2009
Socially complicated
Implementing human rights
agreements, like gender equity or outlawing child labor
Environmental InitiativesMany different and competingstakeholdersDiverse vested interestsHigh stakes
24
7/28/2019 20090601 Quinn Patton Michael A
25/57
Michael Quinn PattonCES, 2009
Socially complicatedsituations
pose the challengeof coordinating and
integrating
many players
25
7/28/2019 20090601 Quinn Patton Michael A
26/57
Michael Quinn PattonCES, 2009
Stakeholder MappingHigh Interest/
Low Power
THE INVOLVED
High Interest/High Power
THE PLAYERS
THE CROWD
Low interest/Low Power
CONTEXT SETTERS
Low Interest/High Power
26
7/28/2019 20090601 Quinn Patton Michael A
27/57
Michael Quinn PattonCES, 2009
Michael Quinn Patton
ComplexCentre is loosely connected to network.Cause effect difficult to track; nonlinear,
interdependent relationshipsHighly context dependent.Outcomes emergent, not predictable.
27
7/28/2019 20090601 Quinn Patton Michael A
28/57
Michael Quinn PattonCES, 2009
Following a Recipe A Rocket to the Moon Raising a
Child
Sending one rocketincreasesassurance that nextwill be ok
High level of expertisein many specializedfields +coordination
Rockets similar incritical ways
High degree of
certainty of outcome
Formulae have onlya limited application
Raising one childgives no assuranceof success with thenext
Expertise can helpbut is not sufficient;relationships arekey
Every child isunique
Uncertainty of outcome remains
Complicated Complex
] The recipe is essential
] Recipes are tested toassure replicability of later efforts
] No particularexpertise; knowing howto cook increasessuccess
] Recipes producestandard products
] Certainty of sameresults every time
Simple
28
7/28/2019 20090601 Quinn Patton Michael A
29/57
Michael Quinn PattonCES, 2009
Complex Nonlinear Dynamics Nonlinear: Small actions can have large
reactions. The Butterfly Wings Metaphor Emergent: Self-organizing, Attractors Dynamic: Interactions within, between,
and among subsystems and parts withinsystems can volatile, changing
Getting to Maybe: Uncertainty,unpredictable, uncontrollable
29
C li d C l
7/28/2019 20090601 Quinn Patton Michael A
30/57
Michael Quinn PattonCES, 2009
Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical
and necessary
Sending one rocketincreases assurancethat next will be ok
High level of expertise in manyspecialized fields +coordination
Separate into partsand then coordinate
Rockets similar incritical ways
High degree of certainty of outcome
Formulae have only alimited application
Raising one childgives no assurance of success with the next
Expertise can helpbut is not sufficient;relationships arekey
Cant separate partsfrom the whole
Every child is unique Uncertainty of
outcome remains
Complicated Complex
] The recipe is essential
] Recipes are tested toassure replicability of later efforts
] No particularexpertise; knowing howto cook increasessuccess
] Recipe notes thequantity and nature of
parts needed
] Recipes producestandard products
] Certainty of sameresults every time
Simple
30
7/28/2019 20090601 Quinn Patton Michael A
31/57
Michael Quinn PattonCES, 2009
A Leader's Framework for DecisionMaking by David J. Snowden andMary E. Boone, Harvard B us iness Review,
November, 2007:Wise executives tailor their approach to fit
the complexity of the circumstancesthey face.
31
7/28/2019 20090601 Quinn Patton Michael A
32/57
Michael Quinn PattonCES, 2009
Example
The McGill-McConnell Leadership
Program ExampleSimple elements
Complicated elements
Complex elements
32
7/28/2019 20090601 Quinn Patton Michael A
33/57
Simple outcomes
Increase knowledge and skills of
participants
Evaluation: Pre-post data anddocumentation of learning
Michael Quinn PattonCES, 2009
33
7/28/2019 20090601 Quinn Patton Michael A
34/57
Complicated Impacts
Change participants organizations
Evaluation:
Case studiesof
organizational change
Michael Quinn PattonCES, 2009
34
7/28/2019 20090601 Quinn Patton Michael A
35/57
Complex Vision
Infuse energy into the moribund not-
for-profit (voluntary) sector Make the sector more dynamic
Create network of leaders whoactively engage in change
Michael Quinn PattonCES, 2009
35
7/28/2019 20090601 Quinn Patton Michael A
36/57
Evaluating the Complex
Real time follow-up of network
connections and actions Follow-up is an intervention
Rapid feedback of findings permitsinfusion of resources in support of emergent outcomes
Michael Quinn PattonCES, 2009
36
7/28/2019 20090601 Quinn Patton Michael A
37/57
Michael Quinn PattonCES, 2009
Process Use
Infusing ev aluat ive th in k ing as a
primary type of process use.
Capacity-building as anevaluation focus of
process use .
37
7/28/2019 20090601 Quinn Patton Michael A
38/57
Paradigms and Lenses
The importance of interpretive
frameworks Complexity as an interpretive
framework
Michael Quinn PattonCES, 2009
38
7/28/2019 20090601 Quinn Patton Michael A
39/57
Michael Quinn PattonCES, 2009
Complex Situations Highly emergent (difficult to
plan and predict) Highly dynamic, rapidly
changing Relationships are
interdependent and non-linear rather than simple and linear (cause-effect) 39
7/28/2019 20090601 Quinn Patton Michael A
40/57
Michael Quinn PattonCES, 2009
Contingency-basedDevelopmental
Evaluation
40
7/28/2019 20090601 Quinn Patton Michael A
41/57
DEVELOPMENTAL EVALUATION
DEFINEDEvaluation processes, including asking evaluative
questions and applying evaluation logic, to support program, product, staff and/or organizationaldevelopment . The evaluator is part of a team whose
members collaborate to conceptualize, design and testnew approaches in a long-term, on-going process of continuous improvement, adaptation and intentional
change . The evaluator's primary function in the teamis to elucidate team discussions with evaluativequestions, data and logic, and facilitate data-based
decision-making in the developmental process.Michael Quinn Patton
CES, 200941
7/28/2019 20090601 Quinn Patton Michael A
42/57
Michael Quinn PattonCES, 2009
Other names
Real time evaluation Emergent evaluation Action evaluation Adaptive evaluation
42
7/28/2019 20090601 Quinn Patton Michael A
43/57
Michael Quinn PattonCES, 2009
CONTRASTS
Traditional
evaluations Testing models
Complexity-based,
DevelopmentalEvaluation
Supportinginnovation andadaptation
43
7/28/2019 20090601 Quinn Patton Michael A
44/57
Michael Quinn PattonCES, 2009
TraditionalEvaluation
Render definitive
judgments of success or failure
DevelopmentalEvaluation
Provide feedback,generate learnings,support directionor affirm changesin direction in real
time
44
7/28/2019 20090601 Quinn Patton Michael A
45/57
Michael Quinn PattonCES, 2009
TraditionalEvaluation
Render definitive judgments of success or failure
Measure success
againstpredetermined goals
DevelopmentalEvaluation
Provide feedback, generatelearnings, support direction or affirm changes in direction
Develop newmeasures andmonitoringmechanisms asgoals emerge &evolve
45
7/28/2019 20090601 Quinn Patton Michael A
46/57
Michael Quinn PattonCES, 2009
Traditional Developmental
Evaluation Evaluation Evaluator external,
independent,objective
Evaluator part of a
team, a facilitator and learning coachbringing evaluative
thinking to the table,supportive of theorganizations goals
46
7/28/2019 20090601 Quinn Patton Michael A
47/57
Michael Quinn PattonCES, 2009
Traditional Developmental
Evaluation Evaluation Evaluator
determines thedesign based on theevaluators
perspective aboutwhat is important.The evaluator
controls theevaluation.
Evaluator
collaborates withthose engaged inthe change effort to
design anevaluation processthat matches
philosophically andorganizationally.
47
7/28/2019 20090601 Quinn Patton Michael A
48/57
Michael Quinn PattonCES, 2009
Traditional Developmental
Evaluation Evaluation Design the
evaluation based onlinear cause-effectlogic models
Design the
evaluation tocapture systemdynamics,
interdependencies,and emergentinterconnections
48
7/28/2019 20090601 Quinn Patton Michael A
49/57
Michael Quinn PattonCES, 2009
Traditional Developmental
Evaluation Evaluation Aim to produce
generalizablefindings across time& space .
Aim to produce
context-specificunderstandings thatinform ongoing
innovation
49
7/28/2019 20090601 Quinn Patton Michael A
50/57
Michael Quinn PattonCES, 2009
Traditional Developmental
Evaluation Evaluation Accountability
focused on anddirected to externalauthorities and
funders.
Accountability
centered on theinnovators deepsense of
fundamental valuesand commitments and learning.
50
7/28/2019 20090601 Quinn Patton Michael A
51/57
Michael Quinn PattonCES, 2009
Traditional Developmental
Evaluation Evaluation Accountability to
control and locateblame for failures
Learning to respond
to lack of controland stay in touchwith whats
unfolding And thereby
responds t ra tegical ly
51
7/28/2019 20090601 Quinn Patton Michael A
52/57
7/28/2019 20090601 Quinn Patton Michael A
53/57
Michael Quinn PattonCES, 2009
Traditional Developmental
Evaluation Evaluation Evaluation
engendersfear o f fa i lu re .
Evaluation supports
hu ng er for l earn ing .
53
7/28/2019 20090601 Quinn Patton Michael A
54/57
Michael Quinn PattonCES, 2009
Conditions
High innovation Development
High uncertainty Dynamic
Emergent
54
7/28/2019 20090601 Quinn Patton Michael A
55/57
Michael Quinn PattonCES, 2009
Challenge:Matching the evaluationprocess and design to thenature of the situation:
Contingency-based
Evaluation
55
7/28/2019 20090601 Quinn Patton Michael A
56/57
Michael Quinn PattonCES, 2009
References
Getting to Maybe: How the World Is Changed?
Frances Westley, Brenda Zimmerman, MichaelQ. Patton, Random House Canada, 2006
Utilization-Focused Evaluation , 4 th ed.,Michael Quinn Patton, Sage, 2008.
Developmental Evaluation , Michael Quinn Patton,Guilford Press, forthcoming.
56
7/28/2019 20090601 Quinn Patton Michael A
57/57
Resources
The J.W. McConnell Foundation onDevelopmental Evaluation
http://www.mcconnellfoundation.ca/default.aspx?page=139
h l