Embed Size (px)
Citation preview
Transformative Evaluation Capacity Building
Jean A. King2017 Minnesota Evaluation Studies Institute
Goals for the morning
1. Understand definitions, the rationale for ECB, and its underlying assumptions
2. Outline how to plan and implement the components of effective ECB (i.e., how to build the evaluation capacity of an organization)
3. Know how to recognize and overcome likely challenges to this work
WHO IS IN THIS SESSION TODAY?Continuing the MESI tradition of connecting. . .
1. WHAT IS TRANSFORMATIVE ECB? WHY ECB? WHY NOW?
A quick discussion of rationale, assumptions, and terms
A distinction to remember
•Evaluation capacity (EC)- the outcome
•Evaluation capacity building (ECB)- the process
Definitions so we’re clear:Evaluation capacity
“The competencies and structures required to conduct high quality evaluation studies
(capacity to do), as well as the organization’s ability to integrate evaluation
findings into its decision-making process (capacity to use)”
Bourgeois, Whynot, & Thériault (2015, p. 47)
Another name for evaluation capacity: Free range evaluation
Definitions so we’re clear:Evaluation capacity building
Intentional work to increase the ability of an organization to conduct and use
evaluation
Review: What, exactly, is evaluation capacity building (ECB)?
Creating an evaluation system
both for
(1) its results
AND
(2) the explicit purpose of building people’s capacity to evaluate again
For the ECB practitioner, the focus
“. . .[is] on responding to requests for evaluation services while simultaneously considering how today’s work will contribute to sustaining the unit in the longer term”
-Compton, Glover-Kudon, Smith, & Avery, 2002, p. 55
Two overarching goals of ECB
1. Increase an organization’s capacity to DO evaluation• Design, implement, and manage
effective evaluation projects
• Access, build, and use evaluative knowledge and skills
Two overarching goals of ECB
2. Increase an organization’s capacity to USE evaluation• Create support for program
evaluation as a performance improvement strategy
• Cultivate a spirit of continuous organizational learning, improvement, and accountability
What assumptions underlie ECB?
•People really want to conduct program evaluation/value evaluation
•Their context makes ECB possible (centralized and decentralized control)
•They are able to: • Learn the skills (diversity, redundancy)
•Use the results (who, how)
•They are willing to devote resources to it
•They are committed for the long term
Where did ECB come from?
•Process use (Patton, 1997)
•Developmental evaluation (Patton,
2007)
•Participatory evaluation (Cousins and
colleagues. . .)
•Organizational learning (Preskill &
Torres, 1999)
•Evaluation capacity building (Baizerman, Compton, & Stockdill, 2002)
We did it; Patton named it
Process use- the purposeful use of
the evaluation process to teach
evaluative inquiry
Patton also added another incredibly helpful idea
Developmental evaluation
“Evaluation processes, including asking evaluative questions and applying evaluation logic, to support program, product, staff and/or organizational development” (Patton, 2007)
“. . . a long-term, on-going process of continuous improvement, adaptation and intentional change”
“Participatory evaluation”
Brad Cousins and his colleagues’ ongoing formal
research agenda
Organizational learning
Literature review
•An emergent and growing literature on evaluation capacity building
•A lack of practical tools for effective integration of program evaluation in organizations
•A reason to draw on other disciplines (organization development, total quality management, adult learning, and social psychology)
Three ECB frameworks to considerOn handout:
1. Preskill and Boyle (2008) Multidisciplinary Model of ECB
2. Labin, Duffy, Meyers, Wandersman, and Lesesne (2012) Integrative Evaluation Capacity Building Model
Next in this PPT
3. King and Stevahn (2013) ECB Continuum
The ECB continuum addresses the relationship between. . .
•The evaluator
•The organization in which the evaluation occurs
Evaluation capacity building (ECB) continuum
Formative / Summative
evaluation study
Evaluation specifically for
building capacity to evaluate
Evaluation for organization development
←----------------------------------------------------------------------→
Use of single studyprocess / results
ECB = creating capacity to
conduct evaluations
Capacity to sustain change
Back to the rationale:Why ECB? Why now?
•Rising accountability demands
•The cost of evaluation
•The availability of technology
•The value (and fun) of the ECB process
Why evaluation capacity building?
From the inside out
Systematize people’s natural
evaluative instincts and
processes
From the outside in
Use what we evaluators have learned to foster
an organic evaluation
process
How can ECB become transformative?
•Key question: Can it?
•What can evaluators transform?
•What can program leaders/staff transform?
Cousins & Whitmore’s model for collaborative inquiry
What do you think?
What is the [realistic]
transformative potential of evaluation
capacity building?
2. HOW TO BUILD (TRANSFORMATIVE) EVALUATION CAPACITY: DO AND USE
Literally bringing ECB to life
How to build evaluation capacity
To do evaluation
To use evaluation
But first--initial steps to consider
1. Assess the organizational context
2. Identify people to work with
Examine the organizational context
Internal
•Environment that supports change
• Supportive leadership
•Evaluation champions
• Interest in and demand
for evaluation
• Input into decisions
• Socializing common
External
•Environment that supports change
•External mandates for accountability
•Expectation of good stewardship
Ways to assess the organization’s evaluation ecosystem (1/2)
•Analysis of existing documents or data •Climate survey/evaluation needs assessment • Shared value for evaluation?• Centralized/decentralized control?• Existing evaluation skills?
•Interviews (individual, group, or focus group) with key informants
Ways to assess the organization’s evaluation ecosystem (2/2)
•Within the organizationoForce field analysisoPower mapping
Figure out whom will you work with•Remember the personal factor- Identify evaluation champions who will nurture evaluative thinking in themselves and others
•Find individuals who understand or intuitively “get” evaluation • “Evaluation nerds”
• “Evaluation enthusiasts”
•Create an advisory group to manage and monitor the capacity-building process
Potential advisory group members• The evaluation champions
• Highly respected staff who know the organization’s people and culture well
• Competent people who can get tasks done, regardless of the challenges
• At least one person with (1) a sense of humor and (2) the ability to keep the capacity-building process in historical context
Advisory group questions to consider
•How big a group is big enough?
•Should you include the nay-sayers?
•How are end-users/clients involved in the evaluation capacity building?
How to build evaluation capacity
To do evaluation
To use evaluation
Process use as an ECB tool
∙ Become purposeful about your role as the facilitator of process use∙ Make sure you have done your initial
work:∙ Assess the organizational context ∙ Identify and support evaluation champions
∙ Work to construct an evaluation process and structures to support continuing evaluative thinking
Building the capacity to DO evaluation
1. Needed infrastructure
2. Required resources
3. Your role and possibilities as an ECB practitioner
1. Evaluation infrastructure
•A purposeful ECB plan for the organization
•Infrastructure to support the technical evaluation process
•Purposeful socialization into the organization’s evaluation process
A purposeful ECB plan for the organization
•An evaluation advisory/ oversight group
•An appropriate con-ception of evaluation written into policies & procedures
•A written ECB document
Infrastructure to support the technical evaluation process
•Question framing
•Measuring needs
•Evaluation design
•Data collection, analysis, and interpretation
•Internal reporting Effective communication plan
Purposeful socialization into the organization’s evaluation process
•Clear expectations of roles
•Incentives for participation (What’s in it for them?)
•Formal training/professional development in evaluation
•Opportunities to learn evaluation by doing it
Incentives for participation
•What are the reasons people don’t evaluate?
•How might incentives address these deterrents?
•Do you have to pay people to engage in evaluative thinking?
2. Required resources (1/2)
•Formal training/professional development opportunities
•Coaching, both formal & informal
•Personnel, internal and external
•Relevant research bases
•Information on evaluation resources
Required resources (2/2)
Sources of support for evaluation in the organization, including:
•Fiscal support from the administration
•Time within the work day
•Basic resources
•Revenue-generating strategies to support evaluation (perhaps)
3. Important ECB roles
•Group leader
•Teacher
•Cheerleader
•Coach
•Technical expert
•Critical friend
What power do evaluators have?
• As the moon reflects the sun,
an evaluator reflects the
power of another
• Our so-called “power” is
bounded, typically a function
of our relationships with
those who hold positional
authority
ECB leadership challenges
•You don’t have positional power
•BUT PEOPLE THINK YOU DO!
•You always always have to be nice
•EVEN WHEN OTHERS ARE NOT!
Possibilities of evaluator roles
•Technical expert on research design, measurement, statistics
•Facilitator of group interaction
•Coach of others doing their own evaluations
•Others. . .
Interpersonal Participation Quotient (IPQ)
LOW
HIGHEvaluator
Programleaders,
staff, community members
Involv
emen
t in
decis
ion m
aking
an
d imp
lemen
tation
Participant-directed
CollaborativeEvaluator-directed
ZONES
Types of participant involvement
•Mere awareness
•Passive support or minimal participation
•Active participation in the evaluation process
•Commitment to consider and ultimately use the evaluation results
Discussion question
Consider the different roles for evaluator and potential user for each of the three zones.
How might these different roles play out in the development of evaluation capacity?
Types of ECB studies
•Evaluator-led (modeling the process)
•Co-led (guiding practice, co-laboring, sharing responsibilities)
•“Them-led” (helping them succeed, maintaining evaluation standards)
Types of ECB studies (visually)
56
CollaborativeEvaluator-directed
Participant-directed
Different approaches to ECBEvaluator-directed
MODELING
• Let people see how evaluation is done
• Build a model for formal program evaluation that people can understand
• A comfortable way to relinquish control and build trust
Collaborative
TEACHING
• Model shared decision making, talking through evaluative decisions
• Teach specific evaluation skills explicitly
• Create working relation-ships and build trust
Different approaches to ECBParticipant-directed
COACHING
• Good way to see what evaluative strengths exist in the organization
• Opportunity for people to learn evaluation by doing evaluation
• Unique features/process may emerge (appropriate to organization)
WHAT WOULD WORK
IN YOUR CONTEXT?
Consider:
Which of the approaches might
work for which parts of ECB in your setting?
A helpful parallel (perhaps?)
Evaluator-directed
Collabor-ative
Participant-directed
Evalua-tion
Evaluator conducts
study(modeling)
“Guided evaluation”(teaching)
Supported practice
(coaching)
Teach-ing
reading
Teacher reads
aloud to children
“Guided reading”
Indepen-dent reading
How to build evaluation capacity
To do evaluation
To use evaluation
Building the capacity to USE evaluation: Involve people
1. Purposeful socialization (a repeated point)
2. Functional peer learning structures
The general process of evaluation
62
U-S-I-N-G T-H-E- P-R-O-C-E-S-S A-N-D R-E-S-U-L-T-S
Planning▪Develop object description▪Assess context▪Plan evaluation study
Collecting data
Analyzing/interpreting data
“Reporting” the results
Immediate impact
Long-term impact
Four factors affecting evaluation use1. Relating to primary users of the evaluation •MQP’s “personal factor"• Positive attitude toward evaluation • Stakeholder model & its extensions (e.g., participatory evaluation)
2. Relating to the associated organization •Organizational culture and politics• External context • "Clout factor"
63
Four factors affecting evaluation use3. Relating to the evaluator
•Technical skills
•Personal characteristics (King & Stevahn’s “interpersonal factor")
4. Relating to the evaluation process
•"Evaluation quality"
•Timing and timeliness 64
Involving people helps increase use1. Understand the context better and tailor
the evaluation to it2. Create more meaningful participation3. Increase ownership of the evaluation
process4. Increase the utility of results and the
likelihood of use5. Have a good time, even have fun
BUT what about potential bias and threats to “objectivity”?
In other words. . .
“. . . the first step in helping to assure evaluation use. . . and
that your work does not go for naught, is to establish
relationships with stakeholders and for you to
think about and focus on evaluation use throughout
the process” (Alkin, 2011, pp. 205-206)
1. Again, purposeful socialization into the evaluation process
•Clear expectations of roles
•Incentives for participation (What’s in it for them?)
•Formal training/professional development in evaluation
•Opportunities to learn evaluation by doing it
2. Peer learning structures
•Purposeful trust building
•An effective communication system and feedback mechanisms
•Interdependent evaluation roles
•Opportunities for reflection
•Physical proximity (if possible)
3. CHALLENGES TO BUILDING EVALUATION CAPACITY
What can possibly go wrong?
Potential barriers to ECB
1. No resources (time, money especially)
2. Everyone’s an evaluator (“We’re already doing it”)
3. Lack of interest (“We’re overwhelmed”)
4. Lack of evaluation skills and confidence
5. Need for trust and openness
6. All change is hard (BTW)
7. Sustainability
Review: Goals for the morning
1. Understand the rationale for ECB, its underlying assumptions, and definitions
2. Outline how to plan and implement the components of effective ECB (i.e., how to build the evaluation capacity of an organization)
3. Know how to recognize and overcome likely challenges to this work
