Duffy ECB Working Paper

8/3/2019 Duffy ECB Working Paper

1/68

Evaluation Capacity 1

Running head: EVALUATION CAPACITY BUILDING

Theory and Research on Evaluation Capacity Building

Jennifer L. Duffy

University of South Carolina


2/68


Theory and Research on Evaluation Capacity Building

The use of participatory, collaborative and empowerment approaches to

evaluation is a growing area of interest in within the field, in part in order to increase the

utilization of evaluations conducted (Preskill, Zuckerman, & Matthews, 2003).

Empowerment evaluation in particular emphasizes the importance of building capacity

for evaluation, and suggests that the community should own the evaluation (including

making decisions about evaluation questions and design) and should also learn how to

conduct their own evaluations (Wandersman et al., 2005). However, while interest in this

area is growing (as is suggested by a growing number of recent publications and the

recently started topical interest area in the American Evaluation Association), evaluation

capacity building (ECB) has only recently been distinguished as an area of work distinct

from the work of carrying out evaluation (Baizerman, Compton, & Stockdill, 2002a).

The purpose of this paper is to synthesize research and theory relevant to ECB,

not a particular approach to evaluation or evaluation theory. It is important to keep in

mind that while certain schools of evaluation (particularly collaborative, participatory,

and empowerment evaluation) explicitly make ECB a part of their work to varying

degrees, ECB is not limited to a particular brand of evaluation. ECB can be applied to

more traditional forms of evaluation as well, particularly ECB strategies that focus on

increasing demand for and utilization of evaluation (as has been done in international

ECB work). However, while ECB is not necessarily tied to collaborative, participatory, or

empowerment evaluation approaches, the majority of the research identified has

approached ECB from these perspectives.


3/68


In the following sections, definitions and theoretical issues relevant to ECB are

described, methodological challenges to research on ECB are identified, and research on

ECB is reviewed. In the final section, theoretical issues related to ECB will be

reexamined in light of the research reviewed, and further questions for research on ECB

will be identified.

Definitions of Key Concepts

This paper begins with a review of definitions of evaluation capacity and

evaluation capacity building. The concepts being examined within this paper are

complex, abstract, and cut across multiple disciplines, making the specification of the

definitions used particularly important.

Defining Evaluation Capacity

Several definitions of evaluation capacity are offered in the literature. Boyle,

Lemaire, and Rist (1999) define evaluation capacity as the human capital (skills,

knowledge, experience, etc.) and financial/material resources necessary for doing

evaluation (p. 5). Milstein & Cotton (2000) define evaluation capacity as the ability to

conduct an effective evaluation (i.e. one that meets accepted standards of the discipline)

(p. 1). They also propose a framework that includes five main elements of evaluation

capacity: motivational forces, organizational environment, workforce or professional

development, resources and support, and learning from experience. Gibbs, Napp, Jolly,

Westover, & Uhl (2002) define evaluation capacity as the extent to which a CBO has the

necessary resources and motivation to conduct, analyze, and use evaluations (p. 261).

The General Accountability Office (2003) defines evaluation capacity within federal

agencies as the ability to systematically collect, analyze, and use data on program


4/68


results and identifies key components of that ability as an evaluation culture, access to

high quality data and evaluation expertise, and collaborative partnerships with state and

local agencies who carry out the agencies work.

These definitions vary in the level of analysis to which they are applied. These

levels of analysis have implications for how ECB is defined and understood. Boyle and

colleagues (1999) definition is not anchored to a specific level; it can be applied to

individual-level evaluation capacity, or the capacity of a team, organization, or broader

system (like a regional or national government). Elements of the Milstein & Cotton

(2000) definition also can be applied across different levels, but the focus on elements of

the organizational environment suggest the necessity of going beyond the individual

level. Gibbs et al. (2002) explicitly place evaluation capacity at the organizational-level.

The basic definition provided by GAO is applicable across multiple levels, but some of

the specific elements of evaluation capacity identified are specific to the work done by

federal agencies. Each of these definitions includes resources, and two definitions also

explicitly include motivation as an aspect of evaluation capacity.

Arnold (2006) suggests that in understanding the evaluation capacity of the

individuals within an organization is necessary for understanding the evaluation capacity

of the organization. She suggests that individual evaluation capacity may be viewed on a

continuum ranging from doubters (who are skeptical of the value of evaluation) to

evaluation scholars (who are experts on evaluation). According to this framework,

organizational capacity for evaluation is reflected by an organization which has many

evaluation practitioners, with few doubter or scholars.

Defining Evaluation Capacity Building


5/68


Definitions ofevaluation capacity building found in the literature tend to be

complex, multifaceted, and laden with jargon. In addition, some literature (particularly in

the international development field) refers to evaluation capacity developmentand in

some cases use it interchangeably with ECB. Schaumberg-Muller (1996) used a broad

definition of ECB in a survey of international development donor agencies, including:

activities which provide support for systems of evaluation, audit, feedback and learning

from policies, programmes or projects performed at various levels it excludes activities

aimed solely at planning and appraisal activities (p. 5). The most common ECB

activities identified on that survey included training, technical assistance, and the

provision of equipment.

Boyle et al. (1999) do not define ECB, but do define evaluation capacity

development as activities and initiatives taken to implement an evaluation regime and

define evaluation regime as the configuration of evaluation capacity, evaluation practice,

organizational arrangements, and institutionalization within a national or regional

government (p. 6). In other words, Boyle et al. define evaluation capacity development as

activities intended to promote evaluation capacity and practice within governments,

including attending to where evaluation is placed within the government and

whether/how evaluation is institutionalized.

The editors of a special issue of an evaluation journal dedicated to the explication

of ECB developed a complex conceptual definition of ECB based upon their knowledge

and the available literature in the area (Baizerman, Compton, & Stockdill, 2002a;

Stockdill, Baizerman, & Compton, 2002). This conceptual definition is quite complex

and contains 11 distinct elements. Stockdill et al. define ECB as a context-dependent,


6/68


intentional action system of guided processes and practices for bringing about and

sustaining a state of affairs in which quality program evaluation and its appropriate uses

are ordinary and ongoing practices within and/or between or more

organizations/programs/sites (p. 8). While Stockdill et al. state that each of these 11

elements is important in helping to distinguish ECB from evaluation practice, they also

provide a simplified working definition of ECB: ECB is the intentional work to

continuously create and sustain overall organizational processes that make quality

evaluation and its uses routine (p. 14).

These definitions of ECB (and evaluation capacity development) share several

element. Each identifies ECB as an activity separate from actually carrying out

evaluations. In addition, Stockdill et al. (2002) and Boyle et al. (1999) both note the

importance of institutionalizing and evaluation practice within organizations. There are

also some key distinctions between the definitions. The conceptual definition provided by

Stockdill et al. focuses on the importance of sustaining the use of evaluation over time,

and emphasizes the importance of considering the context where the ECB is to take

place. Boyles definition is focused on building evaluation capacity within governmental

contexts, and as such it focuses on the importance of examining the legal authorization

for evaluation and which branch of government has responsibility for evaluation

activities. In contrast, the definition provided by Stockdill and colleagues focuses on the

organizational level, though it is also applicable to ECB that covers multiple

organizations. None of these definitions explicitly mentions building evaluation capacity

at the individual level, though some ECB activities mentioned by Schaumberg-Muller

(1996), such as training, are typically applied at the individual level.


7/68


Key Concepts Relevant to Evaluation Capacity Building

Evaluative inquiry and organizational learning.

The concept of evaluative inquiry deliberately ties the work of ECB to the idea of

organizational learning. Preskill and Torres (1999) define evaluative inquiry as an

ongoing process for investigating and understanding critical organizational issues that

is fully integrated with an organizations work practices, and as such, it engenders (a)

organization members interest and ability in exploring critical issues using evaluation

logic, (b) organization members involvement in evaluative processes, and (c) the

personal and professional growth of individuals within the organization. Torres and

Preskill (2001) suggest that this integration of evaluation logic and processes into the

regular work of the organization increases the utility of evaluation by explicitly moving

beyond evaluating the effects of specific programs or initiatives and instead focusing on

changing the functioning of the organization itself (which affects all activities of the

organization). They distinguish evaluative inquiry from similar fields of evaluation (like

empowerment evaluation and participatory evaluation) based on the focus on integrating

evaluative activities into the existing work process, ongoing learning at multiple levels,

cultural shift, and valuing of different perspectives.

Evaluative inquiry might then be conceptualized as a specific form of ECB at the

individual, organizational, and team levels. Rather than carrying out an evaluation, the

evaluator functions as a change agent or coach, facilitating the process of inquiry and

setting up processes to support this work. The social constructivist theory of learning

guides the specific mechanisms identified for promoting individual, team, and

organizational learning.


8/68


Preskill and Torres (1999) emphasize that organizational infrastructure that

supports learning is necessary for evaluative inquiry to be incorporated into the work of

the organization. In particular, they identify having an organizational culture that is

supportive of asking questions, taking risks, and learning from mistakes as a key part of

this infrastructure.

Process use of evaluation.

Process use of evaluation is one concept closely related to ECB. Process use

refers to the idea that in addition to the use of evaluation findings (e.g. using information

about outcomes to make decisions about program or policy effectiveness), the process of

participating in an evaluation may have additional effects for individuals and

organizations (Patton, 1997). At the individual level, process use can result in changes in

thinking and behavior (e.g. a shift to thinking in a more evaluative way about program

activities). Organizational level changes in procedure or culture may also result from

process use (e.g. institutionalizing the regular collection of program data and using it for

program improvement). Patton identifies four main ways evaluation logic and processes

can be used: enhancing stakeholders shared understanding of the activity being

evaluated, supporting the program, increasing participants engagement and ownership,

and program or organizational development. Building upon the definition of process use

outlined by Patton and their work assessing the impact participating in evaluation

activities had on staff members of an organization they worked with, Forss, Rebien, &

Carlsson (2002) identified five types of process use: learning to learn, developing

networks, extending communication, strengthening the project, and boosting morale.


9/68


They also point out that these different types of process use are likely to affect different

stakeholders in the evaluation in different ways.

Patton (1997) emphasizes that changes due to process use may be an unintended

ripple effect that is incidental to the evaluation process, or they may be intentionally

incorporated into the evaluation process. A review of empirical examples of process use

suggested that this type of use often occurs due to luck (Forss et al., 2002). However,

process use can be planned for and it has been theorized that such preparation is likely to

increase the amount of process use that takes place (Forss et al. 2002; Patton, 1997;

Preskill & Torres, 2000). It has been noted that the degree of involvement in evaluation

by organization members likely affects the amount of process use that takes place, since

those who have a high degree of involvement and ownership have the most exposure to

the evaluation process and thus are more likely to be affected (Patton, 1997). Preskill and

Torres suggest that in order to promote greater learning from participation in the

evaluation process, evaluators should take a clinical approach to evaluation, work to

span the evaluation-program boundary to become more involved in the development of

program activities, and assess the readiness for organizational learning from evaluation of

the groups with whom they work.

Milstein and Cotton (2000) explicitly include process use/learning from

experience participating in evaluation as an element of their evaluation capacity

framework. Stockdill et al. (2002) suggest that in order for an activity to be classified as

ECB, it must be intentional work aimed at building capacity (not accidental or incidental

learning). This does not conflict with the idea of process use, since that can be


10/68


intentional or unintended, and Patton recommends more explicitly building opportunities

for process use into evaluation activities.

Cousins, Goh, Clark, and Lee (2004) present a theoretical framework that

integrates evaluative inquiry within an organizational learning system. They suggest that

both evaluative inquiry and use of evaluation (particularly process use) contribute to the

development of evaluation capacity within the organization, and that over time, through

sustained evaluative inquiry and continued and routine use of evaluation finding and

processes, evaluation capacity becomes integrated within organizational culture (p. 107).

ECB thus is seen as a pathway to a broader organizational change or shift in

organizational culture. This model suggests that evaluation activities precede the

development of evaluation capacity, which precedes a broader shift in organizational

culture (or the development of an evaluation culture). This contrasts with theorists who

suggest that evaluation culture is a necessary precursor of the development of evaluation

capacity.

Mainstreaming evaluation.

Another concept closely tied to ECB is mainstreaming evaluation, or making

evaluation a part of organizations everyday activities, including organizational culture,

work ethic, and job responsibilities (Sanders, 2003). Duignan (2003) suggests that in

order for evaluation to be mainstreamed, people at all levels throughout our

organizations and policymaking processes are being more evaluative about what they

do they must have appropriate evaluation skills, systems, structures, and resources to

support them in taking a more evaluative approach to their work (p. 12). Duignan

emphasizes that it is not necessary for people to call what they are doing evaluation in


11/68


order for evaluation to be mainstreamed. This approach requires evaluators to give

evaluation away (i.e. to share the skills and tools of evaluation with people who are not

professional evaluators). This necessitates building the evaluation capacity of the

individuals and organizations that are expected to mainstream evaluation into their work.

Grudens-Schuck (2003) suggests that the mainstreaming of evaluation amounts to

the incorporation of evaluation into the organizational culture. She points out that there

are benefits and problems associated with evaluation becoming a part of organizational

culture. Once something has become a part of organizational culture, it is difficult to

change, and it may take considerable effort to get people to unlearn what they already

know so they are able to take on and use new knowledge (e.g. to begin using a new

approach to evaluation or to think about evaluation in a new way).

Key Theoretical Issues Relevant to Evaluation Capacity Building

Relationship between evaluation practice and evaluation capacity building.

Stockdill and colleagues (2002) in particular emphasize the importance of making

a distinction between the practice of evaluation itself (whether or not it involves the

participation of other stakeholders) and activities where the intent is to build evaluation

capacity. While it has been suggested that the act of participating in an evaluation can

lead to important changes not directly related to the evaluation findings (see e.g. process

use of evaluation, discussed above), Stockdill et al. make the case that ECB goes beyond

simple evaluation practice. They emphasize that the practice of ECB is not (or should not

be) a matter of luck or incidental learning that takes place within the context of an

evaluation. Simply carrying out an evaluation (or a series of evaluations) is not enough to


12/68


build the capacity of an organization to incorporate and sustain evaluation as a regular

part of daily activities.

Demand for evaluation and the creation of an evaluation culture.

A culture supportive of evaluation activities is frequently identified as a necessary

element (or important precursor) of ECB (Boyle, 1999; Boyle et al., 1999; Preskill &

Torres, 1999; Owen, 2003). Boyle and colleagues have proposed a demand and supply

framework for evaluation capacity development. This framework suggests that in order

for evaluation to be institutionalized, both ability to carry out evaluation (or supply of

evaluation) and interest in utilizing evaluation (or demand for evaluation) are necessary.

Where the demand for evaluation is weak, evaluation is unlikely to be used, even when

the financial and human resources necessary to carry out evaluation are in place. Boyle

(1999) suggests that developing educated consumers of evaluation will increase demand

for evaluation services by professional evaluators and can also lead to an evaluation

ethos where evaluation is valued as an integral part of the governmental decision-

making process (p. 145). He recommends training and other support be provided to

potential evaluation users to increase their ability to be involved in evaluation, including

determining how evaluation can be useful and how to use the evaluation results. The

framework outlined by Boyle et al. suggests that the demand for and supply of evaluation

should be developed more or less simultaneously, because when either one outpaces the

other, the evaluation results yielded are undesirable (because either high quality

evaluation is not available or the evaluation resources are unlikely to be used.)

Owen (2003) proposes a tentative theoretical model of the creation and causation

of evaluation culture, based upon his experience working with organizations which have


13/68


developed such a culture and research on dissemination of implementations. He suggests

that introducing evaluation to an organization is similar to the process of introducing any

other major innovation (e.g. by ensuring that the benefits of evaluation are clear to staff

members). According to this model, action by the organizational leadership and specific

aspects of the way evaluation is implemented internally contribute to the development of

an evaluation culture in which there is a role for evaluation in decision-making within an

organization (p. 43). Similar to the model proposed by Cousins et al. (2004), Owens

model suggests that implementation of evaluation activities within an organization (at

least in a limited way) precedes the development of a culture which supports evaluation.

Owen also notes that resistance to efforts to change an organizations culture is one

possible barrier to the development of an evaluation culture.

According to Toulemonde (1999), an evaluation culture provides the collective

pressure that makes decision-makers overcome their reluctance, even when evaluation

deeply contradicts their self-interest. He suggests that evaluation culture can be built

through intense, sustained communication of the importance of evaluation by well-

respected messengers, or what he calls preaching or a sermon in favor of evaluation.

The message is that evaluation is an important component of sound public

management. Suggested methods of communication include workshops, newsletters,

conferences and journals.

Trochim (2005) does not offer specific suggestions for how to develop an

evaluation culture, but he identifies a number of elements of what an evaluation culture

might look like (from the perspective of evaluators). In particular, he suggests that and

evaluation culture should be teaching-oriented; emphasizing the connections between


14/68


evaluation and everyday thinking, and encouraging all staff members to become involved

in evaluating what they and their organizations do. He notes that this requires the

development of evaluations that are relatively simple, practical, and easily understood

and carried out by non-professional evaluators. It also requires resources to educate staff

members about evaluation.

Stages of Evaluation Capacity.

Several authors have suggested that it is useful to identify levels or developmental

stages of evaluation capacity. Love (1991) identifies six developmental stages of internal

(organizational) evaluation capability: ad hoc evaluation, systematic internal evaluation,

goal evaluation, effectiveness evaluation, efficiency evaluation, and strategic benefit.

Loves stages refer not to an organizations ability to carry out evaluations, but instead to

its ability to plan evaluations and use information collected from them in a way that

maximizes their benefit to the organization. At the first stage of evaluation capability (ad

hoc evaluation), the value of evaluation is recognized by managers, but is used in an

isolated way to evaluate specific projects, and there is typically little systematic

collection of data that could be useful for evaluation. In subsequent stages, evaluation

data is collected in increasingly more systematic ways and with greater attention to its

utility, evaluations seeks to answer more challenging questions, and evaluation results are

tied more closely to organizational planning. At the sixth level (strategic benefit, the

highest identified by Love) evaluation is used by senior managers to examine the

strategic value of activities to the company and to the broader society.

A different model of stages of organization-level evaluation capacity was

developed by Gibbs et al. (2002). They developed this model based on information


15/68


gathered through qualitative interviews about evaluation with community-based

organizations and health department staff and technical assistance providers who work

with them. Three stages of evaluation capacity were identified: compliance, investment,

and advancement. At the compliance stage, funding agency requirements are the primary

motivator of evaluation activities. Community based organizations may lack the

resources necessary for more than basic compliance with requirements, or they may see

little value in evaluation activities (or even view them as a burden that takes away from

program activities). At the investment stage, the primary motivation for evaluation

activities is program improvement. Organizational leadership supports the use of

evaluation, and organizational resources are devoted to making sure that staff members

have the time and skills necessary to carry out evaluation activities. At the advancement

stage, there is broad organizational support for evaluation, and more sophisticated

evaluations focused on demonstrating the outcomes of their programs are carried out.

Significant resources are devoted to evaluation, and these organizations may partner with

an external evaluator or have staff members who have specialized evaluation skills.

Professionalization of Evaluation

There are multiple viewpoints regarding whether evaluation should be primarily

in the hands of professional evaluators, or whether evaluation should be presented as an

activity that can and should be carried out as part of daily organizational practice. In

general, authors who approach ECB from the international development perspective

suggest that the development of a class of professional evaluators (including formal

education for evaluators) is a necessary component of evaluation capacity (Boyle, 1999;

Toulemonde, 1995). Toulemonde even offers a system for classifying evaluators from


16/68


amateurs (who perform some evaluation, but are limited by a lack of knowledge/ability to

apply different evaluation techniques) to professional evaluators (who have specialized

training in multiple methods and frameworks for evaluation and spend the majority of

their work time on evaluation). In contrast, Duignan (2003) cautions that the

professionalization of evaluation can reify evaluation and make it a separate activity that

may or not be done to programs rather than a making evaluation a part of everyday

organization activities (p. 12) and thus counter to the goals of ECB as defined by

Stockdill et al. (2002) and mainstreaming of evaluation as outlined by Sanders (2003).

These differences regarding how professionalization of evaluation relates to ECB

make sense when considering the different perspectives from which ECB has been

approached. From an international development perspective, a major concern is the

predominance of evaluators from outside the host country (primarily meeting the

evaluation needs of donor agencies, not the country where the programs being evaluated

are taking place). Given that context, the focus on developing professional evaluators

who can conduct evaluation of aid programs funded by donors and help generate demand

for evaluation within their own country makes a great deal of sense. In settings where

there is already a supply of in-county evaluators, there is a shift in focus to giving

evaluation away by building the capacity needed for organizations to make evaluation a

part of their ongoing daily work.

Situations where evaluation capacity building may be contraindicated.

Some authors have questioned whether ECB is an appropriate strategy when

working with organizations with limited resources. Miller, Kobayashi, and Noble (2006)

suggest that particularly for small community-based organizations, building the capacity


17/68


to carry out their own evaluation may be unsustainable. Particularly for smaller

organizations with few resources, learning about and carrying out evaluation may be

viewed as a burden on staff. They suggest instead a model they call insourcing

whereby organization staff members are trained in a few basic evaluation-related

activities (e.g. logic model development, data collection and entry) to minimize the

burden placed on staff members. Evaluation activities which require more specialized

knowledge (like the development of the evaluation design and data analysis) would then

be carried out by professional researchers or evaluators. Miller et al. suggest that in

insourcing, the cost of hiring external evaluators to do this work can be decreased by

creating networks of community based organizations that have similar types of programs

and need similar types of evaluation instruments. The evaluator can create design

standard evaluation tools across these organizations, spreading the cost across the groups

participating. Insourcing is in a sense a hybrid of more traditional evaluation by external

evaluators and models which require more intensive capacity building from within an

organization.

Miller et al. (2006) suggest that minimizing the burden of evaluation on

organization staff members and focusing their participation around the activities most

likely to be useful and interesting to them (creating the program logic model and

interpreting the evaluation results) is more likely to develop support for evaluation among

staff members than is more intense involvement that is viewed as a burden that takes

away from other duties (like service provision).

Methodological Challenges


18/68


ECB has been characterized as a field in its infancy(Stockdill et al., 2002) and

as such the field is struggling with issues of definition and measurement of constructs like

evaluation capacity, as well as defining what activities are considered ECB. This is

further complicated by the extraordinarily multi-national and multidisciplinary nature of

the field. Studies reviewed here span ECB efforts on 4 different continents and in fields

as diverse as education, international development, HIV prevention, and substance abuse

prevention.

A related challenge is that the development of measures of evaluation capacity (at

the individual and organizational levels) is in early stages. A review of the existing

measures of evaluation capacity is beyond the scope of this paper, but it is worth noting

that a number of potential measure of evaluation capacity at the individual and

organizational levels have been put forward in recent years. A particular challenge to

developing measures of evaluation capacity is the degree to which ECB work is context

specific (Stockdill et al., 2002). Measures developed for ECB using a particular

framework or for use in a particular setting may need to be adapted in order to be used by

other researchers.

Wing (2004) identifies a number of challenges inherent in assessing the

effectiveness of capacity building initiatives, including the abstract nature of measureing

capacity, the dynamic and constantly changing nature of capacity building, lack of clarity

about whose goals should be measured and whether the outcome of interest is knowledge

or behavior change, and the tension between the need to document results and the

gradual, long term nature of building capacity.

Research on Evaluation Capacity Building


19/68


As was noted in the introduction, ECB has relatively recently been identified as a

separate field of inquiry and research. A number of authors have identified a need for

further empirical research on ECB (Baizerman et al., 2002b; Cousins et al., 2004). Much

of the existing research to date consists of reflective case studies which are based

primarily on the reflections of the authors on the experience of ECB (Cousins et al.).

These accounts are most often from the evaluators perspective, and provide varying

levels of detail about what the capacity building provided entailed, whose capacity was

build, and what evidence there is to support their assertions. Some provide considerable

explanation of the context where capacity building took place, the approach taken by the

evaluator, and how results of ECB are being evaluated (e.g. Compton et al., 2002). Other

case study authors mention that ECB took place in the context of a collaborative or

participatory evaluation, but provide little detail about what was done to help

participants build capacity for carrying out evaluation (e.g. Biott & Cook, 2000).

In addition to the growing number of reflective case studies on ECB, some

researchers have begun to use quantitative and qualitative methods to further

understanding of the effects of ECB efforts. As noted in the previous section, there are a

number of methodological challenges to examining capacity building efforts, particularly

related to assessment of evaluation capacity. The studies reported here are largely

preliminary efforts. Relatively few of the studies reviewed here look at the effects of

capacity building across multiple levels (e.g. at the individual and organizational levels),

though several do (Campbell et al., 2004; Chinman et al., 2006; Stevenson et al., 2002).

Most have small sample sizes, and several authors do not clearly report on the number of

individuals included in analyses (Brown & Reed, 2002; Nagao et al., 2005). Only one of


20/68


the studies examined here included a comparison group (Chinman et al., 1996); the others

relied primarily on single group pre-post comparisons or qualitative methods. An

additional concern is that while definitions of ECB focus largely on the integration of

evaluation into the broader activities and culture of organizations, relatively few studies

examine the effect of ECB efforts on organizations (MacDonald et al., 2003; Robinson &

Cousins, 2004). Despite concerns about the sustainability of evaluation capacity over

time, very few studies have examined whether or how capacity has been sustained

beyond the end of ECB initiatives, and those who have followed up have not done so

more than a year after the fact.

While it is important to be aware of the limitations described here, the studies

reviewed here represent the state of the art of what is known about evaluation capacity

building. They are valuable in helping to clarify what work people in the field are

classifying as ECB, what approaches seem to be successful or acceptable to the

organizations and individuals receiving capacity building interventions, and what

challenges and difficulties have been identified in ECB. Each of these themes is

discussed in more detail below. Table 1 provides further information on each of the

studies examined.

Strategies used for ECB.

A wide variety of different strategies have been identified in the available

literature on ECB. These strategies can be conceptualized in a rough continuum from less

intense evaluator involvement (e.g. group interventions, evaluator is less involved in

carrying out evaluation) to more intense involvement (more one-on-one involvement,

evaluator has greater responsibility for carrying out evaluation activities).


21/68


A commonly identified capacity building strategy was to provide participants with

written materials about evaluation. Manuals were the most frequently mentioned type of

material (Brandon & Higa, 2004; Campbell et al., 2004; Chinman et al., 1996; Keener &

Snell-Johns, 2006; Kirsh et al., 2005; & World Bank, 2004). Other materials provided

included evaluation resource lists or resource libraries, evaluation websites, and

evaluation newsletters (Kirsh, 2005; Schnoes et al., 2000). Stevenson et al. (2002) also

identified examples of high quality evaluation plans and shared those exemplary reports

with the organizations they were training. These materials might be a part of training, but

most commonly would be used by individuals working independently.

Many authors reported using workshops or other types of group trainings to

increase participants knowledge and skills about evaluation (Arnold, 2006; Brandon &

Higa, 2003; Brown et al., 2005; Campbell et al., 2004; Chinman et al., 1996; Compton et

al., 2002; Harper et al., 2003; Keener et al., 2005; Kirsh et al., 2005; Lennie et al., 2004;

MacDonald et al., 2003; Nagao et al., 2005; OSullivan & DAgostino, 2002; Stevenson

et al., 2002; Valery & Shakir, 2005; Woodland & Hind, 2002; World Bank, 2004).

Topics of training that were identified included information on the basics of doing

program evaluation and creating program logic models, workshops focused on planning

evaluations, and workshops on analyzing evaluation results. Perhaps the most ambitious

training was a four day training program designed by Nagao et al. to provide educators

(with little to no prior experience conducting evaluation) with the all skills necessary to

plan and conduct self-evaluation of their schools.

It is noteworthy that several researchers report that planned trainings on

evaluation topics were not carried out. King (2002) reported that although she had


22/68


initially envisioned holding workshops for staff of the school district where she was

working to build evaluation capacity, this plan was aborted because of the difficulty of

securing the necessary time, and the fact that such workshops were seen as a lower

priority for the district. Schnoes et al. (2000) also report that while they offered to

provide training on evaluation methods to three comprehensive community-initiatives

with whom they were conducting an empowerment evaluation, trainings did not take

place due to lack of time and the relatively low priority placed on formal evaluation by

participants.

Arnold (2006) raised concerns about the effectiveness of group trainings on

evaluation concepts for increasing evaluation capacity. Based upon her experiences with

providing training on logic models to 4-H educators (with little previous experience in

evaluation), she notes that the information provided in group trainings was rarely applied

when the educators returned to their regular work settings. She suggests that more

intensive one-on-one training or coaching may be necessary to facilitate mastery of

evaluation skills. This fits with findings of a review of the literature on implementation of

innovations in the field, which suggests that training alone is often not sufficient to

promote transfer of learning into practice (Fixsen et al., 2005).

The provision of technical assistance or consultation on evaluation issues is

another strategy identified in many of the studies of ECB reviewed here (Arnold, 2006;

Brandon & Higa, 2004; Campbell et al., 2004; Chinman et al., 1996; Keener & Snell-

Johns, 2006; MacDonald 2003; OSullivan & DAgostino, 2002; Schnoes et al., 2000;

Stevenson et al., 2002; Valery & Shakir, 2005; Woodland & Hind, 2002). Technical

assistance provides an opportunity to get individualized assistance with questions or


23/68


concerns related to the use of evaluation in practice. Brandon & Higa tracked the topics

for which technical assistance was requested, and report that they had many requests for

help identifying data collection methods (72% of the projects they worked with),

identifying the focus of their evaluation (56%), and recommendations related to data

analyses (56%). Authors of one study which did not include technical assistance

following training report that many respondents requested assistance when a six-month

follow-up was conducted (Nagao et al., 2003).

Many authors reported that they worked collaboratively with organizations to

carry out evaluation activities as a means of building capacity (Arnold, 2006; Biott &

Cook, 2000; Brandon & Higa, 2004; Compton et al, 2002; Fetterman, 2005; Keener &

Snell-Johns, 2006; King, 2002; Lentz et al., 2005; Harper et al., 2003; Robinson &

Cousins, 2004; Suarez-Balcazar et al. 2003; & Valery & Shakir, 2005). It is suggested

that the action of participating in evaluation builds evaluation capacity. Specific areas

where collaborative processes were noted include the helping organizations to take

stock of their programs and identify their expectations for the program, developing logic

models, developing evaluation plans, and developing data collection tools. Often the

collaborative process was not clearly described, and it is not always possible to tell how

deeply involved organization staff are. Several authors report this process is facilitated

through regular meetings (in person or by phone) devoted to discussing evaluation issues.

Others recommended that teachable moments be identified and used throughout the

evaluation process. Schnoes et al. (2000) note that they ended up taking a more directive

role in evaluation processes because of a lack time and motivation for involvement

among program staff. One author clearly articulated a vision of participation in


24/68


evaluation activities as the curriculum of ECB efforts, where the participants were

students of evaluation as well as participants (King, 2002). Arnold (2006) described an

intentional process of evaluation coaching by which assistance with evaluation is

gradually scaled back as participants gain confidence. However, she also notes that this is

possible of the availability of a full time staff member dedicated to ECB activities, which

is a resource that few programs have.

The strategies described above largely focus on the development of evaluation

capacity at the individual level. A number of strategies are also mentioned that might be

conceptualized more as organization-level strategies. One such strategy is the

development of an evaluation team or expertise within the organization (Compton et al.,

2002; GAO, 2003; King, 2002; Lennie et al., 2004; Robinson & Cousins, 2004; Valery &

Shakir, 2005, Woodland & Hind, 2002). These teams vary from informal groups working

on a specific evaluation activity together (e.g. carrying out focus groups) to more formal

groups which take on a greater role in conducting evaluation within the organization.

Other organizational interventions include making evaluation a part of staff job

descriptions (Brown et al., 2005), development of evaluation guidance documents

(Milstein et al., 2002), building leadership support for evaluation (GAO, 2003; World

Bank, 2004), and developing networks or other resources for identifying outside experts

to provide specialized guidance with challenging evaluation issues (Compton et al., 2002;

GAO, 2003; World Bank, 2004).

A wide variety of ECB strategies have been identified in the literature reviewed.

The majority of these are primarily focused on building individual-level evaluation

capacity. Another distinction can be made between activities like that provide less intense


25/68


evaluator involvement in the evaluation process, and those which require more intense

involvement on the part of evaluators. A full list of the strategies identified and the

authors that mentioned them is included in Table 2.

Outcomes of ECB.

As noted above, the nature ECB work and challenges with measuring evaluation

capacity make assessing the outcomes of ECB challenging. Somewhat ironically, the

methods used in most studies on ECB make attributing outcomes to ECB strategies

problematic. The results of studies of ECB which provide some empirical evidence of the

outcomes identified are described below. These results are grouped by the type of

evidence used. Outcomes which are based only on the authors report are not included in

this section. However, a list of the ECB outcomes identified in the studies reviewed, the

authors who reported each outcome, and the evidence provided for that outcome is

included in Table 3.

Several studies have examined the results of ECB using pre-post surveys.

Chinman et al. (2006) compared changes in knowledge, attitude, and skills related to

planning and evaluation of coalition members who participated in a capacity building

program with coalition members working on programs that were not a part of this effort.

They did not find a significant difference across the two groups, but secondary analyses

within the treatment group showed that individuals who had participated more in

evaluation activities showed greater increases in knowledge, attitude, and skills than

those who participated less. Stevenson et al. (2002) administered pre- and post-surveys to

participants in three different ECB workshops. They report that participants confidence

in their ability to use evaluation skills increased following their participation in the


26/68


workshops, but in some cases this confidence had dropped slightly by the time they

participated in the next workshop. Comparing surveys administered at the beginning and

end of a year long ECB project, Brandon and Higa (2004) also reported increases in

confidence in ability to do evaluation activities, but reported no significant change in

attitudes towards evaluation. However, they note that low participation rates make

interpreting the results problematic, since those who completed the post-survey tended to

be those who persisted in ECB, not those who had dropped out the efforts.

Three studies examined ECB outcomes using retrospective pre-post surveys.

These surveys asked respondents to assess what they knew about evaluation prior to the

ECB effort they participated in, and what they currently knew. In one case (Brown &

Reed, 2002) this comparison was from the beginning to end of a day long workshop. In

two other studies, the start of the ECB work was several years prior to completion of the

survey (Arnold, 2006; Keener & Snell-Johns, 2006). While this type of survey avoids the

problem of respondents overestimating their knowledge or ability before receiving

training (i.e. they dont know what they dont know), and is convenient because it does

not require baseline data collection, it may be problematic to ask people to accurately

assess their levels of knowledge or ability several years after the fact. Arnold (2006)

reports that ECB participants (most of whom had been involved in the effort for five

years) reported increased own evaluation skills and increased use of evaluation. Keener

and Snell-Johns (2006) also report increases in confidence in evaluation knowledge and

skills over the period when ECB took place (mean time of involvement at the time of the

assessment was 13.8 months). They also note that capacity specific to the program being

worked on increased more than did more general evaluation capacity. Participants in a


27/68


one-day evaluation training workshop reported increases in knowledge of key areas at the

end of the workshop (Brown & Reed, 2002).

Several authors have explored the effects of ECB using in depth qualitative

analyses of field notes, interviews, and focus groups (Lennie et al., 2004; Robinson &

Cousins, 2004). Lennie et al. examined the empowering and disempowering effects of

ECB on individuals who were highly involved in one of two community wide ECB

initiatives. It should be noted that this is the only study reviewed to explicitly mention

looking for possible harmful effects of ECB efforts. Lennie at al. report empowering

effects including: increased knowledge of planning and evaluation, increased evaluation

and communication skills, and the application of knowledge, skills and information. They

also report a number of disempowering effects for a small number of participants,

including: insufficient knowledge sometimes led to confusion (especially at first), lack of

technical resources, lack of influence/voice in the process, and frustration/lack of

confidence among some participants. Robinson and Cousins used qualitative methods to

examine the impact of long term collaborative evaluation on organizational learning.

They report that the participating organization demonstrated increased levels of

organizational learning over the course of the evaluation, including greater organizational

memory, an increase in the shared understanding of the program, and a greater amount of

double loop learning.

Much of the research on ECB outcomes is based on participants self reports.

However, some studies have examined ratings by external observers (typically the

evaluator or researcher) to provide external of assessment of ECB at the program or

organizational level (Chinman et al., 2006; Campbell et al., 2004; OSullivan &


28/68


DAgostino, 2002; & Stevenson et al., 2002). Chinman et al. rated quality of

implementation of planning and evaluation activities using a innovation-configuration

map designed for the components of the evaluation framework used. The small number

of programs involved limited their analyses, but based on a descriptive analysis, they

report that programs that received the intervention showed increases in capacity over

time, while comparison programs did not show an increase. Campbell et al. compared

evaluation reports submitted by the 10 sexual violence prevention programs they worked

with prior to the start of ECB and after ECB work was completed. Content analysis of the

10 reports showed significant increases in several areas, including the number of

programs using more sophisticated evaluation methods, collecting and analyzing their

own data, and using evaluation data for program improvement. OSullivan and

DAgostino (2002) also rated the evaluation reports or agencies they worked with to

build evaluation capacity. They reported an increase in evaluation quality, as the number

of programs reporting only service units or data-free outcomes decreased; and the

number of programs reporting outcomes supported by data increased. Stevenson et al.

rated the organizations that they worked with on their completion of 13 evaluation steps

prior to starting ECB and one year later. They report significant increases in the number

of agencies completing 9 of the 13 steps at the second measurement (including increases

in the organizational resources for evaluation, completion of preliminary evaluation steps,

conducting data collection, and planning for using evaluation results). They also noted

that organization-level improvement was positively correlated with the amount of

technical assistance received by these agencies.


29/68


Relatively few studies included information about whether evaluation capacity

was sustained following the ECB intervention, but those who did used follow-up

interviews with participants. Brown and Reed (2002) report that at a 6-month follow-up

following an evaluation training workshop, respondents were more likely to report that

they or someone else at their organization was carrying out various evaluation activities

than they had been prior to the workshop. Brown and Reed also noted that respondents

who reported a supportive organizational culture were more likely to have made progress

in implementing an evaluation system than those who reported the culture was not

supportive. Campbell et al. (2004) report that the majority of organizations that remained

open at the 1-year follow-up survey following ECB activities were continuing to conduct

evaluations (and organizations who were required to do evaluation were more likely to be

continuing to do so). Nagao et al. (2005) also report that most workshop participants were

working to implement evaluation in their schools at a 6-month follow-up interview.

Nagao et al. also noted that at this interview, many respondents identified a need for

consultation or technical assistance with some aspects of their evaluations.

To summarize across an extremely varied group of studies, the limited empirical

findings related to ECB outcomes suggest that efforts to increase evaluation capacity

have met with some success when evaluation capacity is measured at the individual and

organizational levels. Methodological limitations of most of these studies make it

difficult to attribute a causal relationship between ECB efforts and changes in evaluation

capacity; most are more appropriately considered outcome monitoring than outcome

evaluation (one exception is the work by Chinman et al. (2006), which uses a quasi-

experimental design). In addition, the nature of the ECB studies described here make


30/68


impossible to disentangle which ECB strategies had the greatest impact on the

development of evaluation capacity. Despite these limitations in the evidence base, a

number of findings have interesting implications. Findings from Chinman et al. (2006)

suggest that actual participation in planning and evaluation activities is tied to increases

in capacity in these areas. Likewise, Keener and Snell-Johns (2006) research suggests

areas of evaluation capacity directly associated with the work being done (i.e. program

specific evaluation capacity) increases more than more general evaluation capacity. Both

these findings are suggestive of support for the practice knowledge that learning by doing

evaluation is a key pathway to evaluation capacity. It is also noteworthy that very few

harmful outcomes of ECB were reported, although the only study that did was also the

only study which explicitly mentioned looking for them (Lennie et al., 2004).

Challenges to ECB.

Limited organizational resources necessary for carrying out evaluation

(particularly staff time) was the most commonly identified barrier to ECB (Brown et al.,

2005; Campbell et al., 2004; Chinman et al., 1996; King, 2002; Kirsh et al, 2005; Lennie

et al., 2004; Stevenson et al. 2002; Schnoes et al., 2000; Woodland & Hind, 2002). A

related concern was the difficulty of maintaining participation in ECB over time,

particularly in organizations and fields with very high staff turnover (Brandon & Higa,

2004; Brown et al., 2005; Chinman et al., 1996; Compton et al., 2002; Lennie et al.,

2004; Milstein et al., 2002; Stevenson et al. 2002; & Suarez-Balcazar et al., 2003).

While not a challenge per se, the need for ongoing guidance or technical

assistance in order to be able to carry out evaluation was noted by several authors

(Arnold, 2006; Campbell et al., 2004; Chinman et al., 1996, Nagao et al., 2005; &


31/68


Stevenson et al., 2002). Several expressed concern that the ready availability of technical

assistance might reduce the motivation of organization staff to develop technical

expertise of their own, and identified the slippery slope that exists between providing

technical assistance and doing it for them (Chinman et al., 1996; Stevenson et al.,

2002). Other challenges identified by multiple studies include: difficulties identifying

appropriate data collection tools and analyzing data, fear of evaluation, confusion among

participants about evaluation concepts, and lack of proven methods for ECB. Table 4

provides a listing of challenges to ECB identified in different studies.

Conclusions

Bringing Together ECB Theory and Existing Empirical Research

There are a number of areas where the existing research and theory related to

ECB are congruent, but also some areas where there are disconnects between the theory

and research. One area of disconnect is that the definitions of ECB, mainstreaming

evaluation, and evaluative inquiry all make central the idea of making evaluation a part of

the regular work of organizations. This idea of institutionalizing evaluation is central to

the theory of ECB, but to date is largely missing from the research on ECB. Relatively

few strategies were identified in the literature for bringing about a state of affairs in

which quality program evaluation and its appropriate uses are ordinary and ongoing

practices (Stockdill et al., 2002, p. 8). While several case studies identified some

indications that evaluation had been institutionalized to some degree (e.g. MacDonald et

al. (2003) described how evaluation gradually moved from a special occurrence to a

mandated part of all new programs), none of the more methodologically rigorous studies

examined this issue.


32/68


Another area of disconnect between ECB theory and research is that evaluation

theory (particularly drawing from international development) has focused on the

importance of cultivating demand for evaluation and organizational cultures supportive of

evaluation. The empirical studies reviewed here offered relatively few strategies aimed at

helping to develop such a culture. One useful recommendation that comes from

MacDonald (2003) is to incorporate evaluation gradually into the organization, so that

people have an opportunity to see how it works before they are asked to implement it.

This recommendation is congruent with Owens model of how to develop evaluation

culture theory. Several of the case studies reviewed here mention increased prominence

of or value placed on evaluation as a result of ECB activities (Arnold, 2006; Compton et

al., 2002; Lentz et al., 2005; MacDonald et al., 2003; Valery & Shakir, 2005) but only

MacDonald and colleagues present evidence of this. It is possible that work done to

increase the support (and demand) for evaluation within organizational culture might

reduce some of the commonly identified barriers to ECB work (the lack of organizational

resources for evaluation and difficulty maintaining staff participation).

A third area of potential disconnect between evaluation research and theory has to

do with the distinction (made most strongly by Stockdill et al. (2002)) between the

practice of carrying out evaluation and the practice of building evaluation capacity. In

practice, some of the case studies reviewed appeared to make little distinction between

carrying out a collaborative or participatory evaluation and building evaluation capacity.

To further complicate the picture, Patton (1997) has suggested that that participation in

evaluation can contribute to individual or organizational learning through process use,

and that the amount of participation in evaluation is likely to affect the amount of


33/68


learning that takes place. This theory is supported by findings from Chinman et al. (2006)

suggesting that those who participated in more evaluation activities built more capacity,

and from Keener & Snell-Johns (2006) that evaluation capacity directly relevant to the

activities in which individuals participated increased more than general evaluation

capacity. These findings suggest that providing opportunities to participate in evaluation

may be a powerful ECB tool. The point made by Stockdill et al. (2002) is well taken; just

carrying out an evaluation, even if it is collaborative or participatory, should not

necessarily be considered ECB. However, if evaluation participation is planned for in a

way that is mindful of the goal of evaluation capacity building and deliberately structures

participation in a way to maximize learning (e.g. King, 2002) then it makes sense to

classify it as a strategy for ECB (as well as the practice of evaluation).

Recommendations for Next Steps for the Field of ECB

It has been suggested that ECB is a field in its infancy (Stockdill et al., 2002;

Cousins et al., 2004) and this review of the research relevant to ECB does not contradict

that. Like an infant, the research literature on ECB is growing at a rapid pace (as is

suggested by the fact that the vast majority of the studies reviewed here were published

since the year 2000), but it still has a long way to go before reaching maturity. There is a

striking lack of research using strong methods (either quantitative or qualitative)

particularly given the complexity of the ECB process and the multi-level nature of the

work. While this is understandable given the newness of the field, MacKay emphasizes

the importance of ECB practitioners walking the walk by evaluating their own efforts,

and notes that this can be a strong tool for promoting the value of evaluation within an

organization.


34/68


The growing body of case studies focused on ECB experiences provides a useful

start for the development and refinement of ECB theory. However, while some provide

rich detail, these case studies are hampered in many instances by a lack of a systematic

way of compiling experiences learned, and by the limitations inherent in the fact that

most are from the perspective of the evaluator or ECB practitioner. It is crucial to learn

more about ECB from the perspective of those in the field working to build their own

evaluation capacity or not (e.g. Schnoes et al., 2000). Qualitative methods, such as

grounded theory, likely have much to offer here, particularly given the need to develop

and refine ECB theory, and the complex and multidimensional nature of the factors at

pay in ECB.


35/68


References

Arnold, M. E. (2006). Developing evaluation capacity in Extension 4-H Field Faculty: A

framework for success.American Journal of Evaluation, 27(2), 257-269.

Baizerman, M., Compton, D. W. & Stockdill, S. H. (2002a). Editors Notes.New

Directions for Evaluation, 93, 1-6.

Baizerman, M., Compton, D. W. & Stockdill, S. H. (2002b). New directions for ECB.

New Directions for Evaluation, 93, 109-119.

Baizerman, M., Compton, D. W. & Stockdill, S. H. (2002c). Summary and analysis of the

case studies: Themes across the cases.New Directions for Evaluation, 93, 101-

107.

Biott, C., & Cook, T. (2000). Local evaluation in a national early years excellence centres

pilot programme: Integrating performance management and participatory

evaluation.Evaluation, 6(4), 399-413.

Botcheva, L., White, C. R., & Huffman L. C. (2002). Learning culture and outcomes

measurement practices in community agencies.American Journal of Evaluation,

23(4), 421-434.

Boyle, R. (1999). Professionalizing the evaluation function: Human resource

development and the building of evaluation capacity. In R. Boyle & D. Lemaire

(eds.)Building Effective Evaluation Capacity, 135-151. New Brunswick, NJ:

Transaction Publishers.

Brandon, P. R., & Higa, T. A. (2004). An empirical study of building the evaluation

capacity of K-12 site managed project personnel. The Canadian Journal of

Program Evaluation, 19(1), 125-142.


36/68


Brown, N. L., Luna, V., Ramirez, M. H., Vail, K. A., & Williams, C. A. (2005).

Developing an effective intervention for IDU women: A harm reduction approach

to collaboration.AIDS Prevention and Education, 17(4), 317-333.

Brown, R. E., & Reed, C. S. (2002). An integral approach to evaluating outcome

evaluation training.American Journal of Evaluation, 23(1), 1-17.

Campbell, R., Dorey, H., Naegeli, M., Grubstein, L., Bennett, K., Bonter, F., et al.

(2004). An empowerment evaluation model for sexual assault programs:

Empirical evidence of effectiveness.American Journal of Community

Psychology, 34, 251-262.

Chinman, M., Hunter, S. B., Ebener, P., Paddock, S., Stillman, L., Imm, P., &

Wandersman, A. (2006). The Getting To Outcomes demonstration and

evaluation: An illustration of the Prevention Support System. Manuscript

submitted for publication.

Compton, D. W., Glover-Kudon, R., Smith, I. E., & Avery, M. E. (2002). Ongoing

capacity building in the American Cancer Society (ACS) 1995-2001.New


Cousins, J. B., Goh, S. C., Clark, S., & Lee, L. E. (2004). Integrating evaluative inquiry

into the organizational culture: A review and synthesis of the knowledge base.

The Canadian Journal of Program Evaluation, 19(2), 99-141.

Duignan, P. (2003). Mainstreaming evaluation or building evaluation capability? Three

key elements.New Directions for Evaluation, 99, 7-21.


37/68


Fetterman, D. M. (2005). Empowerment evaluation: From the digital divide to academic

distress. In D. M. Fetterman & A. Wandersman (Eds.),Empowerment evaluation

principles in practice (pp. 92-122). New York: Guilford.

Forss, K., Rebien, C. C., & Carlsson, J. (2002). Process use of evaluations: Types of use

that precede lessons learned and feedback.Evaluation, 8(1), 29-45.

General Accounting Office. (2003). Program Evaluation: An Evaluation Culture and

Collaborative Partnerships Help Build Agency Capacity (GAO-03-454).

Washington, DC: General Accounting Office.

Grudens-Schuck, N. (2003). The rigidity and comfort of habits: A cultural and

philosophical analysis of the ups and downs of mainstreaming evaluation.New


Harper, G. W., Contreras, R., Bangi, A., & Pedraza, A. (2003). Collaborative process

evaluation: Enhancing community relevance and cultural appropriateness in HIV

prevention.Journal of Prevention and Intervention in the Community, 26(2), 53-

69.

Keener, D. C. & Snell-Johns, J. (2006).Measuring Evaluation Capacity: Results from a

Pilot Administration of the South Carolina Evaluation Capacity Survey.

Unpublished manuscript, University of South Carolina.

Keener, D. C., Snell-Johns, J., Livet, M., & Wandersman, A. (2005). Lessons that

influenced the current conceptualization of Empowerment Evaluation: Reflections

from two evaluation projects. In D. M. Fetterman & A. Wandersman (Eds.),

Empowerment evaluation principles in practice (pp. 73-91). New York: Guilford.


38/68


King, J. A. (2002). Building the evaluation capacity of a school district.New Directions

for Evaluation, 93, 63-80.

Kirsh, B., Krupa, T., Horgan, S., Kelly, D., & Carr, S. (2005). Making it better: Building

evaluation capacity in community mental health. Psychiatric Rehabilitation

Journal, 28(3), 234-241.

Lennie, J. (2005). An evaluation capacity-building process for sustainable community IT

initiatives: Empowering and disempowering impacts.Evaluation, 11(4), 390-414.

Lennie, J., Hearn, G., Simpson, L., da Silva, E. K., Kimber, M., & Hanrahan, M. (2004).

Building community capacity in evaluating IT projects: Outcomes of the

LEARNERS project. Queensland, Australia: Creative Industries Research and

Applications Centre.

Lentz, B. E., Imm, P. S., Yost, J. B., Johnson, N. P., Barron, C., Lindberg, M. S., &

Treistman, J. (2005). Empowerment evaluation and organizational learning: A

case study of a community coalition designed to prevent child abuse and neglect.

In D. M. Fetterman & A. Wandersman (Eds.),Empowerment evaluation

principles in practice (pp. 155-182). New York: Guilford.

Mackay, K. (2002). The World Banks ECB experience.New Directions for Evaluation,

93, 81-99.

McDonald, B., Rogers, P., & Kefford, B. (2003) Teaching people to fish? Building

evaluation capability of public sector organizations.Evaluation, 9(1), 9-29.

Miller, T. I., Kobayashi, M. M., & Noble, P. M. (2006). Insourcing, not capacity

building, a better model for sustained program evaluation.American Journal of

Evaluation, 27(1), 83-94.


39/68


Milstein, B., Chapel, T. J., Wetterhall, S.F., & Cotton, D. A. (2002). Building capacity

for program evaluation at the Centers for Disease Control and Prevention.New


Milstein, B., & Cotton, D. (2000, March).Defining Concepts for the Presidential Strand

on Building Evaluation Capacity. Working paper circulated in advance of the

November, 2000, meeting of the American Evaluation Association.

Nagao, M., Kuji-Shikatani, K., & Love, A. J. (2005). Preparing school evaluators:

Hiroshima pilot test of the Japan Evaluation Societys Accreditation Project. The

Canadian Journal of Program Evaluation, 20(2), 125-155.

OSullivan, R. G., & DAgostino, A. (2002). Promoting evaluation through collaboration:

Findings from community-based programs for young children and their families.

Evaluation, 8(3), 372-387.

Owen, J. M. (2003). Evaluation culture: A definition and analysis of its development

within organizations.Evaluation Journal of Australasia, 3(1), 43-47.

Patton, M. Q. (1997). Utilization-focused Evaluation. Thousand Oaks, CA: Sage

Publications, Inc.

Preskill, H., & Torres, R. T. (1999).Evaluative Inquiry for Learning in Organizations.

Thousand Oaks, CA: Sage Publications, Inc.

Preskill, H., & Torres, R. T. (2000). The learning dimension of evaluation use.New


Preskill, H., Zuckerman, B., & Matthews, B. (2003). An exploratory study of process use:

Findings and implications for future research.American Journal of Evaluation,

24(4), 423-442.


40/68


Robinson, T. T., & Cousins, J. B. (2004). Internal participatory evaluation as

organizational learning: A longitudinal case study. Studies in Educational

Evaluation, 30, 1-22.

Sanders, J. R. (2003). Mainstreaming Evaluation.New Directions for Evaluation, 99, 3-6.

Schnoes, C. J., Murphy-Berman, V., & Chambers, J. M. (2000). Empowerment

evaluation applied: Experiences, analysis, and recommendations from a case

study.American Journal of Evaluation, 21(1), 53-64.

Stevenson, J. F., Florin, P., Mills, D. S., & Andrade, M. (2002).Building evaluation

capacity in human service organizations: A case study.Evaluation and Program

Planning, 25, 233-243.

Stockdill, S. H., Baizerman, M., & Compton, D. W. (2002). Toward a Definition of the

ECB Process: A Conversation with the ECB Literature.New Directions for

Evaluation, 93, 7-25.

Suarez-Balcazar, Y., Orellana-Damacela, L., Portillo, N., Sharma, A., & Lanum, M.

(2003). Implementing an outcomes model in the participatory evaluation of

community initiatives.Journal of Prevention and Intervention in the Community,

26(2), 5-20.

Torres, R. T., & Preskill, H. (2001). Evaluation and organizational learning: Past, present,

future.American Journal of Evaluation, 22(3), 387-395.

Trevisan, M. S. (2002). Evaluation capacity in K-12 school counseling programs.

American Journal of Evaluation, 23(3), 291-305.


41/68


Trochim, W. M. The Research Methods Knowledge Base, 2nd Edition. Internet WWW

page, at URL: (version current

as of January 16, 2005).

Toulemonde, J. (1999). Incentives, constraints, and culture-building as instruments for

the development of evaluation demand. In R. Boyle & D. Lemaire (eds.),

Building Effective Evaluation Capacity, (pp. 153-174), New Brunswick, NJ:

Transaction Publishers.

Toulemonde, J. (1995). The emergence of an evaluation profession in European

countries: the case of structural policies. Knowledge and Policy, 8(3), 43-54.

Valery, R., & Shakir, S. (2005). Evaluation capacity building and humanitarian

organization.Journal of Multidisciplinary Evaluation, 3, 78-112.

Woodland, J., & Hind, J. (2002). Capacity building evaluation of capacity building

programs. Paper presented at the 2002 meeting of the Australasian Evaluation

Society International Conference, October/November, 2002.

World Bank (2004). Evaluation Capacity Development: OED Self Evaluation.

Washington, DC: The World Bank Operations Evaluation Department.


42/68

Evaluation Capacit

Table 1

Characteristics of the Studies Reviewed

Study Sample Context Purpose/Questions Methods Used Key Arnold (2006) 4-H field

educators in thestate of Oregon(30 out of 37participated insurvey)

Evaluation capacitybuilding effort forfield coordinatorswho had little timeor training for doingplanning andevaluation

Examine changes tothe evaluationcapacity of 4-H fieldcoordinators followinga 5-year ECBprogram

Retrospectivepre-post survey,reflective casestudy

Quafromeducdevethe tincreprac

Biott & Cook(2000)

1 center for earlychildhooddevelopment in arural area ofnorth England

(center staff,parents and othercommunitymembersparticipated)

Local evaluationusing aparticipatoryevaluationframework; both a

local and nationalevaluation wererequired in order toreceive funding forthis pilot program

Describe the tensionbetween localparticipatoryevaluation focusedon local program

improvement incontext and nationalevaluation focusedon performancemanagement

Reflective casestudy

Impoindicfor eShobase

rolesstaffcapa

Brandon & Higa(2004)

Staff of 17projects across 2school districts inHawaii; 59individualsparticipated inmeetings andworkshops (41

respondentscompleted pre-survey, 37completed post-survey, only 20completed both)

The school districtrequired theprograms to beevaluated, andcontracted with theevaluators toprovide evaluationprofessional

development tothe program staffinvolved

Examine the effectsof professionaldevelopment onparticipantsevaluation capacity

Pre-post survey(limited by lowresponse rate,particularly onpost-survey)

Pre-conf(evamoreundesigndifficin ab

objeevalNo ceval


43/68

Evaluation Capacit

Study Sample Context Purpose/Questions Methods Used Key

Brown & Reed,2002

Participants inevaluationtrainingworkshops held

for local CBOsheld in Michigan(number ofparticipants notprovided)

United Waysponsoredvoluntary trainingson evaluation;

participants wereexpected totransfer/disseminate whatthey learned totheir organizations

Examine the extent towhich trainingparticipants reportedincreased knowledge,

skills, and thetransfer of knowledgeand skills to their jobs

Retrospectivepre-post surveyat training, 6-month follow up

survey

Mosundebeliefor th

perfowereThosenviless eval

Brown et al., 2005 4 HIV PreventionCBOs in southernCalifornia

Collaboration ofcounty, evaluationcontractor and 4CBOs to evaluatea new HIVprevention program

Illustrate how onegroup of agenciesovercame barriers tobuild capacity fordelivery andevaluation of theprogram


Highresotrainchalevalby thwere

Campbell et al.(2004)

10 Rapepreventionprograms; 24rape victimsservicesprograms in thestate of Michigan

State funderswanted programsto evaluateprograms, andbrought together anevaluation team todevelop localevaluations

Assess theeffectiveness of anempowermentevaluation approachintended to buildevaluation capacity inlocal programs

Participantssatisfactionratings,comparison ofquality ofevaluationreports at thestart and end ofthe program, and1-year follow-upinterviews

Baseevalprevsignagenincluevalanalfindi100%90%(opewere

(eithoutc


44/68


45/68

Evaluation Capacit


examined theirexperience anddeveloped casestudies

Harper et al.(2003)

1 HIV/AIDSservice CBO inChicago

Collaborativeempowermentevaluation;partnershipbetween CBO anduniversity faculty &students

Describe thedevelopment,implementation, anduse of processevaluation by a teamof communitymembers anduniversity faculty andstudents


Succthe dcollenumThe baseyout

Keener & SnellJohns (2006);Keener et al.(2005)

2 communityinitiatives, 15staff membersparticipated in the

survey

Initiatives hired anevaluation team toconductempowerment

evaluation andbuild self-evaluation capacity

Examine whetherthere were changesin evaluation capacityfollowing ECB

Retrospectivepre-post survey(Keener & Snell-Johns) and

reflective casestudy (Keener etal.)

Survsignevaltheir

ECBevalto thon) ievalseenimpo(regathe sthe e


46/68

Evaluation Capacit


King (2002) 1 school districtin Minnesota

Internal evaluatorhired to carry outevaluations andbuild capacity of

the staff to engagein evaluation anddata-baseddecision-making;high level ofsupport fromsuperintendent, butlimited resources

Describe the processof building evaluationcapacity, includingsuccesses and

challenges


Comthoscoupsign

ECBevalvisibapprpartiand adm

Kirsh et al. (2005) Members of afederation ofcommunitymental healthagencies in

Ontario; 168people from 101agenciesparticipated inworkshops(representing54% of federationmembership)

The federationidentified a need forevaluation capacitydevelopment andentered a

partnership with 2universities to buildevaluation capacityamong members

Describe a province-wide ECB effortincluding successes,challenges, and waysto sustain capacity

development

Reflective casestudy & survey ofparticipants

97%dointhosmodrepo

provinitiainfornot peffec


47/68

Evaluation Capacit


Lennie et al.(2004); Lennie(2005)

2 ruralcommunities inAustralia,including 5

collaboratingorganizations and15 coreparticipants whowere included inmost analyses

Initiative intendedto increasecommunityinvolvement in and

capacity forplanning andevaluating ITprojects.Participatory actionresearch involvingresearchers andcommunitymembers

Examinedempowering anddisempoweringeffects on individual

participants

Quantitative &qualitativemethodsincluding

questionnaires,focus groups,interviews, andfield notes

Lennincluplaneval

and skillsDiseinclusomtechinfluof coparti

Mackay (2002) ECB outreachactivities at theWorld Bank

The World Bankhas a long historyof involvement inevaluation, but

began ECB workmore recently

Describe ECBactivities of the WorldBank and how ECBbecame a valued

activity of the Bank


Maclessothe WHe a

is shECB

McDonald et al.(2003)

1 Division withinstate governmentof Victoria,Australia

Increasing pressureon the Division foraccountability andoutcomes, coupledwith a shift in focusfrom many smallerprojects to fewerlarger ones

Identification ofsuccessful capacitybuilding strategies

Case study of 5-year ECBinitiative(including use ofqualitative andquantitativemeasures frominternalevaluation ofECB activities)

Partevalexistfuturare sorgamainactiv


48/68


49/68

E

valuationCapacity

49

Study

Sam

ple

Context

Purpose/Questions

MethodsUsed

KeyFindings

Robinson&

Cousins,2004

Asingle

organization(a

partnershipof

gov

ernment

age

nciesand

national

organizations)in

Can

ada

Participatory

evaluationofa

singleprogram

beingimplemente

d

bytheorganizatio

n

Empiricallyexamine

thelinksbetween

participatory

evaluationand

organizational

learning

Longitudinal

single

case

study,including

qualitative

analysisof28

interviews,focus

group

,participant

obser

vationand

reflec

tion.

Qualitativeanalysis

showed

increasedlevelsoforganizational

learning,includingg

reater

organizationalmem

ory,anincrease

inthesharedunderstandingofthe

program,andagreateramountof

doublelooplearnin

g

Schnoesetal.

(2000)

3Comprehensive

Com

munity

Initiatives(CCIs)

in3

communities

acrossthestate

ofN

ebraska

Thestateof

Nebraska

contractedwiththe

evaluatorsto

evaluatethe3CC

Is

andtheevaluators

optedtousean

empowerment

evaluation

approach

Describethe

challenges

experiencedasthe

evaluatorsattempted

tobuildcapacityas

partofan

empowerment

evaluation

Reflectivecase

study,surveyof

CCIp

rogram

coord

inatorsand

board

members

abouttheir

perce

ptionsof

theevaluation

effort

InvolvementofCCI

staffmembersin

evaluationwasmuc

hlessactivethan

anticipated,leading

totheevaluators

takingamoredirectiveapproach.

CCIstaffreportedth

attheywere

busyandevaluation

wasnota

priority.Asurveysh

owedthat50%o

f

CCIstaffandboard

members

preferredtheevalua

torsactas

traditionalexternale

valuators.There

werevariableviews

ofthenature

andpurposeofevaluation.

Stevensonetal.

(2002)

13CBOscarrying

out

substance

use

prevention

(inc

luding25

participants

across3different

wor

kshops)

Evaluatorshiredb

y

thestatetobuild

capacityofCBOs

toevaluationtheir

preventionefforts.

Examinetheeffects

ofcapacitybuilding

efforts

Individuallevel

capac

ity

meas

uredwith

pre-posttraining

questionnaires

Agencylevel

capac

ity

meas

uredwith

evaluationstaff

rating

sofCBO

comp

letionof

evaluationsteps

Surveysshowedinc

reasedlevelsof

confidenceinability

tocarryout

evaluationactivities,withjumpsin

confidenceoccurringfollowingthe

trainingdeali

Documents

Duffy ECB Working Paper