Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
99LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME
Abstract:
Résumé:
Corresponding author: B. Cousins, Faculty of Education, University of Ottawa,145 Jean Jacques Lussier, Ottawa, ON Canada, K1N 6N5;
The Canadian Journal of Program Evaluation Vol. 19 No. 2 Pages 99–141ISSN 0834-1516 Copyright © 2004 Canadian Evaluation Society
INTEGRATING EVALUATIVE INQUIRY INTOTHE ORGANIZATIONAL CULTURE: A REVIEWAND SYNTHESIS OF THE KNOWLEDGE BASE
J. Bradley CousinsSwee C. GohShannon ClarkUniversity of OttawaOttawa, Ontario
Linda E. LeeProactive Information Service Inc.Winnipeg, Manitoba
The purpose of this article is to explore, through an extensivereview and integration of recent scholarly literature, the con-ceptual interconnections and linkages among developments inthe domains of evaluation utilization, evaluation capacity build-ing, and organizational learning. Our goal is to describe andcritique the current state of the knowledge base concerning thegeneral problem of integrating evaluation into the organiza-tional culture. We located and reviewed 36 recent empirical stud-ies and used them to elaborate a conceptual framework thatwas partially based on prior work. Methodologically, our resultsshow that research in this area is underdeveloped. Substantive-ly, they show that organizational readiness for evaluation maybe favourably influenced through direct evaluation capacitybuilding (ECB) initiatives and indirectly through doing and us-ing evaluation. We discuss these results in terms of an agendafor ongoing research and implications for practice.
Cet article a pour but d’explorer, grâce à un examen poussé et àl’intégration de la littérature académique récente, les intercon-nexions et liens conceptuels entre les faits nouveaux dans lesdomaines de l’utilisation de l’évaluation, du renforcement descapacités d’évaluation, et de l’apprentissage organisationnel.L’objectif est de décrire et de critiquer l’état actuel de la base deconnaissances en ce qui a trait au problème général de l’inté-gration de l’évaluation à la culture organisationnelle. Nous avonsidentifié et examiné 36 études empiriques récentes utilisées pour
THE CANADIAN JOURNAL OF PROGRAM EVALUATION100
élaborer un cadre conceptuel partiellement fondé sur des tra-vaux antérieurs. Du point de vue méthodologique, les résultatsdémontrent que la recherche dans ce domaine est encore sous-développée. Du point de vue «validation», ils démontrent quel’état de préparation de l’organisation à l’évaluation peut êtrefavorablement influencée directement par des initiatives de ren-forcement des capacités d’évaluation (RCE) et indirectement parl’exécution et l’utilisation d’évaluations. La discussion proposeun plan pour la recherche continue et identifie les implicationspour la pratique.
The concept of evaluation utilization has been of inter-est to evaluation scholars and practitioners for quite some time. Thisparticular domain of interest has undergone considerable evolutionand development due in no small measure to the pioneering effortsof such noted theorists and researchers as Weiss (1981), Alkin,Daillak, and White (1979), and Patton et al. (1977). Perhaps amongthe most dramatic early developments in the conceptualization ofthe utilization construct was the recognition that its use need notbe limited to directly observable, instrumental consequences of evalu-ation findings and recommendations; conceptual development orlearning about the object being evaluated was identified as a legiti-mate if not pervasive consequence of evaluation activity. Indeed,such learning might be conceived to be an instance of “capacity build-ing,” a topic of considerable interest within evaluation circles as webegin the new millennium.
In response to a call from the president of the American EvaluationAssociation (AEA), Milstein and Cotton (2000) opened up the poten-tial for dialogue among professional evaluators about concepts rel-evant to “evaluation capacity building” (ECB) by providing athoughtful framework for consideration. They define evaluation ca-pacity as “the ability to conduct an effective evaluation (i.e., one thatmeets accepted standards of the discipline)” (pp. 1–2). They thenproceed to differentiate ECB from other kinds of capacity buildingsuch as the ability of individuals, organizations, or communities toachieve broad social or organizational goals. Presumably the kindsof instrumental and conceptual consequences of evaluation men-tioned above would fall into this latter category. To the extent thatevaluation helps individuals, organizations, or communities under-stand phenomena of interest or take appropriate actions in order toachieve valued goals, it would be considered a capacity building ex-ercise, but one that is differentiated from ECB per se. In their words,“while undeniably important, this form of capacity is different from
101LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME
the capacity needed to conduct effective evaluation activities in thefirst place” (Milstein & Cotton, p. 2).
A recent volume of New Directions for Evaluation by Compton, Baizer-man, and Stockdill (2002) takes this perspective on ECB one step fur-ther. These authors frame ECB not only in terms of the ability to doquality evaluation but also to use it within the organizational context.Specifically, they define the term, albeit unparsimoniously, as:
A context-dependent intentional action system of guidedprocesses and practices for bringing about and sustain-ing a state of affairs in which quality program evalua-tion and its appropriate uses are ordinary and ongoingpractices within and/or between one or more organiza-tions/programs/sites. (Stockdill, Baizerman, & Compton,2002, p. 8, emphasis added)
In this conceptual definition the consequences of evaluation, in termsof its appropriate uses, are thrown into the mix. We would arguethat such an extension is laudable and that a sharp distinction be-tween ECB and capacity building associated with the consequences(i.e., uses) of evaluation is unwarranted and perhaps counterpro-ductive. Moreover, the integration of evaluation into the culture oforganizations has, in our view, as much to do with the consequencesof evaluation as it does the development of skills and knowledge ofevaluation logic and methods.
From our point of view, there is much to be learned from an integra-tion of what we know about evaluation utilization and ECB. Wechoose to frame evaluation as an organizational learning system. Inother words, evaluative inquiry, as part and parcel of organizationalculture and operations, has the potential to serve as a potent meansfor organizations to develop their organizational learning capacity(OLC). Through doing evaluation and developing the capacity to doit, organizations become more adroit in constructing shared repre-sentations of knowledge and structures, predisposed to generate newknowledge, inclined to capture and interpret external information,and apt to question basic assumptions about the organization, itsgoals, and strategies for achieving them. Such consequences, in theparlance of evaluation utilization, represent group or organizationlevel “process use.”
The purpose of this article is to explore, through an extensive re-view and integration of recent scholarly literature, the conceptual
THE CANADIAN JOURNAL OF PROGRAM EVALUATION102
interconnections and linkages among developments in the domainsof evaluation utilization, evaluation capacity building, and organi-zational learning. We argue that the capacity to do effective evalua-tion is inextricably linked to the kinds of capacity building outcomesof evaluation that we might generally associate with evaluation uti-lization. Further, we propose that ECB is tightly connected to or-ganizational development and capacity building to the extent thatit contributes to an organization’s propensity to learn. A product ofour review and integration is a conceptual framework that linksevaluation activities and consequences with organizational capac-ity building. We begin by introducing this framework and thenproceed to systematically review and integrate recent empirical lit-erature using it as an organizing mechanism. Finally, we develop aresearch agenda around the problem of integrating evaluative in-quiry into the organizational culture.
CONCEPTUAL FRAMEWORK
Using some of our prior conceptual work (Cousins, 1996, 2003; Goh,2000; Goh & Richards, 1997) as a platform, we developed the theo-retical framework that appears in Figure 1 through reading, crit-iquing, and synthesizing recent scholarly literature linkingevaluation to organizational development and learning. The frame-work is a conceptual representation of the key variables of interest(spheres) and relationships among them (arrows). In this sectionwe describe the framework, which then serves as an advance organ-izer for the review of empirical literature to follow.
The spheres of interest are roughly divided into those associatedwith the organization itself and those associated with evaluationactivities and consequences situated within the organizational con-text. Organizational consequences are shown to depend on an or-ganization’s capacity to learn, which in turn is sustained by variousorganizational support structures. One such support structurewould be evaluative inquiry that leads to various consequences in-cluding the use of evaluation findings. We considered evaluativeconsequences to be organizational consequences in this context.Through the use of evaluative inquiry and experiences in usingevaluation, the organization develops its evaluation capacity. Suchcapacity becomes integrated with OLC, manifest in the develop-ment of the cultural variable “organizational readiness for evalua-tion.” We now consider the individual components of the frameworkmore directly.
103LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME
Figure 1Conceptual Framework of Evaluative Inquiry as an Organizational Learning System
Organizational Dimensions
Organizational learning is a psychosocial construct that has gar-nered considerable attention from organization theorists, educationaladministrators, and other corporate managers over the past decade(Cousins, 1996; Fiol & Lyles, 1985; Garvin, 1993; Goh & Richards,1997; Huber, 1991; Leithwood, Aitken, & Jantzi, 2001). It is wellknown that organizations differ quite enormously in their ability tolearn and adapt to rapidly changing environments (Fiol & Lyles,1985; Senge, 1990), but what is not readily understood is how tofoster the development of significant and sustainable OLC (Garvin,1993).
Organizational Consequences
Fundamental to conceptions of organizational learning is the devel-opment among organization members of shared mental representa-tions or understandings of the organization and how it operates.
EVALUATIVEINQUIRY
ParticipationMethod choice
Internal/externalPurpose
ECB
ORGANIZATIONALPARAMETERS
EVALUATIONCAPACITY
EVALUATIONCONSEQUENCES
ORGANIZATIONALCONSEQUENCES
ORGANIZATIONALSUPPORT
STRUCTURES
Evaluation Evaluation
knowledge and skilllogic
Organizationalreadiness
for evaluation
Communication structures
Reward systems Professionaldevelopment
Knowledgeproduction
Processuse
Use of findings:•Instrumental•Conceptual•Symbolic
Shared representation
Question basic assumptions
Problem solving anddecision making
Goalattainment
Mission
ORGANIZATIONALLEARNINGCAPACITY
Leadership Teamwork
Cultureof experi-mentationKnowledge transfer
THE CANADIAN JOURNAL OF PROGRAM EVALUATION104
Most theorists agree that organizational learning cannot happen inthe absence of individual learning by organization members. Thismulti-dimensional construct ranges from low-level, first-order, orsingle-loop learning, where change is incremental, to high-level, sec-ond-order, or double-loop learning where fundamental assumptionsabout the organization and its operation are brought to the surface,questioned, and ultimately altered (Fiol & Lyles, 1985; Huber, 1991;Lant & Mezias, 1992; Lundberg, 1989).
Organizational Learning Capacity
The allure of understanding organizations as learning systems ispowerful, but as Garvin (1993) suggests, it is imperative that wemove on to clearer guidelines for practice and operational advice asto how to build a learning organization. To this end Goh (2000) syn-thesized and integrated the theoretical literature and argues thatlearning organizations have the following core strategic buildingblocks:
• Mission and vision: clarity and employee support of themission, strategy, and espoused values of the organization;
• Leadership: leadership that is perceived as empoweringemployees, encouraging an experimenting culture, andshowing strong commitment to the organization;
• Experimentation: a strong culture of experimentation thatis rewarded and supported at all levels in the organization;
• Transfer of knowledge: the ability of an organization totransfer knowledge within and from outside the organiza-tion and to learn from failures;
• Teamwork and cooperation: an emphasis on teamwork andgroup problem-solving as the mode of operation and for de-veloping innovative ideas.
Although they are presented as separate dimensions, Goh (2000)proposed that these building blocks are interdependent and mutu-ally supportive conditions in a learning organization; in essence theydefine an organization’s learning capacity. Such capacity determinesthe extent to which organizational consequences occur.
Organizational Support Structures
OLC building blocks are also said to depend on organizational struc-tures and supports such as low job formalization and the acquisi-tion of appropriate knowledge and skills by organization members
105LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME
(Goh, 2000). Reward systems, in the form of formal and informalincentive mechanisms, also represent structural support for OLC,as do various communication structures within the organization,which serve to foster the horizontal and vertical flow of knowledgeand information. Professional development activities, formal andinformal, represent yet another organizational support structure.
Evaluation Dimensions
We define evaluation as systematic inquiry leading to judgementsabout program (or organization) merit, worth, and significance, andsupport for program (or organizational) decision making. Judgementimplies making comparisons between systematically gathered data(observation) and some standard or basis for comparison (anotherprogram, performance at a previous point in time, an external bench-mark). We include in our definition such activities as “needs assess-ment,” systematic inquiry into identified needs to which programs(or organizations) are intended to respond, and “monitoring” sys-tematic observation that does not necessarily imply judgement butnevertheless provides support for decision making.
Evaluative Inquiry
Evaluation purposes range from hard-nosed judgement-orientedsummative exercises to formative, improvement-oriented ones in-tended solely to provide support for decision making. These evalua-tions can be conducted by members or groups external to theorganization, by those internal to it, or by a blend of the two. Itvaries in the extent to which it involves non-evaluator stakeholders(i.e., those not trained as evaluators), ranging from exclusive use assources of data and information to deep and full partnerships be-tween evaluators and non-evaluator stakeholders in carrying outthe evaluation (Cousins & Whitmore, 1998). Multiple methods canbe employed in evaluation, ranging from quantitative to qualitativeand often involving a mix of the two. In some cases, method choiceimplies deeper epistemological considerations, but it is importantto distinguish between method and methodology (Kushner, 2002).
We consider ECB to be a special case of evaluative inquiry. We con-cur with the definition of ECB put forward by Stockdill et al. (2002)(see above), but are of the view that actions associated with devel-oping infrastructure, processes, and procedures that work to developevaluation capacity represent interventions that have a distinct
THE CANADIAN JOURNAL OF PROGRAM EVALUATION106
evaluative component (Preskill & Torres, 1999). And so, evaluativeinquiry can lead to various forms of utilization or more directly andintentionally to building evaluation capacity.
Evaluative inquiry, within our framework, is conceived to be an or-ganizational support structure — not unlike communication struc-tures, professional development structures, and the like — with thepotential to foster organizational learning. It is enhanced and dis-tinguished in Figure 1 because it provides the central focus for ourpaper. We are interested to know how evaluative inquiry can be in-tegrated into the organizational culture. It is important to note thatevaluative inquiry is a structure that can in and of itself be sup-ported by other organizational support structures, such as trainingand professional development, resources, and infrastructure.
Evaluation Consequences
Ultimately, evaluation leads to knowledge production, the validity,credibility, sophistication, timeliness, and relevance of which dependon the evaluation processes in place. It also leads to forms of the useof the knowledge produced. Conventional forms are captured underthe heading “use of findings,” including instrumental uses (supportfor discrete decisions), conceptual uses (enlightenment, learning),and symbolic uses (persuasion, compliance) (Shulha & Cousins,1997). But “process use,” a term coined by Patton, is an additionaldiscernable consequence of stakeholder participation in, or knowl-edge of, evaluation activities (Cousins, 1996; Patton, 1997, 1998;Preskill & Caracelli, 1997; Shulha & Cousins, 1997). By virtue oftheir proximity to the evaluation, stakeholders may develop in waysthat are quite independent of the findings or substantive knowledgeemerging from the inquiry. For example, participation has the po-tential to lead to the development of research skills and the capac-ity for self-critique, self-determination, and systematic inquiry atthe level of individual stakeholders. There may also be collectiveeffects at the group, team, or organizational level. Process use ofthis sort is of central interest to us in this article. We see it as amore indirect mode of evaluative inquiry leading to the developmentof evaluation capacity.
As pointed out above, we consider evaluation consequences to be aspecial case of organizational consequences. Instrumental, concep-tual, and symbolic uses of evaluative findings can be framed as ele-ments of organizational decision making, problem solving, andlearning.
107LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME
Evaluation Capacity
Following Stockdill et al. (2002), evaluation capacity is representedby the organizational processes and practices that are in place tomake a quality evaluation and its uses routine. To the extent towhich evaluation capacity is present within the organization, mem-bers of the organization develop their knowledge of evaluation logicand methods. They also develop their skills in actually doing evalu-ation, including those associated with planning; instrument devel-opment; data collection, processing, and analysis; interpretation; andreporting and follow-up. In essence, through sustained evaluativeinquiry and, in particular, continued and routine use of evaluationfindings and processes, evaluation capacity becomes integratedwithin the organizational culture. It becomes what might be labeled“organizational readiness for evaluation,” a term originally coinedby Mayer (1975, cited by Seiden, 2000) in relation to Davis andSalasin’s (1975/1983) A-VICTORY model of organizational develop-ment and change.
Many interventions have been proposed as ways to enhance OLCwithin organizations, including strategic planning, employee train-ing initiatives, focused organizational development and school im-provement projects, and the installation of management informationsystems. Evaluative inquiry is an intervention that holds much prom-ise in this regard. The literature suggests that organizational readi-ness for evaluation is a cultural dimension that becomes part of OLC.In theory, if evaluation becomes integrated into the ongoing activi-ties within an organization, it may become a learning system thatfosters the development of shared values and understanding amongorganization members. Several evaluation theorists have recognizedthe potential for evaluation to foster such collective developmentaleffects (Cousins, 1996; Cousins & Earl, 1992; Forss, Cracknell, &Samset, 1994; Owen & Lambert, 1995; Patton, 1999; Preskill, 1994;Preskill & Torres, 1999; Torres, Preskill, & Piontek, 1996).
REVIEW AND INTEGRATION OF CURRENT EMPIRICALLITERATURE
In essence this article represents a form of meta-analysis of em-pirical research in the area. Described below are the sampling cri-teria, characteristics of the sample, and the method used to analyzefindings across studies. We then turn to an integration of the find-ings.
THE CANADIAN JOURNAL OF PROGRAM EVALUATION108
Sampling Criteria
Despite early interest in organizational readiness for evaluation,the construct appears to have moved off the radar screen of evalua-tion scholars until recently (Seiden, 2000). With burgeoning inter-est in evaluation’s relationship to organizational learning, the termhas once again come into favour among researchers. For this rea-son, we sampled available empirical research from within the pastfive years only. We implemented computer searches of standard databases — Educational Resources Information Clearinghouse (ERIC),Psychlit, Sociofile — to identify recent published work. Keywordsincluded “evaluative inquiry,” “organizational learning,” “organiza-tional readiness for evaluation,” and “evaluation utilization.” Wesampled only published research subject to peer review or disserta-tion/thesis committee review. From an initial sample, we trackeddown other works through bibliographic follow-up. We also scannedrecent issues of well-known evaluation journals (e.g., American Jour-nal of Evaluation, Canadian Journal of Program Evaluation, Evalu-ation, and Evaluation and Program Planning).
The resulting sample is therefore multidisciplinary with empiricalstudies reported in many different evaluation practice and organi-zational contexts. To be included, studies had to be empirical, thatis, based on systematic observation of evaluation or organizationalpractice. Organizational learning studies that were located had toinclude an evaluative inquiry element in order to be retained.
Sample Descriptive Characteristics
Ultimately we identified a sample of 36 studies, which are describedin Table 1. Fifteen of the studies are what we might loosely callempirical (e.g., Boaz, 2002; King, 2002; VanderPlaat, Samson, &Raven, 2001). That is to say, they are based on reflective accounts ofthe author (usually an evaluator) with regard to one or more spe-cific case examples. These narratives are based on observation andinterpretation of lived experiences with evaluation, yet authors donot specify methods for capturing their observations nor other rele-vant sources of evidence supporting the case. While rich in detailand insight, it is not possible for the analyst to judge the credibilityor trustworthiness of the account on the basis of conventional can-ons for social inquiry. These papers are narratives implicitly in ac-cordance with constructivist principles of inquiry and attendantrelativist epistemological assumptions.
109LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME
Tabl
e 1
Desc
riptiv
e Ch
arac
teris
tics
of E
mpi
rical
Stu
dies
Stud
ySa
mpl
eCo
ntex
tM
etho
dsPu
rpos
e / R
esea
rch
Mai
n Fi
ndin
gsQu
est io
n(s)
Bal th
asar
&Ri
eder
(200
0)
Biot
t & C
ook
(200
0)
Boaz
(200
2)
Bran
don
& H
iga
(200
4)
Eval
uato
rs,
prog
ram
dire
ctor
san
d fe
dera
l of f i
ceof
ene
rgy
deci
sion
mak
ers
from
4 o
utof
40
eval
uatio
ns
Not a
pplic
able
Not a
pplic
able
41 s
choo
lad
min
istra
tors
and
teac
hers
in 1
7sc
hool
s
Seco
nd p
hase
of a
larg
erth
ree-
stud
y pr
ojec
t tha
tin
volv
ed th
e ev
alua
t ion
of th
e Sw
iss
Ener
gy20
00 p
rogr
am
Eval
uatio
n of
the
natio
nal E
arly
Yea
rsEx
celle
nce
Cent
res
Pilo
tPr
ogra
mm
e (E
YECP
P) in
the
UK
The
eval
uatio
n of
the
Bette
r Gov
ernm
ent f
orOl
der P
eopl
e (B
GOP)
inth
e UK
Scho
ol-b
ased
trai
ning
initi
ativ
e to
enh
ance
eval
uatio
n kn
owle
dge
and
skill
s in
the
USA
Met
a-ev
alua
t ion:
8in
terv
iews
plu
sdo
cum
ent r
evie
w in
each
of 4
cas
e st
udie
s
Refle
ctiv
e ca
se s
tudy
:wr
itten
from
pers
pect
ive
ofev
alua
tor
Refle
ctiv
e ca
se s
tudy
:W
ritte
n fro
m p
oint
of
view
of t
he e
valu
ator
Long
itudi
nal f
ield
stud
y: q
uest
ionn
aire
san
d re
trosp
ectiv
ein
terv
iews
.
-The
form
s of
lear
ning
thro
ugh
eval
uat io
n-T
he c
ondi
t ions
of l
earn
ing
thro
ugh
eval
uat io
n
To e
xplo
re c
ross
cut
ting
eval
uatio
n is
sues
To e
xam
ine
how
evid
ence
-ba
sed
polic
y m
akin
gpr
esen
ts c
halle
nges
for
rese
arch
ers
Chan
ge in
atti
tude
s to
ward
eval
uatio
n; c
hang
e in
conf
iden
ce to
do
eval
ua-
tion;
per
cept
ions
of
eval
uatio
n ca
pabi
litie
s
Dif fe
rent
kin
ds o
f lea
rnin
g in
eac
h ca
se s
tudy
:di
rect
—wh
ere
feed
back
info
led
to th
e re
com
men
-da
t ions
of t
he e
valu
atio
ns b
eing
use
d, in
dire
ct—
new
appr
ecia
t ion
of th
e si
tuat
ion
is o
bser
ved
and
argu
men
ts c
an in
f luen
ce th
e co
mm
unic
atio
n of
polit
ical
issu
es. I
t is
nece
ssar
y to
dis
tingu
ish
the
type
s of
lear
ning
and
effe
ct o
f eva
luat
ions
at t
hele
vels
of o
pera
tion,
stru
ctur
e, a
nd s
trate
gy.
As th
e EY
ECPP
pro
ject
pro
gres
sed
ther
e we
rein
crea
sed
tens
ions
bet
ween
per
form
ance
man
age-
men
t and
par
ticip
ator
y ev
alua
tion.
Diff
icul
t to
dolo
cally
giv
en li
mite
d re
sour
ces
and
pres
sure
; key
to a
wor
thwh
ile ro
le fo
r loc
al e
valu
ator
s is
in th
eir
cont
ribut
ion
to le
arni
ng.
Rese
arch
ers
can
be m
ore
effe
ctiv
e in
max
imizi
ngut
iliza
tion
for p
olic
y an
d pr
actic
e if
they
go
beyo
nd th
e tra
ditio
nal r
oles
of d
eliv
erin
g re
ports
and
pres
entin
g fin
ding
s to
fund
ers.
The
like
lihoo
dof
use
for p
olic
y de
pend
ent o
n th
e fo
rm in
whi
chits
find
ings
are
mad
e av
aila
ble.
No im
prov
emen
t in
attit
udes
. Sel
f-con
fiden
ce a
sev
alua
tors
impr
oved
. Ass
essm
ent o
f cap
abili
ties
as e
valu
ator
s fa
vour
able
. (con
tinue
d on
nex
t pag
e)
THE CANADIAN JOURNAL OF PROGRAM EVALUATION110Ta
ble
1 (c
ontin
ued)
Stud
ySa
mpl
eCo
ntex
tM
etho
dsPu
rpos
e / R
esea
rch
Mai
n Fi
ndin
gsQu
est io
n(s)
Chac
on-
Mos
coso
et a
l .(2
002)
Com
pton
et a
l .(2
001)
Com
pton
et a
l.(2
002)
Conn
or &
Tanj
asiri
(199
9)
The
Hom
eles
sCh
i ld C
are
Serv
ices
Orga
niza
t ion
inSe
vil le
, Spa
in
Not a
ppl ic
able
Not a
pplic
able
2 pr
ojec
t dire
ctor
sin
Col
orad
oHe
alth
yCo
mm
uniti
esIn
itiat
ive
(CHC
I)
Form
ativ
e ev
alua
t ion
with
hom
eles
s ch
i ld c
are
cent
ers
in S
evi l l
e, S
pain
Impl
emen
tat io
n of
Col la
bora
t ive
Eval
uat io
nFe
llows
Pro
ject
(CEF
P),
invo
lvin
g Am
eric
anCa
ncer
Soc
iety
, gra
duat
est
uden
ts, a
nd fa
culty
inth
e US
A
ECB
effo
rts w
ithin
the
Amer
ican
Can
cer S
ocie
tysi
nce
1995
; nat
ionw
ide,
com
mun
ity-b
ased
,vo
lunt
eer o
rgan
izatio
n in
the
USA
Two
out o
f 29
com
mun
ities
that
are
impl
emen
ting
the
Colo
rado
Hea
lthy
Com
mun
ities
Initi
ativ
e(C
HCI)
in th
e US
A
Ref le
ctiv
e ca
se s
tudy
:wr
i t ten
from
poi
nt o
fvi
ew o
f eva
luat
ors
Ref le
ctiv
e ca
se s
tudy
:wr
i t ten
from
the
pers
pect
ives
of t
hose
invo
lved
in th
e de
sign
and
impl
emen
tatio
n of
the
CEFP
Refle
ctiv
e ca
se s
tudy
:wr
itten
from
the
pers
pect
ives
of 4
ECB
prac
titio
ners
invo
lved
from
the
begi
nnin
g in
1995
Case
stu
dies
:pa
rtici
pant
obse
rvat
ions
of
eval
uato
rs;
disc
ussi
ons
with
proj
ect d
irect
ors
To d
escr
ibe
exam
ple
ofm
utua
l cat
alyt
ic m
odel
of
form
ativ
e ev
alua
t ion
To d
escr
ibe
CEFP
and
iden
t i fy
less
ons
lear
ned
afte
r firs
t 3 y
ears
of
impl
emen
tatio
n
To te
ll th
e st
ory
of E
CB a
tth
e AC
S th
roug
h lo
okin
g at
cont
ext,
hist
ory
of E
CB,
prac
tice
prin
cipl
es a
ndth
eir o
pera
tiona
lizat
ion,
asse
ssm
ent o
f ECB
, and
less
ons
lear
ned
To e
xam
ine
cros
s-cu
tting
eval
uatio
n is
sues
Mod
el is
impl
emen
ted
succ
essf
ully
; eva
luat
ion
woul
d no
t hav
e be
en a
s se
nsi t i
ve w
i thou
t the
act iv
e co
oper
atio
n of
loca
l pra
cti t i
oner
s an
dre
pres
enta
t ives
at t
he c
entre
; eva
luat
or k
ey to
proc
ess.
Less
ons
set t i
ng u
p CE
FP: a
dmin
istra
t ive
and
tech
supp
ort ;
annu
al t r
aini
ng, o
ppor
tuni
t ies
for c
onsu
l-ta
tion,
out
side
fund
ing,
eva
luat
ion
expe
rtise
,cl
arity
re: e
valu
atio
n. Im
plem
enta
tion
less
ons:
com
mon
fram
ewor
k; tr
aini
ng, p
rogr
am e
valu
atio
nex
perti
se, s
tude
nt e
ntry
ski
lls, t
ensi
ons
betw
een
repo
rt an
d th
esis
requ
irem
ents
, tim
e in
tens
ive,
ince
ntiv
es.
ECB
trigg
ered
by
chan
ges
in th
e la
rger
con
text
,es
peci
ally
refo
rm in
pro
gram
dev
elop
men
t.Va
rious
stra
tegi
es to
enh
ance
ECB
, but
mos
tef
fect
ive
is e
valu
atio
n fe
llows
hip
colla
bora
tive.
Seve
ral p
rinci
ples
gui
de E
CB, a
mon
g th
em is
eval
uate
eva
luat
ion
unit;
mos
tly in
form
al to
dat
ebu
t som
e fo
rmal
asp
ects
; tra
inin
g st
aff i
n us
e of
eval
uatio
n pr
oces
ses
and
prod
ucts
.
Citiz
en in
volv
emen
t fos
tere
d ow
ners
hip
of th
epr
ojec
t and
sug
gest
s th
at a
ctua
l use
will
be
mor
elik
ely.
Impo
rtanc
e of
a b
road
-bas
ed a
dvis
ory
coun
cil t
hat k
ept t
he p
roje
ct in
touc
h wi
th it
sco
mm
unity
. New
par
tner
ship
s we
re fo
rmed
.Ev
alua
tion
capa
city
in th
e co
mm
unity
dev
elop
ed.
111LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME
Maj
or in
f luen
ce o
f fun
ding
age
ncy
expe
ctat
ions
,av
aila
bil i t
y of
eva
luat
ion
reso
urce
s, le
ader
ship
,an
d st
aff ,
and
avai
labi
l i ty
of e
valu
atio
n to
ols
and
tech
nolo
gy. S
tage
mod
el o
f EC:
com
plia
nce,
inve
stm
ent ,
adva
ncem
ent .
Mus
t tai
lor E
CB to
stag
e.
Cont
inua
l inv
olve
men
t of g
rant
ees
and
prog
ram
and
eval
uat io
n ex
perts
impo
rtant
to m
akin
ggu
idan
ce a
nd te
chni
cal a
ssis
tanc
e sy
stem
s m
ore
user
-frie
ndly
and
cos
t-ef fe
ctiv
e.
The
mod
el p
rovi
ded
a co
ncep
tual
fram
ewor
k fo
rth
e en
tire
eval
uatio
n pr
oces
s. E
nsur
ed c
lose
scru
tiny
of th
e co
ntex
tual
fact
ors;
bro
ught
the
need
s of
the
eval
uatio
n in
to fo
cus.
Bal
ance
dev
alua
tion
prod
uced
that
cou
ld s
peak
to th
e ne
eds
of d
ispa
rate
aud
ienc
es.
Eval
uatio
n re
quire
men
t was
initi
ally
per
ceiv
ed a
sbu
rden
som
e. M
any
alum
ni s
choo
ls s
usta
ined
the
prog
ram
’s ev
alua
tion
lega
cy e
vide
nced
thro
ugh
data
-driv
en g
oal-r
elat
ed in
quiry
, sys
tem
atic
refle
ctio
n an
d pl
anni
ng, a
nd th
e pr
esen
ce o
f an
eval
uatio
n-m
inde
d cu
lture
.
Prog
ress
revi
ewed
on
4 go
als
for 2
yea
rs o
f ECB
work
: (1)
dev
elop
sta
ff co
mm
itmen
t and
ski
lls in
prog
ram
eva
luat
ion
and
its u
se—
parti
al; (
2) b
uild
an in
frast
ruct
ure
for d
ata
colle
ctio
n, a
naly
sis,
and
pres
enta
tion—
parti
al; (
3) fa
cilit
ate
exis
ting
scho
olim
prov
emen
t pro
cess
—so
me;
(4) c
reat
e ne
twor
k
To d
escr
ibe
bel ie
fs a
ndat
t i tud
es to
ward
eva
lua-
t ion;
to id
ent i f
y fa
ctor
sin
f luen
cing
eva
luat
ion
capa
city
To a
sses
s th
e ef
f icac
y of
eval
uat io
n ca
paci
ty-
bui ld
ing
ini t i
ativ
e
To d
escr
ibe
the
appl
icat
ion
of th
e 10
-sta
ge m
odel
of
educ
atio
nal i
nnov
atio
nwi
thin
the
cont
ext o
ftra
nsfo
rmat
ion
To p
rovi
de e
vide
nce
ofho
w th
e pr
oces
ses
and
prac
tices
sur
roun
ding
the
use
of d
ata
for w
ise
deci
sion
-mak
ing
beco
me
intri
nsic
ally
der
ived
To d
escr
ibe
expe
rienc
e an
dex
plor
e is
sues
aris
ing
indi
stric
t’s e
ffort
to e
nhan
ceev
alua
tion
capa
city
; how
the
work
was
orig
inal
lyco
ncei
ved
and
how
and
Qual
i tat iv
e f ie
ld s
tudy
:co
nten
t ana
lysi
s of
audi
otap
ed in
terv
iew
trans
crip
ts
Case
stu
dy: c
onte
ntan
alys
is o
f tec
hnic
alas
sist
ance
requ
ests
that
cam
e in
dur
ing
one-
year
per
iod
Refle
ctiv
e ca
se s
tudy
:wr
itten
from
pers
pect
ive
ofev
alua
tor
Expl
orat
ory
case
stud
y: s
emi-s
truct
ured
inte
rvie
ws
Refle
ctiv
e ca
se s
tudy
:re
trosp
ectiv
eob
serv
atio
ns o
f aut
hor,
a un
iver
sity
pro
fess
orhi
red
for 2
yea
rs a
s an
inte
rnal
eva
luat
or
8 si
tes
in w
hich
Cen
ters
for D
isea
se C
ontro
lfu
nded
HIV
pre
vent
ion
prog
ram
s; u
sual
ly 5
or
fewe
r ful
l -tim
eem
ploy
ees
in th
e US
A
Desc
ript io
n an
d an
alys
isof
eva
luat
ion
train
ing
and
tech
nica
l ass
ista
nce
ini t i
ativ
e in
the
cont
ext o
fHI
V pr
even
tion
prog
ram
sin
the
USA
Eval
uatio
n of
Lea
rnin
g in
Engl
ish
for A
cade
mic
Purp
oses
(LEA
P) in
Sout
h Af
rica
Mid
dle
and
seco
ndar
ysc
hool
s in
volv
ed in
MSI
P in
Can
ada
Larg
e ur
ban
scho
oldi
stric
t in
Min
neso
ta th
atis
com
mitt
ed to
enha
ncin
g ev
alua
tion
func
tion
and
capa
city
.Au
thor
hire
d as
inte
rnal
8 co
mm
uni ty
-ba
sed
orga
niza
-t io
ns, 9
hea
l thde
partm
ents
, 28
tech
nica
l ass
ist-
ance
pro
vide
rs
58 fu
nded
hea
l thde
partm
ent
pers
onne
l
Not a
pplic
able
6 pr
inci
pals
from
alum
ni M
anito
baSc
hool
Impr
ove-
men
t Pro
gram
(MSI
P) s
choo
ls
Not a
pplic
able
Gibb
s et
al .
(200
2)
Gil l i
am e
t al .
(200
3)
Jaco
bs (2
000)
Katz
et a
l.(2
002)
King
(200
2)
(con
tinue
d on
nex
t pag
e)
THE CANADIAN JOURNAL OF PROGRAM EVALUATION112Ta
ble
1 (c
ontin
ued)
Stud
ySa
mpl
eCo
ntex
tM
etho
dsPu
rpos
e / R
esea
rch
Mai
n Fi
ndin
gsQu
est io
n(s)
of p
eopl
e to
rout
inize
inqu
iry—
limi te
d. L
esso
nsle
arne
d: e
ach
proj
ect a
n in
stru
ctio
nal o
ppor
tuni
ty;
dang
er o
f cre
atin
g m
ore
dem
and
than
can
han
dle;
part i
cipa
tory
app
roac
h ef
fect
ive.
Know
ledg
e m
anag
emen
t and
org
aniza
t iona
lm
emor
y ar
e re
curs
ive
soci
al s
yste
ms.
Sof
tkn
owle
dge
alon
g wi
th h
ard
test
sco
re d
ata
allo
wed
orga
niza
tiona
l lea
rnin
g to
occ
ur. I
mpo
rtanc
e of
wisd
om in
lear
ning
.
Eval
uatio
n ev
olve
d fro
m p
artic
ipat
ory
appr
oach
with
dep
ende
ncy
on e
valu
ator
, to
empo
werm
ent
self-
dire
cted
eva
luat
ion,
to c
apac
ity-b
uild
ing
focu
s dr
iven
by
teac
her e
mpo
werm
ent.
Inte
rnal
and
exte
rnal
acc
ount
abili
ty a
re p
ossi
ble.
Dem
and
is th
e m
ain
prer
equi
site
; syn
ergi
es w
ithot
her c
apac
ity-b
uild
ing
work
is e
ssen
tial;
tailo
rEC
B ac
cord
ing
to c
ount
ry c
ircum
stan
ces;
con
duct
base
line
diag
nosi
s. S
tage
s of
dev
elop
men
tde
scrib
ed: b
ank
is in
crea
sing
ly c
omm
itted
to E
CBas
a w
orth
while
act
ivity
and
it is
bei
ng a
ccor
ded
are
ason
ably
hig
h pr
iorit
y.
why
i t ch
ange
d
The
deve
lopm
ent a
nd a
p-pl
icat
ion
of o
rgan
izat io
nal
mem
ory
as a
n an
alyt
ic c
on-
stru
ct fo
r und
erst
andi
ngca
se d
ata;
the
utili
ty o
fm
emor
y as
a c
onst
ruct
help
ful i
n un
ders
tand
ing
orga
niza
tiona
l out
com
esan
d pe
rform
ance
s
To d
escr
ibe
evol
utio
n of
the
role
of e
valu
atio
n in
capa
city
bui
ldin
g fo
r the
Man
itoba
Sch
ool I
mpr
ove-
men
t Pro
gram
To te
ll th
e st
orie
s of
ECB
less
ons
lear
ned
in th
eBb
nk’s
expe
rienc
e wi
thgo
vern
men
ts in
div
erse
and
diffi
cult
envi
ronm
ents
, and
how
ECB
is b
ecom
ing
rec-
ogni
zed
and
valu
ed w
ithin
the
bank
itse
lf
Case
stu
dy: n
arra
t ive
anal
ysis
by
auth
orus
ing
inte
rvie
ws,
obse
rvat
ions
, and
docu
men
ts
Case
stu
dy: p
artic
ipan
tob
serv
atio
n; e
xcer
pts
from
prio
r dat
aco
llect
ion/
repo
rts
Refle
ctiv
e ca
se s
tudy
:de
velo
ped
from
the
pers
pect
ive
of a
ban
kEC
B co
ordi
nato
r
eval
uato
r for
2-y
ear
perio
d in
the
USA
Unde
rsta
ndin
gre
mem
berin
g as
an
orga
niza
t iona
l lea
rnin
gco
nstru
ct in
sch
ools
inth
e US
A
Eval
uatio
n su
ppor
t for
fund
ed s
choo
lim
prov
emen
t pro
gram
inM
anito
ba, C
anad
a
Wor
ld B
ank
expe
rienc
esin
dev
elop
ing
ECB
indi
vers
e an
d di
fficu
ltgo
vern
men
tal c
onte
xts
over
a 2
0-ye
ar p
erio
d
Adm
inis
trato
rs,
teac
hers
, and
stud
ents
in th
e US
city
sch
ool d
istri
ct
Eval
uato
r ref
lec-
tions
on
eval
uatio
nro
le in
mul
ti-ye
ar,
mul
ti-si
te C
anad
ian
scho
ol im
prov
e-m
ent i
nitia
tive
Not a
pplic
able
Krus
e (2
003)
Lee
(199
9)
Mac
kay
(200
2)
113LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME
Mos
t ly in
form
al le
arni
ng ta
king
pla
ce. O
L oc
curs
mor
e wi
th o
wner
/man
ager
s. W
ork-
base
d le
arni
ngwa
s in
cide
ntal
and
any
kno
wled
ge g
aine
d wa
son
ly u
sed
in th
e sh
ort t
erm
. Firm
s th
at e
mpl
oyed
inte
ntio
nal l
earn
ing
stra
tegi
es w
ere
slow
er a
ndm
ore
del ib
erat
e, b
ut w
ere
mor
e ef
fect
ive
atin
f luen
cing
the
dire
ctio
n an
d de
l iver
y of
a f i
rm’s
prog
ram
s.
Impe
tus
cam
e fro
m s
enio
r adm
inis
trat io
n. S
ixpr
inci
ples
gui
ded
the
mul
t i -ye
ar, m
ult i -
face
ted
ECB
ini t i
ativ
e. S
ucce
ssfu
l ECB
but
not
inve
sti-
gate
d sy
stem
atic
ally
; no
effe
cts
on o
rgan
izatio
nsp
ecifi
ed. A
dmin
role
in c
reat
ing
trust
ing
envi
ron-
men
t piv
otal
; mus
t dea
l with
def
ence
s of
re-
sour
ces,
righ
teou
snes
s, a
nd re
sist
ance
.
The
work
grou
p m
et it
s ob
ject
ive
and
muc
h m
ore.
The
scho
ol is
now
in th
e pr
oces
s of
set
ting
addi
tiona
l cur
ricul
um g
oals
, and
dat
a ar
e no
wbe
ing
colle
cted
on
the
indi
cato
rs th
at w
ere
form
edin
to c
ase
conf
eren
ce te
am c
heck
lists
.
Colla
bora
tive
eval
uatio
n ca
n re
sult
in im
prov
edqu
ality
of e
valu
atio
ns. S
truct
ural
requ
irem
ents
were
impo
rtant
fact
ors.
Tool
kit a
ppea
rs to
be
a su
cces
sful
met
hod
for
teac
hing
man
ager
s to
dev
elop
logi
c m
odel
s an
dim
plem
ent e
valu
atio
n te
chni
ques
; som
e su
gges
-tio
ns fo
r im
prov
emen
t.
To f i
nd o
ut th
e ty
pe a
ndin
cide
nces
of O
L an
d th
elo
cus
of O
L in
thei
r fi rm
s;to
exa
min
e m
ore
clos
ely
the
OL in
som
e f i r
ms;
toco
mpa
re a
nd c
ontra
stco
mpa
nies
that
are
sim
i lar
in s
ize in
term
s of
OL
Why
stre
ngth
enin
g ev
alua
-t io
n ca
paci
ty b
ecam
e an
expl
ici t
prio
ri ty;
prin
cipl
esan
d pr
oced
ures
; effe
cts
ofca
paci
ty b
uild
ing;
less
ons
lear
ned
Wha
t spe
cific
eva
luat
orro
les,
phi
loso
phie
s, a
ndin
terp
erso
nal d
ynam
ics
hold
the
pote
ntia
l to
fost
erpr
oces
s in
fluen
ce?
Can
colla
bora
tive
eval
ua-
tion
and
netw
orki
ngim
prov
e th
e qu
ality
of s
elf-
eval
uatio
n?
To d
escr
ibe a
Too
lkit
initi
a-tiv
e des
igne
d to
hel
p pu
blic
heal
th m
anag
ers i
n On
tario
impr
ove t
heir
know
ledg
ean
d sk
ills i
n lo
gica
l mod
elde
velo
pmen
t and
pro
gram
eval
uatio
n
Fiel
d st
udy:
mul
t i -m
etho
d st
aged
stu
dyin
volv
ing
tele
phon
esu
rvey
, int
ervi
ews,
case
stu
dies
Ref le
ctiv
e ca
se s
tudy
:re
port
is in
tegr
atio
n of
inte
rnal
and
ext
erna
lvi
ewpo
ints
of
resp
ectiv
e au
thor
s/pa
rtici
pant
s
Refle
ctiv
e ca
se s
tudy
:wr
itten
from
pers
pect
ive
ofev
alua
tor
Long
itudi
nal c
ase
stud
y: re
ports
from
eval
uatio
n fa
irpr
esen
tatio
ns g
iven
by
eval
uatio
n co
ntra
ctor
sin
2 c
onse
cutiv
e ye
ars
Refle
ctiv
e ca
se s
tudy
:wr
itten
from
the
poin
tof
vie
w of
eva
luat
ors
and
ECB
prov
ider
s
A re
port
of a
stu
dy o
for
gani
zat io
nal l
earn
ing
in th
e sm
all b
usin
ess
sect
ion
of th
e UK
econ
omy
Mam
mot
h EC
B ef
fort
inla
rge
nat io
nal U
SApu
blic
hea
l th p
reve
ntio
nan
d co
ntro
l org
aniza
tion
CDPC
ove
r mul
ti-ye
arpe
riod.
Man
y pa
rtner
orga
niza
tions
A US
A sc
hool
that
prov
ides
inst
ruct
ion
toph
ysic
ally
and
med
ical
lydi
sabl
ed c
hild
ren
from
Kto
12
Eval
uatio
n Vo
ices
(EV)
appr
oach
was
impl
e-m
ente
d in
the
cont
ext o
fea
rly c
hild
hood
edu
ca-
tion
prog
ram
eva
luat
ion
in th
e US
A
An e
valu
atio
n ca
paci
ty-
build
ing
initi
ativ
e in
the
Onta
rio, C
anad
a pu
blic
heal
th s
yste
m
6000
org
aniza
-t io
ns fr
om y
ello
wpa
ges;
600
sm
all
busi
ness
man
ag-
ers;
60
mat
ched
case
stu
dies
Not a
ppl ic
able
Not a
pplic
able
Prog
ram
pra
cti-
tione
rs p
artic
ipat
-in
g in
eva
luat
ion
netw
orki
ng a
ctiv
i-tie
s
Not a
pplic
able
Mat
lay
(200
0)
Mils
tein
et a
l .(2
002)
Mor
abito
(200
2)
O’Su
lliva
n &
D’Ag
ostin
o(2
002)
Porte
ous
et a
l.(1
999)
(con
tinue
d on
nex
t pag
e)
THE CANADIAN JOURNAL OF PROGRAM EVALUATION114Ta
ble
1 (c
ontin
ued)
Stud
ySa
mpl
eCo
ntex
tM
etho
dsPu
rpos
e / R
esea
rch
Mai
n Fi
ndin
gsQu
est io
n(s)
Five
mai
n ca
tego
ries
of in
f luen
ce o
n pr
oces
s us
e:fa
cil i t
atio
n of
eva
luat
ion
proc
esse
s; m
anag
emen
tsu
ppor
t ; ad
viso
ry g
roup
mem
ber c
hara
cter
ist ic
s;co
mm
unic
atio
n at
tribu
tes;
org
aniza
t ion
char
acte
r-is
t ics.
Exa
mpl
es o
f pro
cess
use
iden
t i fie
d: le
arn-
ing
abou
t eva
luat
ion;
abo
ut th
e pr
ogra
m; o
vera
l lev
alua
t ion
expe
rienc
e.
Sign
ifica
nt o
rgan
izatio
nal c
onse
quen
ces
ob-
serv
ed, e
spec
ially
cap
acity
bui
ldin
g. C
ondi
tions
for p
artic
ipat
ory
eval
uatio
n we
re fa
vour
able
.
Prog
ram
term
inat
ed o
n th
e ba
sis
of n
on-e
valu
a-tiv
e in
form
atio
n; s
ome
deve
lopm
ent o
f sha
red
unde
rsta
ndin
g, e
ngag
emen
t, se
lf-de
term
inat
ion.
Envi
ronm
enta
l fac
tors
affe
ctin
g us
e: ru
mou
rs o
f am
erge
r with
ano
ther
com
pany
had
pro
foun
d ef
fect
.
Wid
er s
choo
l com
mun
ities
mus
t be
invo
lved
indi
scus
sion
s of
SIP
from
com
mitm
ent t
hrou
ghim
plem
enta
tion.
Fac
tors
that
influ
ence
the
deve
l-op
men
t of p
rofe
ssio
nal c
omm
uniti
es: l
eade
rshi
pst
yle
of p
rinci
pal,
past
eve
nts,
pol
itics
of a
lloca
t-in
g re
sour
ces,
org
aniza
tion
of s
choo
ls, a
nd
Wha
t did
adv
isor
y gr
oup
mem
bers
lear
n fro
min
volv
emen
t in
eval
ua-
t ion?
How
did
they
lear
n?W
hat f
acto
rs s
uppo
rted
orhi
nder
ed le
arni
ng?
To in
vest
igat
e th
e or
gani
-za
tiona
l con
sequ
ence
s of
inte
rnal
par
ticip
ator
y ev
al-
uatio
n in
a C
anad
ian
na-
tiona
l tra
inin
gor
gani
zatio
n
Use
and
non-
use
of th
eev
alua
tion
resu
lts, w
ithpa
rticu
lar e
mph
asis
on
proc
ess
use;
fact
ors
con-
tribu
ting
to u
se a
nd n
on-
use
of th
is p
rivat
e se
ctor
eval
uatio
n
How
do s
choo
l im
prov
e-m
ent p
rogr
ams
fost
er th
ede
velo
pmen
t of p
rofe
s-si
onal
com
mun
ities
? W
hat
orga
niza
tiona
l fac
tors
supp
ort a
nd/o
r im
pede
the
Case
stu
dy o
f two
eval
uat io
ns; i
nter
view
s
Long
itudi
nal c
ase
stud
y; p
artic
ipan
tob
serv
atio
n;in
terv
iews
, foc
usgr
oup
Desc
riptiv
e ca
se s
tudy
:sa
tisfa
ctio
n in
terv
iews
,su
rvey
s
Colle
ctiv
e ca
se s
tudy
of th
ree
scho
ols:
cons
truct
ivis
tor
ient
atio
n, s
choo
l-ba
sed
obse
rvat
ion,
conf
eren
ces
and
Year
ly e
valu
atio
ns o
f the
Tel l
a Fr
iend
pro
gram
of
the
ACS
in th
e US
A
Cana
dian
nat
iona
ltra
inin
g pr
ogra
m re
form
initi
ativ
e
Eval
uatio
n of
a s
ales
prog
ram
with
in a
busi
ness
ser
vice
sor
gani
zatio
n (X
YZCo
rpor
atio
n) in
the
USA
USA
dist
rict-w
ide
scho
olim
prov
emen
t ini
tiativ
e
Pres
kil l
et a
l .(2
003)
Robi
nson
&Co
usin
s (i2
004)
Russ
-Eft
et a
l.(2
002)
Scrib
ner e
t al.
(199
9)
16 in
terv
iews
wi th
advi
sory
gro
upm
embe
rs a
nd s
en-
ior a
dmin
istra
tors
of A
mer
ican
Can
-ce
r Soc
iety
(ACS
)
9 Ca
nadi
an k
eyin
form
ant n
atio
nal
prog
ram
pra
cti-
tione
rs
23 s
ales
peo
ple,
10 im
plem
enta
tion
spec
ialis
ts, a
nd 9
cons
ulta
nts;
40
repr
esen
tativ
es o
fcl
ient
com
pani
es
3 ru
ral m
iddl
esc
hool
s: s
choo
lad
min
and
lead
er-
ship
team
mem
-be
rs a
nd o
ther
teac
hers
, N =
35
115LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME
teac
her w
ork.
The
cond
i t ion
s fo
r org
aniza
t iona
l lea
rnin
g ar
e th
esa
me
as th
e co
ndi t i
ons
that
are
ass
ocia
ted
with
the
3 sc
hool
lead
ersh
ip v
aria
bles
: prin
cipa
ltra
nsfo
rmat
iona
l lea
ders
hip,
act
ivel
y in
volv
edad
min
istra
t ive
team
s, a
nd d
istri
bute
d le
ader
ship
.Or
gani
zat io
nal l
earn
ing
is a
lso
a si
gni f i
cant
med
iato
r of p
rinci
pal a
nd le
ader
ship
team
ef fe
cts
on te
ache
rs’ w
ork
and
on s
tude
nt o
utco
mes
.
CSLA
bui
lds
prog
ram
and
eva
luat
ion
expe
rtise
by
crea
ting
a fra
mew
ork
for p
rofe
ssio
nal d
evel
op-
men
t tha
t can
be
used
in a
flex
ible
man
ner;
help
sid
entif
y re
sour
ces;
com
plem
ents
ext
erna
l eva
lua-
tion
by b
uild
ing
loca
l cap
acity
; use
s da
ta to
enco
urag
e st
aff t
o co
ntem
plat
e ho
w th
eir v
alue
shi
nder
or c
ontri
bute
to a
pro
gram
’s su
cces
s
Curri
culu
m fo
rum
: 8 is
sues
add
ress
ed b
ut m
ost
were
adm
inis
trativ
e; li
ttle
lear
ning
took
pla
ce.
Soci
al ju
stic
e &
equ
ity fo
rum
: inq
uiry
and
act
ion
were
inte
gral
, fee
dbac
k so
licite
d fro
m s
tude
nts.
Teac
hers
’ vie
ws o
n di
vers
ity a
nd d
iscr
imin
atio
nwe
re w
illin
gly
ques
tione
d bu
t not
vie
ws a
bout
curri
culu
m.
Need
s as
sess
men
t use
ful f
or fo
cusi
ng te
chsu
ppor
t, in
form
ing
refo
rm o
f sta
te p
ropo
sal
guid
elin
es. P
artic
ipan
t con
fiden
ce v
arie
d ac
ross
four
ski
ll ca
tego
ries;
mor
e pr
ogre
ss o
n pr
elim
i-
deve
lopm
ent o
f pro
fes-
sion
al c
omm
uni t i
es?
To e
xam
ine
the
natu
re o
for
gani
zat io
nal l
earn
ing
and
the
lead
ersh
ip p
ract
ices
and
proc
esse
s th
at fo
ster
orga
niza
t iona
l lea
rnin
g in
Aust
ral ia
n hi
gh s
choo
ls
To in
trodu
ce a
nd d
escr
ibe
the
appl
icat
ion
of C
SLA
impl
emen
tatio
n th
roug
hev
alua
tion
of s
choo
l-wid
edi
scip
line
plan
Wha
t kin
ds o
f iss
ues
are
addr
esse
d an
d de
cisi
ons
mad
e in
a s
choo
l tha
t has
striv
ed to
dem
ocra
tize
deci
sion
-mak
ing
and
beco
me
a le
arni
ng o
rgan
i-za
tion?
To d
ocum
ent s
ucce
ssfu
lin
crea
se in
age
ncy
capa
c-ity
as
a co
nseq
uenc
e of
ECB
initi
ativ
e; to
iden
tify
A pa
th m
odel
is u
sed
tote
st th
e re
lat io
nshi
psbe
twee
n sc
hool
-leve
lfa
ctor
s an
d sc
hool
outc
ome
mea
sure
s in
term
s of
stu
dent
s’pa
rt ici
pat io
n in
and
enga
gem
ent w
i thsc
hool
in A
ustra
lia
Impl
emen
tatio
n of
cont
inuo
us s
yste
m-
leve
l ass
essm
ent
(CSL
A) in
con
text
of
scho
ol-le
vel s
taff
capa
city
bui
ldin
g in
the
USA
Scho
ol w
ith e
stab
lishe
dst
ruct
ures
for s
hare
dde
cisi
on-m
akin
g an
dco
llect
ive
lead
ersh
ip;
char
acte
ristic
s of
lear
ning
org
aniza
tion
inAu
stra
lia
Stat
e-fu
nded
eva
luat
ion
supp
ort u
nit
com
mis
sion
ed to
Sil in
s et
al .
(200
2)
Smith
&Fr
eem
an (2
002)
Stev
enso
n(2
001)
Stev
enso
n et
al.
(200
2)
Teac
hers
and
stud
ents
from
96
seco
ndar
y sc
hool
sfro
m S
outh
Aust
ral ia
and
Tasm
ania
, N =
2503
(tea
cher
s)an
d N
= 35
00(s
tude
nts)
Not a
pplic
able
One
seco
ndar
ysc
hool
inAu
stra
lia; 1
/3 o
fte
achi
ng s
taff
14 c
omm
unity
-ba
sed
agen
cies
resp
onsi
ble
for
eval
uatin
g th
eir
proj
ects
, doc
umen
tan
d ar
t i fac
t col
lect
ion
Fiel
d st
udy,
2 p
hase
s:(a
) sur
vey
data
was
col le
cted
usi
ng tw
oqu
est io
nnai
res;
(b)
path
ana
lysi
sex
amin
ed th
e na
ture
and
stre
ngth
of
inte
rrela
t ions
hips
betw
een
varia
bles
Refle
ctiv
e ca
se s
tudy
:wr
itten
from
pers
pect
ive
of m
odel
deve
lope
rs/
impl
emen
ters
Case
stu
dy: o
bser
ved
mee
tings
, for
mal
inte
rvie
ws, i
nfor
mal
conv
ersa
tions
,co
llect
ion
and
anal
ysis
of re
leva
nt d
ocum
ents
Long
itudi
nal c
ase
stud
y: e
valu
atio
npr
opos
als,
con
fiden
cera
tings
re: m
ultip
le
(con
tinue
d on
nex
t pag
e)
THE CANADIAN JOURNAL OF PROGRAM EVALUATION116Ta
ble
1 (c
ontin
ued)
Stud
ySa
mpl
eCo
ntex
tM
etho
dsPu
rpos
e / R
esea
rch
Mai
n Fi
ndin
gsQu
est io
n(s)
nary
ste
ps a
nd d
esig
n th
an o
n an
alys
is a
ndut
i l iza
t ion.
Tec
hnic
al a
ssis
tanc
e es
sent
ial .
Exte
rnal
refo
rm v
iewe
d as
ext
rinsi
c m
otiv
ator
,te
ache
rs b
ecom
e in
trins
ical
ly m
otiv
ated
. Maj
orro
le fo
r lea
ders
hip,
stru
ctur
e fo
r cap
acity
bui
ldin
g,co
ntin
uous
flow
of i
nfor
mat
ion
and
com
mun
ica-
tion.
Cul
ture
of c
ontin
uous
dat
a us
e ov
er ti
me.
Elem
ents
of r
efor
m e
nviro
nmen
t im
porta
nt.
An e
mpo
werm
ent e
valu
atio
n wi
th it
s em
phas
is o
nse
lf-as
sess
men
t, fa
cilit
atio
n, a
nd c
apac
ity b
uild
-in
g ca
n m
eet t
he c
halle
nges
of d
iver
sity
yet
guar
ante
e th
at th
e ev
alua
tion
will
stre
ngth
en th
ein
divi
dual
loca
l pro
gram
s.
Mut
ual d
epen
denc
e be
twee
n sh
ared
vis
ion
and
lead
ersh
ip; b
etwe
en H
R de
velo
pmen
t, te
am le
arn-
ing,
and
per
sona
l mas
tery
; bet
ween
mut
ual m
od-
els/
info
rmat
ion
and
anal
ysis
/pro
cess
impr
ove-
men
t; be
twee
n sy
stem
s th
inki
ng/c
usto
mer
focu
s/st
rate
gic
plan
ning
; org
aniza
tiona
l per
form
ance
.
own
prog
ram
s
46 U
S sc
hool
ad-
min
istra
tors
,te
ache
rs, s
taff,
pare
nts,
and
stu
-de
nts
plus
12
dis-
trict
adm
inis
trato
rsin
Edi
son
Proj
ect
scho
ol
Not a
pplic
able
5 ca
seco
rpor
atio
ns
task
s, c
heck
l ist o
fev
alua
t ion
step
sco
mpl
eted
Long
i tudi
nal c
ase
stud
y: in
terv
iew
and
obse
rvat
ion
Refle
ctiv
e ca
se s
tudy
:wr
itten
from
poi
nt o
fvi
ew o
f eva
luat
ors
Mul
tiple
cas
e st
udy:
sem
i-stru
ctur
edin
terv
iews
mai
n ar
eas
of re
sist
ance
How
do th
e pr
oces
ses
and
prac
t ices
sur
roun
ding
the
use
of d
ata
for d
ecis
ion-
mak
ing
cont
ribut
e to
acu
lture
of c
ontin
uous
impr
ovem
ent?
To il
lust
rate
the
utili
ty a
ndfe
asib
ility
of e
mpo
werm
ent
eval
uatio
n in
a c
ompr
ehen
-si
ve s
tate
-fund
ed p
rogr
am
Wha
t are
the
mut
ual
depe
nden
cies
bet
ween
Seng
e’s le
arni
ng o
rgan
iza-
tion
chan
ge s
trate
gies
and
the
MBQ
NA fr
amew
ork?
incr
ease
eva
luat
ion
capa
city
bui
ldin
g: n
eeds
asse
ssm
ent ,
mod
ell in
g,te
ch a
ssis
tanc
e, t r
aini
ngin
the
USA
Edis
on P
roje
ct s
choo
l ,m
ajor
USA
refo
rmin
itiat
ive
fund
ed b
yfo
unda
tion
desi
gned
tocr
eate
inno
vativ
esc
hool
s th
at o
pera
tewi
thin
cur
rent
pub
licex
pend
iture
s
Larg
e st
ate-
wide
Toba
cco
Cont
rol
Prog
ram
, spo
nsor
ed b
yth
e Ca
lifor
nia
Stat
e De
pt.
of H
ealth
in U
SA
Aust
ralia
n co
rpor
atio
nsim
plem
entin
g TQ
M in
cont
ext o
f Mal
colm
Bald
ridge
Nat
iona
lQu
ality
Awa
rd (M
BNQA
)
Suth
erla
nd(2
004)
Tang
et a
l.(2
002)
Terz
iovs
ki e
t al.
(200
0)
117LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME
Eval
uat io
n of
BPR
had
the
fol lo
wing
ben
efi ts
:fa
cil i t
ated
the
chan
ge p
roce
ss a
nd le
d to
bet
ter
OL, l
ed to
pro
cess
inno
vat io
n su
ch a
s th
ein
trodu
ctio
n of
impr
oved
tech
nolo
gica
l cap
abi l i
-t ie
s in
the
com
pani
es.
Use
take
s pl
ace
in s
ocia
l con
text
; 4 ty
pes
ofar
gum
ents
abo
ut e
valu
ativ
e re
sul ts
; man
yex
ampl
es o
f pos
t hoc
legi
t imizi
ng a
ctio
ns;
eval
uatio
n in
cite
s ar
gum
ent a
nd c
an re
sult
inre
ason
ed d
ebat
e; w
ritte
n re
ports
offe
r com
mon
plat
form
for d
ialo
gue
and
argu
men
tatio
n.
Empl
oyin
g a
mor
e in
clus
ive
notio
n of
par
tici-
pant
s ch
ange
s th
e un
ders
tand
ing
of th
e te
rm“p
artic
ipat
ory
rese
arch
.” Pr
ogra
m p
artic
ipan
ts,
staf
f, go
vern
men
t fun
ders
, com
mun
ity p
artn
ers,
man
ager
s, a
nd e
valu
ator
s al
l hav
e a
capa
city
toem
powe
r and
be
empo
were
d. T
he s
ucce
ss o
f aso
cial
inte
rven
tion
is b
ette
r jud
ged
by w
hat
happ
ens
in th
e ov
eral
l env
ironm
ent t
han
by w
hat
happ
ens
to in
divi
dual
s an
d or
gani
zatio
ns in
the
colle
ctiv
e.
Is th
ere
a l in
k be
twee
nev
alua
t ion
act iv
i t ies
that
are
part
of a
BPR
exe
rcis
ean
d or
gani
zat io
nal l
earn
ing
and
inno
vat io
n?
To c
ompa
re h
ow a
genc
yev
alua
t ions
con
duct
ed,
obje
ctiv
es, a
nd ro
leev
alua
tion
play
ed in
over
all f
ram
ewor
k of
man
agem
ent d
evel
opm
ent
How
the
conc
epts
of
parti
cipa
tion,
em
powe
r-m
ent,
and
soci
al c
hang
ear
e co
nstru
cted
Orga
niza
t ions
inco
nstru
ctio
n in
dust
ry in
3 Eu
rope
an c
ount
ries
that
und
erwe
nt a
busi
ness
pro
cess
re-
engi
neer
ing
exer
cise
(BPR
)
Gove
rnm
ent a
genc
ies
inFi
nlan
d in
div
erse
pol
icy
sect
ors
Expe
rienc
es w
ith th
eAt
lant
ic C
omm
unity
Actio
n Pr
ogra
m fo
rCh
ildre
n (C
APC)
regi
onal
eva
luat
ion
inCa
nada
Desc
ript iv
e ca
sest
udie
s: th
e da
ta fr
omse
mi-s
truct
ured
inte
rvie
ws fo
l low
apa
t tern
of a
naly
sis
Met
a-ev
alua
t ion:
sem
i-stru
ctur
edin
terv
iews
Refle
ctiv
e ca
se s
tudy
:wr
itten
from
pers
pect
ive
ofev
alua
tors
3 ca
seco
rpor
atio
ns
27 a
genc
y an
dsu
perv
isin
gde
partm
ent
pers
onne
l
Not a
pplic
able
Vako
la, M
.(2
000)
Valo
virta
(200
2)
Vand
erPl
aat e
tal
. (20
01)
THE CANADIAN JOURNAL OF PROGRAM EVALUATION118
The remaining studies were relatively evenly distributed over fourdesign types. Five were qualitative case studies (e.g., Balthasar &Reider, 2000; Katz, Sutherland & Earl, 2002), and an additionalfive were comparative case studies (e.g., Connor & Tanjasiri, 1999;Scribner, Cockrell, Cockrell, & Valentine, 1999) Six studies werebased on longitudinal data collection (e.g., O’Sullivan & D’Agostino,2002; Robinson & Cousins, 2004; Sutherland, 2003) or multi-methodfield studies (e.g., Matlay, 2000). Many of the studies are NorthAmerican, taking place in the U.S.A. (17) and Canada (6), but sev-eral are from other jurisdictions including the U.K. (Biott & Cook,2000; Boaz, 2002; Matlay, 2000), Australia (Silins, Mulford, &Zarins, 2002; Stevenson, 2001; Terziovski, Howell, Sohal, &Morrison, 2000), Spain (Chacon-Moscoso, Anguera-Argilaga, Perez-Gil, & Holgado-Tello, 2002), Finland (Valovirta, 2002), Switzerland(Balthasar & Rieder, 2000), and South Africa (Jacobs, 2000). Twostudies (Mackay, 2002; Vakola, 2000) involved data collection frommultiple countries.
Method
Having identified and obtained the articles in our sample, we di-vided them among the authors for analysis and summarization. Eachanalyst produced a one-page summary for each of the allotted arti-cles. The summary was structured by the column headings appear-ing in Table 1. In summarizing the articles, analysts looked forevidence and verbatim quotations that illustrated relationshipsamong the variables of interest. Verbatim quotations were capturedin bold with appropriate page numbers. All summaries were com-piled into a single electronic file ordered alphabetically by author.The file was then converted to Folio Views (4.3) and coded accord-ing to the main variables of interest by one of the analysts. In work-ing through the files, the analyst developed the conceptualframework showing relationships among the variables (see Figure1). This figure and descriptive text were then circulated among theother authors for input and verification. The analyst made somemodifications to the framework and proceeded to integrate the find-ings as presented below. Available to the analyst were copies of theoriginal studies as well as the coded summaries.
All in all, we judge the knowledge base linking evaluative