Volume 30 Number 3 Research on Evaluation Use 2009 ......Alkin (2003). Finally, the team conducted a manual review (looking for relevant research based on titles) of the following

Research on Evaluation Use

A Review of the Empirical LiteratureFrom 1986 to 2005

Kelli Johnson

Lija O. Greenseid

Stacie A. Toal

Jean A. King

Frances Lawrenz

University of Minnesota

Boris Volkov

University of North Dakota

This paper reviews empirical research on the use of evaluation from 1986 to 2005 using Cousins

and Leithwood’s 1986 framework for categorizing empirical studies of evaluation use conducted

since that time. The literature review located 41 empirical studies of evaluation use conducted

between 1986 and 2005 that met minimum quality standards. The Cousins and Leithwood

framework allowed a comparison over time. After initially grouping these studies according

to Cousins and Leithwood’s two categories and twelve characteristics, one additional

category and one new characteristic were added to their framework. The new category is

stakeholder involvement, and the new characteristic is evaluator competence (under the

category of evaluation implementation). Findings point to the importance of stakeholder

involvement in facilitating evaluation use and suggest that engagement, interaction, and

communication between evaluation clients and evaluators is critical to the meaningful use of

evaluations.

Keywords: evaluation use; evaluation influence; stakeholder involvement; literature review;

research on evaluation

In recent years, scholars have advanced calls for research on program evaluation andespecially on the impact of evaluations (e.g., Henry & Mark, 2003b; Scriven, 2007). AsHenry and Mark state, there is ‘‘a serious shortage of rigorous, systematic evidence that can

guide evaluation or that evaluators can use for self-reflection or for improving their next

evaluation’’ (2003b, p. 69). A time-honored method for providing guidance entails synthesiz-

ing existing research to identify what is known about evaluations and what remains to be

investigated. This is the approach taken in the current review of evaluation use, one of the few

Authors’ Note: This material is based on work supported by the National Science Foundation under Grant No. REC

0438545. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the

authors and do not necessarily reflect the views of the National Science Foundation. The authors gratefully

acknowledge Stuart Appelbaum for his contributions to this article. Correspondence concerning this article should

be addressed to Kelli Johnson, University of Minnesota, 2221 University Avenue SE, Suite 345, Minneapolis, MN

55414; phone: þ1 (612) 624-1457; e-mail: [email protected].

American Journal of

Evaluation

Volume 30 Number 3

September 2009 377-410

# 2009 American EvaluationAssociation

10.1177/1098214009341660

http://aje.sagepub.com

hosted at

http://online.sagepub.com

377

at WESTERN MICHIGAN UNIVERSITY on June 29, 2010 http://aje.sagepub.comDownloaded from


topics in evaluation on which numerous empirical studies exist. Christie (2007, p. 8) notes,

‘‘Evaluation utilization is arguably the most researched area of evaluation and it also receives

substantial attention in the theoretical literature.’’ We define evaluation use or utilization—

evaluation scholars use the terms interchangeably—as the application of evaluation processes,

products, or findings to produce an effect.

Since the 1970s, naming the types of evaluation use has been the subject of continuing dis-

cussion. In reviewing this discussion to date, Alkin and Taut (2003) label two distinct aspects

of use: process use and use of evaluation findings. Process use is the newer concept, defined by

Patton (1997, p. 90) as ‘‘individual changes in thinking and behavior and program or organiza-

tional changes in procedures and culture that occur among those involved in evaluation as

a result of the learning that occurs during the evaluation process.’’ The use of findings is

traditionally divided into three types: instrumental, conceptual, or symbolic (King & Pechman,

1984; Leviton & Hughes, 1981). Instrumental use refers to instances where someone has used

evaluation knowledge directly. Conceptual use refers to cases when no direct action has been

taken but where people’s understanding has been affected. Symbolic use refers to examples

when a person uses the mere existence of the evaluation, rather than any aspect of its results,

to persuade or to convince.

Moving beyond the first quarter century of use research, the new millennium has witnessed

theoretical activity that has reconceptualized the field’s understanding of its impact. Scholars

now view evaluations as having intangible influence on individuals, programs, and commu-

nities. Focusing solely on the direct use of either evaluation results or processes has not

adequately captured broader level influences (Alkin & Taut, 2003; Henry & Mark, 2003a,

2003b; Kirkhart, 2000; Mark & Henry, 2004). What has potentially emerged from this activity

is a more nuanced understanding of evaluation’s consequences using evaluation influence as a

unifying construct. Kirkhart’s ‘‘integrated theory’’ defines influence as ‘‘the capacity or power

of persons or things to produce effects on others by intangible or indirect means’’ (2000, p. 7).

Kirkhart envisions three dimensions of evaluation influence, represented as a cube-like figure:

source (evaluation process or results), intention (intended or unintended), and time (immedi-

ate, end-of-cycle, long-term).

Mark and Henry (Henry & Mark, 2003a, 2003b; Mark & Henry, 2004; Mark, Henry, &

Julnes, 1999) have also pushed for broadening the way evaluators conceptualize the conse-

quences of their work. They argue that the goal of evaluation is social betterment and suggest

the need to identify the mechanisms through which evaluations lead to this ultimate goal along

differing paths of influence and at different levels (i.e., individual, interpersonal, and collec-

tive). Mark and Henry map out a logic model for evaluation, focusing on evaluation conse-

quences related to the improvement of social conditions. Just as program theory connects

program activities with outcomes while also explaining the processes through which the

outcomes are achieved, program theory of evaluation by Mark and Henry identifies evaluation

as an intervention with social betterment as its ultimate outcome. They label traditional notions

of instrumental, conceptual, and persuasive use more specifically as, for example, skill acqui-

sition, persuasion, or standard setting. These, then, would be the mechanisms through which

social betterment can be achieved.

Building on these ideas, Alkin and Taut (2003) carefully distinguish between evaluation

use and influence. To them, evaluation use ‘‘refers to the way in which an evaluation and

information from the evaluation impacts the program that is being evaluated’’ (Alkin &

Taut, 2003, p. 1). In their view, evaluators are aware of these evaluation impacts, both

intended and unintended. By contrast, ‘‘the concept of influence adds to the concept of use

in instances in which an evaluation has unaware/unintended impacts’’ (p. 9, emphasis in

original).

378 American Journal of Evaluation / September 2009



Structuring the Present Review

In structuring this literature review, we considered several options. Cousins (2003) draws a

logic model for program evaluation that builds on the knowledge utilization literature, but its

focus on participatory evaluation made it inappropriate for a review of evaluation use research.

Cousins, Goh, Clark, and Lee (2004) present a comprehensive framework of evaluative

inquiry as an organizational learning system, but, again, it includes many concepts other than

evaluation use.

Given the emergence of influence as a construct, another possibility was to apply the new

concept to analyze the existing literature. This proved impractical for three reasons. First,

some of the research we reviewed was conducted before Kirkhart’s (2000) work was pub-

lished. Second, given the newness of the term, there was little empirical research on influence,

although we did include it in our searches. Indeed, even studies conducted in the 5 years since

the term emerged (2000–2005) did not necessarily examine evaluation influence; moreover,

examining use through the lens of influence was not necessarily helpful because influence

is indirect and we were examining direct use. Third, and perhaps most important, the concept

of influence presented in Henry and Mark (2003a, 2003b) and Mark and Henry (2004) was not

defined, and the discussion of pathways, processes, and mechanisms did not provide sufficient

clarity to structure the review (Nunneley, 2008; Weiss, Murphy-Graham, & Birkeland, 2005).

We decided, therefore, to use the seminal study that Cousins and Leithwood conducted in

1986—one of the most ambitious and rigorous reviews of empirical research on evaluation use

ever conducted—as the underlying structure for this review, as well as more recent work by

Shulha and Cousins (1997). Although Cousins’ own conceptualizations of the topic have

evolved since this point, the taxonomy of evaluation use presented in the 1986 model was the

most comprehensive, well defined, and concrete.

Cousins and Leithwood Framework

Cousins and Leithwood (1986) identified 65 empirical studies of evaluation use conducted

between 1971 and 1985 through computerized searches of keywords including ‘‘evaluation

utilization,’’ ‘‘data use,’’ ‘‘decision making,’’ and ‘‘knowledge utilization.’’ They supplemen-

ted this process with manual searches of relevant journals and other literature reviews. After

establishing their sample, Cousins and Leithwood coded each study according to its orienta-

tion toward dependent variables (i.e., the type of use examined: use as decision making, use

as education, use as the processing of information, or ‘‘potential’’ use) and its orientation

toward independent variables.

The aspects of evaluation use examined in the 65 empirical studies were clustered into two

categories of factors related to evaluation use: (a) characteristics of evaluation implementa-

tion, and (b) characteristics of the decision or policy setting. Each of these categories contained

six characteristics. The six evaluation implementation characteristics were (a) evaluation

quality, (b) credibility, (c) relevance, (d) communication quality, (e) findings, and (f) timeli-

ness. The six decision- or policy-setting characteristics were (a) information needs, (b) decision

characteristics, (c) political climate, (d) competing information, (e) personal characteristics,

and (f) commitment or receptiveness to evaluation. Using a ‘‘prevalence of relationship’’

index, Cousins and Leithwood (1986) identified evaluation quality as the most important char-

acteristic, followed by decision characteristics, receptiveness to evaluation, findings, and

relevance.

Johnson et al. / Research on Evaluation Use 379



Shulha and Cousins (1997) described developments that had occurred since the review by

Cousins and Leithwood, including the following:

The rise of considerations of context as critical to understanding and explaining use; identification

of process use as a significant consequence of evaluation activity; expansion of conceptions of use

from the individual to the organization level; and diversification of the role of the evaluator to

facilitator, planner and educator/trainer (p. 195).

The present review incorporates these developments as well.

Importantly, these two major reviews of the use literature (Cousins & Leithwood, 1986;

Shulha & Cousins, 1997) differ in that the first considered only empirical research whereas

the more recent included theoretical or reflective case narratives in addition to empirical stud-

ies. Yet, many potentially instructive studies were excluded from the 1997 review, either

because they were conducted as doctoral dissertations or because they were not published

in journals. Neither review took into account the quality of the evidence gathered in the indi-

vidual studies when synthesizing the results. Consequently, the findings from studies in which

there could be serious methodological flaws potentially were presented alongside higher qual-

ity, rigorously conducted studies. To rectify these concerns, the current review included

empirical studies of evaluation use; examined journal articles, dissertations, reports, and book

chapters; and screened each study according to a predetermined set of criteria related to meth-

odological quality. In this review, we employ the term ‘‘use’’ rather than ‘‘influence,’’

although we view use broadly. We attempt to identify it as ‘‘process use’’ or ‘‘use of findings’’

and classify it as instrumental, conceptual, or symbolic.

Method

The research team collected relevant publications by conducting electronic searches for the

terms ‘‘evaluation utilization,’’ ‘‘evaluation use,’’ and ‘‘evaluation influence’’ in PsycINFO,

Education Resources Information Center (ERIC), Education (Sage), Social Services Abstracts,

Sociological Abstracts, and Digital Dissertations in keywords, titles, descriptors, and abstracts.

Additionally, the team consulted other published literature reviews, including Hofstetter and

Alkin (2003). Finally, the team conducted a manual review (looking for relevant research

based on titles) of the following evaluation-related journals: American Journal of Evaluation,

Canadian Journal of Program Evaluation, Evaluation, Evaluation Practice, Evaluation

and Program Planning, Evaluation Review, New Directions for Evaluation, and Studies in

Educational Evaluation. The searches examined only the literature written in English,

although the authors did not exclude research conducted outside the United States.

The searches returned over 600 journal articles, reports, and book chapters and 48 disserta-

tions. After scanning publication titles and abstracts, the team eliminated clearly irrelevant

publications. Then, the team closely reviewed 321 abstracts to assess whether the publication

met the following criteria: (a) an empirical research study (to be considered an empirical study

the article had to present information about the data collection methods used to inform the

claims made); (b) a focus on program or policy evaluation or needs assessment (not personnel

evaluation, accountability/student assessment studies, data-driven decision making, etc.); (c) a

published journal article, book, publicly accessible evaluation report, or dissertation (not a

conference presentation or other nonpublished work); (d) the inclusion of evaluation use or

influence as at least one of the variables under study; and (e) a publication date between

January 1, 1986 and December 31, 2005.




After the abstract review, the team identified 98 publications that warranted a full-text

review; these were subsequently screened again on all five criteria. This process yielded 47

articles that initially comprised the basis for this analysis. At least two trained screeners cate-

gorized and critiqued each study using a standardized review form developed and refined with

the input of several evaluation experts.

The rating form contained questions about each study’s methodology, choice of theory,

operationalization of dependent variable (measures of use), and independent variables (char-

acteristics affecting use). In addition, as noted, a quality rating was assigned to each study. The

quality rating was based on criteria adapted from Guarino, Santibañez, Daley, and Brewer

(2004) and Guba and Lincoln (1989). It considered aspects such as the clarity of the problem

statement, soundness of research design, strength of the link between evidence presented and

conclusions, and the extent to which bias was addressed. The team also assessed the sample

size and selection, measurement of variables, and statistical interpretation of the quantitative

studies, as well as the methodological appropriateness, transparency, descriptive richness, and

statement of researcher biases of the qualitative studies.

The reviewers independently assigned quality ratings across five levels: poor, adequate-low,

adequate-solid, adequate-high, and excellent. If the screeners did not agree on any particular

aspect of the review, the article was brought to a team meeting during which it was discussed

and consensus among the six researchers was reached regarding the rating. Although this

consensus-driven process for reviewing and assigning quality ratings was time-consuming, the

resulting judgments represent the agreement of two professors of evaluation and then four

evaluation doctoral students. We believe our process was both representative and fair.

On completion of this in-depth screening process, 41 of 47 studies (87.2%) were found to beadequate or above. Six of the studies (12.8%) were rated as poor and eliminated from our sam-ple. These ‘‘poor’’ studies suffered from a cursory description of the methods, weak sampling

or data analysis methods, poor measurements of use (e.g., not providing definitions of use and/

or using only one question as a measure of use), poorly supported generalizations, and/or

inadequate attention to likely researcher biases. The 41 studies that exceeded the minimum

quality criteria were used in the analyses presented below and are described in detail in the

Appendix. Six of these studies were published outside the United States.

Findings

Findings from the 41 studies are presented following the framework of 2 categories—

evaluation implementation and decision or policy setting—and 12 characteristics by Cousins

and Leithwood (1986). These two categories were helpful for organizing the majority of the

studies found in this recent literature. Nearly half of the articles (20 of 41) looked at the eva-

luation implementation category, and an equal number (20 of 41) examined the decision- or

policy-setting category. The characteristics under each of these categories were all examined

in at least one article, with the most prevalent characteristic, communication quality, of

Cousins and Leithwood appearing in 11.

However, as suggested by Shulha and Cousins (1997), changes in the conceptualizations

about use have occurred, so new characteristics might be expected to emerge. In fact, 25 of

the 41 studies in this review examined elements that were not covered by the 1986 framework.

Consequently, we added one characteristic—evaluator competence—to the evaluation

implementation category. In addition, we created an entirely new category—stakeholder

involvement—to accommodate the categorization of the 25 studies that examined aspects

of evaluation use that were not represented in the original Cousins and Leithwood framework.




Evaluator competence. This is a new characteristic under evaluation implementation that

has emerged since the development of the Cousins and Leithwood framework. Of the 41 stud-

ies in this review, six addressed the characteristics of evaluators, suggesting that evaluation

professionals play an important role in conducting evaluations that get used, albeit for different

reasons. Although the characteristic of credibility by Cousins and Leithwood gave some con-

sideration to the evaluator’s title or reputation, the definition did not extend to the influential

nature of the evaluator’s personal competence or leadership as a means of affecting the level of

evaluation use. Moreover, whereas the credibility characteristic addresses what the evaluator

does (e.g., methods selected, criteria used), the new evaluator competence characteristic

focuses more on who the evaluator is.

Stakeholder involvement. This is a new category that has been added to the original Cousins

and Leithwood framework to account for more recent research. The addition of this category

reflects the increased research focused on participatory evaluation approaches, stakeholder or

decision-maker participation, and/or stakeholder or decision-maker involvement since 1985.

Under the rubric of stakeholder involvement, we have identified nine characteristics. Eight

of them mirror those identified by Cousins and Leithwood but with the addition of involve-

ment to each. The original framework included research on the impact of direct decision-

maker involvement on use under commitment or receptiveness to evaluation. However, in the

current review, over half (23 of 41) of the studies addressed involvement, and the bulk of these

suggested that it was related to other category characteristics in their relationship with use.

Using the resulting modified Cousins and Leithwood framework, we classified the 41 stud-

ies of evaluation use from 1986 to 2005 according to 3 categories and 22 specific character-

istics. The most frequently studied characteristics were ‘‘involvement and commitment/

receptiveness to evaluation’’ (14 studies), followed by communication quality (11 studies) and

personal characteristics of users (9 studies). The least frequently studied characteristics were

‘‘involvement and information needs’’ and ‘‘involvement and decision characteristics,’’ each

appearing in a single study. About 40% of the studies (16 of 41) examined only a single char-acteristic, with half of that group (8 of 16) studying a characteristic under the stakeholder

involvement category. The remainder of the studies examined multiple characteristics, ranging

from two to nine characteristics per study.

Table 1 defines each category, presents its related characteristics, and lists the studies that

examined each. Because the variables described in the studies did not always allow for obvious

categorization into the framework, this represents the authors’ best effort at accurately inter-

preting and deciding where the studies fit. The Appendix provides a summary of each study’s

focus, types of use, sample, categories, and findings. In terms of the types of evaluation use,

the information presented in the Appendix shows that the clear majority of the studies focused

on use of findings rather than process use. Only three studies examined process use, perhaps

because the concept of explicit process use is fairly recent. Within the use of findings, instru-

mental use was studied more frequently than conceptual use, which was typically linked to

instrumental use when researchers asked respondents whether actions were likely to be taken.

There were only a few studies that examined symbolic use.

Discussion

The purpose of this study was to review empirical research on evaluation use for the 25-year

period between 1986 and 2005. Basing the review on the framework of Cousins and

Leithwood allowed a comparison over time, and including other types of research (e.g.,




Ta

ble

1

Stu

die

sE

xa

min

ing

Use

by

Ca

teg

ory

an

dC

ha

ract

eris

tics

of

Va

ria

ble

s

Cat

egory

Char

acte

rist

icD

escr

ipti

on

of

Char

acte

rist

ic#

of

Stu

die

sR

elat

ionsh

ipto

Eval

uat

ion

Use

Art

icle

sth

atS

tudie

dC

har

acte

rist

ic

Eval

uat

ion

imple

men

tati

on

Com

munic

atio

n

qu

alit

y

Cla

rity

and

freq

uen

cyof

report

ing

resu

lts,

eval

uat

or

advoca

cyfo

r

resu

lts,

bre

adth

of

dis

sem

inat

ion.

Als

oin

cludes

the

type

of

reco

m-

men

dat

ions

inth

ere

port

and

the

pro

cess

of

com

munic

atio

n

bet

wee

nev

aluat

ors

and

clie

nts

11

Fre

quen

tly

among

the

most

import

ant

elem

ents

rela

ted

to

eval

uat

ion

use

.D

etai

led,

acti

onab

le,

evid

ence

-bas

ed

reco

mm

endat

ions

incr

ease

d

use

.B

yco

ntr

ast,

two

studie

s

found

no

rela

tionsh

ipw

ith

use

Bober

and

Bar

tlet

t(2

004)

Boyer

and

Lan

gbei

n(1

991)

Chin

(2003)

Eis

endra

th(1

988)

Johnst

on

(1986)

Mal

en,

Murp

hy,

and

Gea

ry(1

988)

Mar

ra(2

003)

Mar

shan

dG

lass

ick

(1988)

Rock

wel

l,D

ickey

,an

dJa

sa(1

990)

Shea

(1991)

Sle

ezer

(1987)

Tim

elin

ess

Tim

ing

of

the

eval

uat

ion

inla

rger

conte

xt;

tim

elin

ess

of

report

ing

when

eval

uat

ion

isco

mple

ted;

tim

ing

of

dis

sem

inat

ion

todec

isio

n

mak

ers

7M

ost

found

posi

tive

rela

tionsh

ip

bet

wee

nti

min

gan

dev

aluat

ion

use

.O

ne

study

found

that

tim

elin

ess

was

not

import

ant

in

det

erm

inin

guse

Bam

ber

ger

(2004)

Bar

rios

(1986)

Bober

and

Bar

tlet

t(2

004)

Boyer

and

Lan

gbei

n(1

991)

Eis

endra

th(1

988)

Rock

wel

let

al.

(1990)

Shea

(1991)

Eval

uat

or

com

pet

ence

a

Per

sonal

char

acte

rist

ics

of

the

eva-

luat

or

outs

ide

the

eval

uat

ion

pro

-

cess

,le

vel

of

cult

ura

lco

mpet

ence

,

lead

ersh

ipst

yle

of

eval

uat

or

6M

ost

studie

ssu

gges

tth

atev

alua-

tor

com

pet

ence

isim

port

ant

to

eval

uat

ion

use

Bar

rios

(1986)

Boyer

and

Lan

gbei

n(1

991)

Cal

lahan

,T

om

linso

n,

Hunsa

ker

,

Bla

nd,

and

Moon

(1995)

Cousi

ns

(1996)

Gre

ene

(1987)

Shea

(1991)

Eval

uat

ion

qual

ity

Char

acte

rist

ics

of

the

eval

uat

ion

pro

cess

,so

phis

tica

tion

of

met

hods,

rigor,

type

of

eval

uat

ion

model

6S

om

est

udie

sfo

und

ali

nk

bet

wee

nqual

ity

and

use

,

alth

ough

less

import

ant

than

reco

mm

endat

ions

and

com

mu-

nic

atio

n.

One

study

did

not

find

rela

tionsh

ipbet

wee

n

qual

ity

and

use

Bam

ber

ger

(2004)

Bober

and

Bar

tlet

t(2

004)

Johnst

on

(1986)

Rock

wel

let

al.

(1990)

Shea

(1991)

Pott

s(1

998)

(conti

nued

)


383



Ta

ble

1.

(co

nti

nu

ed)

Cat

egory

Char

acte

rist

icD

escr

ipti

on

of

Char

acte

rist

ic#

of

Stu

die

sR

elat

ionsh

ipto

Eval

uat

ion

Use

Art

icle

sth

atS

tudie

dC

har

acte

rist

ic

Fin

din

gs

Nat

ure

of

findin

gs

(e.g

.,posi

tive

or

neg

ativ

e),

exte

nt

of

congru

ence

wit

hau

die

nce

expec

tati

ons,

val

ue

of

findin

gs

for

dec

isio

nm

akin

g

6M

ixed

concl

usi

ons.

Intw

ost

ud-

ies

findin

gs

wer

eim

port

ant

to

use

,th

ough

less

soth

anco

m-

munic

atio

n,

tim

elin

ess,

and

eval

uat

ion

qual

ity

Bar

rios

(1986)

Bober

and

Bar

tlet

t(2

004)

Boyer

and

Lan

gbei

n(1

991)

Johnso

n(1

993)

Mal

enet

al.

(1988)

Wei

ss,

Murp

hy-G

raham

,an

d

Bir

kel

and

(2005)

Rel

evan

ceE

xte

nt

tow

hic

hth

ein

form

atio

npro

-

vid

edin

the

eval

uat

ion

isre

levan

t

toth

edec

isio

nm

aker

,an

dth

e

org

aniz

atio

nal

loca

tion

of

the

eval

uat

or

6M

ixed

concl

usi

ons.

Tw

ost

udie

s

did

not

find

rele

van

ceto

be

import

ant

touse

,but

two

studie

sfo

und

stro

nger

rela

-

tionsh

ips

bet

wee

nin

form

atio

n

rele

van

cean

duse

Bar

rios

(1986)

Bober

and

Bar

tlet

t(2

004)

Boyer

and

Lan

gbei

n(1

991)

Cousi

ns

(1995)

Gre

ene

(1987)

Shea

(1991)

Cre

dib

ilit

yT

he

obje

ctiv

ity,

bel

ievab

ilit

y,

and

appro

pri

aten

ess

of

the

eval

uat

ion

pro

cess

and/o

rof

the

acti

vit

ies

of

the

eval

uat

or

4S

pli

tfi

ndin

gs.

Tw

ost

udie

sfo

und

stro

ng

rela

tionsh

ipw

ith

eva-

luat

ion

use

;tw

ost

udie

sfo

und

no

such

rela

tionsh

ip

Bar

rios

(1986)

Bober

and

Bar

tlet

t(2

004)

Boyer

and

Lan

gbei

n(1

991)

Johnso

n(1

993)

Dec

isio

nor

poli

cy

sett

ing

Per

sonal

char

acte

rist

ics

Char

acte

rist

ics

of

the

eval

uat

ion

use

r,

for

exam

ple

,org

aniz

atio

nal

role

of

dec

isio

nm

aker

,in

form

atio

npro

-

cess

ing

style

,so

cial

char

acte

ris-

tics

,an

dso

on

9D

iffe

rence

sin

use

rs’

lear

nin

g

style

s,jo

bposi

tions,

adm

inis

-

trat

ive

level

,an

dex

per

ience

level

infl

uen

ceth

euse

of

eval

uat

ions

Bober

and

Bar

tlet

t(2

004)

Boyer

and

Lan

gbei

n(1

991)

Car

pin

ello

(1989)

Com

bs

(1999)

Cro

tti

(1993)

Ear

l(1

995)

Hopst

ock

,Y

oung,

and

Zeh

ler

(1993)

Mar

ra(2

003)

San

thiv

eera

n(1

995)

Com

mit

men

tan

d/

or

rece

pti

ven

ess

toev

aluat

ion

Use

rat

titu

des

tow

ard

the

eval

uat

ion

and

com

mit

men

tto

conduct

ing

eval

uat

ion;

the

exte

nt

tow

hic

hth

e

org

aniz

atio

nis

resi

stan

tto

eval

ua-

tion;

the

open

-min

ded

nes

sof

eva-

luat

ion

stak

ehold

ers

8S

om

est

udie

sfo

und

that

com

mit

-

men

t,ac

tive

org

aniz

ing

effo

rts,

and

support

ive

bac

ker

s

incr

ease

duse

.O

ne

study

found

that

atti

tude

tow

ard

eval

uat

ion

did

not

affe

ctuse

Boyer

and

Lan

gbei

n(1

991)

Cro

tti

(1993)

Johnso

n(1

993)

Mal

enet

al.

(1988)

Mar

ra(2

003)

McC

orm

ick

(1997)

Rin

ne

(1994)

San

thiv

eera

n(1

995)

(conti

nued

)


384



Poli

tica

lcl

imat

eT

he

poli

tica

lori

enta

tion

of

the

peo

ple

who

com

mis

sioned

the

eval

uat

ion,

the

exte

nt

tow

hic

hdec

isio

nm

aker

isdep

enden

ton

exte

rnal

sponso

rs,

inte

rnal

rival

ries

,budget

fights

,

and

pow

erst

ruggle

s

6G

ener

ally

,at

tendin

gto

poli

tica

l

clim

ate

was

found

toin

crea

se

use

Eis

endra

th(1

988)

Had

dock

(1998)

Johnst

on

(1986)

Mal

enet

al.

(1988)

San

thiv

eera

n(1

995)

Wei

sset

al.

(2005)

Dec

isio

n

char

acte

rist

ics

The

signif

ican

ceof

the

dec

isio

nor

eval

uat

ion

pro

ble

m,

the

type

of

dec

isio

nto

be

mad

e,th

enovel

tyof

the

pro

gra

mar

ea

5E

ach

of

the

five

studie

sre

port

ed

connec

tions

bet

wee

ndec

isio

n

char

acte

rist

ics

and

eval

uat

ion

use

Bar

rios

(1986)

Bro

wn-M

cGow

an(1

992)

Eis

endra

th(1

988)

Mal

enet

al.

(1988)

New

man

,B

row

n,

and

Riv

ers

(1987)

Com

pet

ing

info

rmat

ion

Info

rmat

ion

rela

ted

toth

esu

bje

ctof

the

eval

uat

ion

and

avai

lable

to

stak

ehold

ers

from

outs

ide

the

eva-

luat

ion

pro

cess

,th

atis

,th

rough

per

sonal

obse

rvat

ion,

that

com

-

pet

esw

ith

eval

uat

ion

dat

a

3C

ontr

adic

tory

findin

gs.

One

study

found

that

ala

rge

amount

of

com

pet

ing

info

rmat

ion

did

not

affe

ctin

stru

men

tal

use

,

wher

eas

anoth

erfo

und

that

hig

h-l

evel

poli

cyoff

icia

lsuse

d

the

eval

uat

ion

resu

lts

only

when

they

wer

esu

pport

edby

oth

erso

urc

esof

info

rmat

ion

Eis

endra

th(1

988)

Johnso

n(1

993)

Wei

sset

al.

(2005)

Info

rmat

ion

nee

ds

of

the

eval

ua-

tion

audie

nce

s

Info

rmat

ion

nee

ds

of

the

eval

uat

ion

audie

nce

,th

ety

pes

of

info

rmat

ion,

the

num

ber

of

audie

nce

sw

ith

dif

-

feri

ng

info

rmat

ion

nee

ds,

tim

e

pre

ssure

,an

dper

ceiv

ednee

dfo

r

eval

uat

ion

2B

oth

studie

sfo

und

that

atte

ndin

g

toth

eau

die

nce

’sin

form

atio

n

nee

ds

posi

tivel

yin

fluen

ced

the

use

of

eval

uat

ion

resu

lts

Hopst

ock

etal

.(1

993)

Rin

ne

(1994)

(conti

nued

)


385



Ta

ble

1.

(co

nti

nu

ed)

Cat

egory

Char

acte

rist

icD

escr

ipti

on

of

Char

acte

rist

ic#

of

Stu

die

sR

elat

ionsh

ipto

Eval

uat

ion

Use

Art

icle

sth

atS

tudie

dC

har

acte

rist

ic

Sta

keh

old

er

involv

emen

t

Involv

emen

tw

ith

com

mit

men

tor

rece

pti

ven

ess

to

eval

uat

ion

Involv

ing

eval

uat

ion

stak

ehold

ers

crea

tes

aco

mm

itm

ent

or

rece

p-

tiven

ess

toev

aluat

ion

14

For

the

most

par

t,co

mm

itm

ent

that

was

stre

ngth

ened

by

involv

emen

tin

the

eval

uat

ion

was

found

toposi

tivel

yin

flu-

ence

eval

uat

ion

use

.In

one

study,

the

involv

emen

tof

a

com

mit

ted

exec

uti

ve

off

icer

was

esse

nti

alto

the

imple

-

men

tati

on

of

eval

uat

ion

findin

gs

Alt

schuld

,Y

oon,

and

Cull

en(1

993)

Ayer

s(1

987)

Bar

rios

(1986)

Bro

wn-M

cGow

an(1

992)

Cal

lahan

etal

.(1

995)

Ear

l(1

995)

Eis

endra

th(1

988)

Gre

ene

(1987)

Gre

ene

(1988)

Had

dock

(1998)

Laf

leur

(1995)

Lee

and

Cousi

ns

(1995)

Rock

wel

let

al.

(1990)

Shea

(1991)

Involv

emen

tw

ith

com

munic

atio

n

qu

alit

y

Sta

keh

old

erin

volv

emen

tpro

mote

s

impro

ved

com

munic

atio

n

5A

llfi

ve

studie

sid

enti

fied

way

sin

whic

hst

akeh

old

erin

volv

emen

t

led

togre

ater

use

Bam

ber

ger

(2004)

Cousi

ns

(1995)

Fors

s,C

rack

nel

l,an

dS

amse

t(1

994)

Gre

ene

(1988)

Laf

leur

(1995)

Dir

ect

stak

ehold

er

involv

emen

t

The

dir

ect

rela

tionsh

ipbet

wee

n

involv

emen

tan

dev

aluat

ion

use

4A

llst

udie

sre

port

edin

volv

e-

men

t’s

posi

tive

infl

uen

ceon

var

ious

types

of

use

Cai

(1996)

Pre

skil

lan

dC

arac

elli

(1997)

Sper

lazz

a(1

995)

Turn

bull

(1999)

Involv

emen

tw

ith

cred

ibil

ity

Sta

keh

old

erin

volv

emen

tle

dto

incr

ease

dcr

edib

ilit

yof

the

eva-

luat

ion

pro

cess

and/o

rth

e

eval

uat

or

4T

hre

eof

the

four

studie

sobse

rved

ast

rong

rela

tionsh

ipw

ith

use

Cousi

ns

(1995)

Gre

ene

(1987)

Laf

leur

(1995)

Shea

(1991)

Involv

emen

tw

ith

findin

gs

Involv

ing

eval

uat

ion

stak

ehold

ers

in

know

ing

and

under

stan

din

gth

e

eval

uat

ion

findin

gs

4T

hre

est

udie

sem

phas

ized

that

involv

emen

tre

late

dto

the

findin

gs

was

import

ant

toev

a-

luat

ion

use

Cousi

ns

(1995)

Gre

ene

(1987)

Laf

leur

(1995)

Shea

(1991)

(conti

nued

)


386



Involv

emen

tw

ith

rele

van

ce

Sta

keh

old

erpar

tici

pat

ion

toin

tegra

te

import

ant

org

aniz

atio

nal

conce

rns

into

the

eval

uat

ion

des

ign

4F

or

the

most

par

t,in

crea

sed

con-

tact

wit

hst

akeh

old

ers

fost

ered

incr

ease

dre

levan

ceth

at

resu

lted

inin

crea

sed

eval

ua-

tion

use

Cousi

ns

(1995)

Gre

ene

(1987)

Laf

leur

(1995)

Shea

(1991)

Involv

emen

tw

ith

per

sonal

char

acte

rist

ics

Involv

emen

tof

eval

uat

ion

stak

e-

hold

ers

atdif

fere

nt

org

aniz

atio

nal

level

s

2F

indin

gs

of

one

study

sugges

tth

at

involv

emen

tof

man

ager

s

affe

cts

use

more

exte

nsi

vel

y

than

involv

ing

oth

erst

aff

Cousi

ns

(1995)

McC

orm

ick

(1997)

Involv

emen

tw

ith

dec

isio

n

char

acte

rist

ics

Involv

ing

of

ara

nge

of

stak

ehold

ers

indif

fere

nt

sett

ings

dep

endin

gon

the

char

acte

rist

ics

of

the

dec

isio

n

that

nee

ds

tobe

mad

e

1T

his

study

found

aposi

tive

rela

-

tionsh

ipbet

wee

nev

aluat

ion

use

and

involv

emen

tby

indi-

vid

ual

sin

nontr

adit

ional

bure

aucr

acie

sw

her

edec

isio

n

mak

ing

involv

esin

put

from

peo

ple

atal

lle

vel

sin

the

org

aniz

atio

n

Johnso

n(1

993)

Involv

emen

tw

ith

info

rmat

ion

nee

ds

The

involv

emen

tof

stak

ehold

ers

faci

lita

ted

the

intr

oduct

ion

of

thei

r

info

rmat

ion

nee

ds

1In

volv

edst

akeh

old

ers’

des

ire

for

info

rmat

ion

and

the

tim

elin

ess

of

the

eval

uat

ion

fost

ered

info

rmat

ion

ow

ner

ship

,w

hic

h

was

posi

tivel

yre

late

dto

use

Rock

wel

let

al.

(1990)

aE

val

uat

or

com

pet

ence

was

not

aca

tegory

inth

eC

ousi

ns

and

Lei

thw

ood

fram

ework

,but

auth

ors

pro

pose

itas

anew

char

acte

rist

icin

the

eval

uat

ion

imple

men

tati

on

cate

gory

.


387



dissertations) broadened its scope. This literature review located 41 empirical studies of

evaluation use conducted between 1986 and 2005 that met minimum quality standards.

Most of the studies (38 of 41) examined the use of findings rather than process use; only

three studies examined process use. The lack of attention to process use in the articles included

in this review might have resulted from the fact that the concept of explicit process use is fairly

recent, and the field is still more focused on outcomes and results. Alternatively, it might be

that empirical studies are more likely to focus on the use of results because measuring process

use is less well defined. Finally, the limited attention to process use might have resulted from

our search strategy, which excluded evaluation capacity building studies, many of which mea-

sured organizational learning through the evaluation process. These studies are not included in

this review but are synthesized in a publication by Cousins et al. (2004). After the findings

were categorized according to the Cousins and Leithwood framework, one additional category

(stakeholder involvement) and one new characteristic (evaluator competence) emerged. These

additions align with the comments of Shulha and Cousins (1997) made more than 10 years ago

about changes in the field, especially the diversification of the evaluator’s role.

The stakeholder involvement category reflects the expansion of participatory evaluation

methods. The framework of Cousins and Leithwood included stakeholder involvement under

the ‘‘commitment and/or receptiveness to evaluation’’ characteristic within the decision- and

policy-setting category. This was sufficient in the mid-1980s because only 10% of the studiesin their review included involvement, and these were all related to the effects of involvement

on stakeholders’ commitment or receptiveness to evaluation. In addition, four of the studies in

the current review directly examined the relationship between stakeholder involvement and

evaluation. This dynamic was not present in any of the studies examined by Cousins and Leith-

wood. The emergence of this new category suggests that evaluators may want to focus on

involving stakeholders as a way to enhance evaluation use. The addition of the evaluator com-

petence characteristic indicates a growing acknowledgment of the importance of the compe-

tence of individual evaluators, both professionally and culturally—and the value of these

characteristics in efforts to increase evaluation use.

Some studies—Shea (1991), Bober and Bartlett (2004), Boyer and Langbein (1991), and

Malen, Murphy, and Geary (1988)—examined multiple characteristics. It seemed possible that

these studies might help us think about evaluation influence by identifying important variables

in a sequence suggestive of a pathway, at least at the individual level. This effort failed because

the studies examined variables related to use, not pathways leading to it. Identifying pathways

was a creative activity rather than a way to summarize the research. As Weiss et al. (2005)

found when they sought influence pathways after the fact in their drug abuse resistance edu-

cation (DARE) study, ‘‘We became bogged down in unique tangles of strings [of pathways]

. . . . We are on less sure ground trying to reconstruct individual and interpersonal processesthat were reported to us some 2 to 8 years after the events.’’ In other words, the existing empiri-

cal research on evaluation use has identified a collection of important variables, but research

on influence pathways will necessitate a different strategy. In settings that have specific out-

come variables and sufficient interval data on other variables, path analysis might be one

potential method. Future research might focus on developing quantitative outcome and process

measures that could then be used to gather enough data to conduct path analyses and determine

models displaying the relationships among the process measures and the outcomes.

It is impossible, finally, to answer the question of which characteristics are most related to

increasing the use of evaluations in a straightforward manner. A meta-analysis of the studies is not

possible because the studies do not operationalize or measure the variables in the same manner.

Cousins and Leithwood compensated for this problem by creating a quantitative index that

weighed the number of positive, negative, and nonsignificant findings for each characteristic to

create a ‘‘prevalence of relationship’’ index. Based on this index, they concluded that evaluation




quality and decision characteristics were most highly related to use, followed by evaluation find-

ings, users’ commitment or receptiveness to evaluation, and evaluation relevance.

This index provides a means of comparing findings across a variety of studies. Unfortu-

nately, drawing conclusions about which characteristics are related to use remains problematic

because this type of meta-synthesis is highly affected by the components that researchers

chose to include, and it may not include what is actually occurring. In addition, the publication

process may exclude studies with inconclusive or negative findings. Instead, the current study

discusses those elements that appear to be most ‘‘empirically supported’’—meaning those ele-

ments that are both highly studied and supported by strong evidence of a positive relationship

to evaluation use. Reframing the conversation to discuss ‘‘empirically supported’’ character-

istics also allows the suggestion of evidence-based practices that evaluators can employ to

increase the use of their evaluations.

Framed with these cautions in mind, we identified the following empirically supported fac-

tors that promote the use of evaluation. Findings highlight the importance of stakeholder invol-

vement in facilitating evaluation use. In several studies, involvement was found to facilitate an

evaluation process that, in turn, improved the evaluation implementation characteristics. In

other studies, stakeholder involvement supported decision making or policy setting that fos-

tered greater capacity for using evaluation information. Stated differently, stakeholder invol-

vement is a mechanism that facilitates those aspects of an evaluation’s process or setting that

lead to greater use. More than just involvement by stakeholders or decision makers alone, how-

ever, the findings from this literature review suggest that engagement, interaction, and com-

munication between evaluation clients and evaluators is key to maximizing the use of the

evaluation in the long run.

Limitations

Features of the research method used in this study, particularly the choice to limit the

review to empirical studies of evaluation use conducted between 1986 and 2005, precluded

consideration of any theoretical articles on evaluation produced during that time period. This

fact is not intended to detract from the positive contributions to the understanding of evalua-

tion use made by the authors of these articles. In addition, the research design included a deci-

sion to limit the search terms to ‘‘evaluation utilization,’’ ‘‘evaluation use,’’ and ‘‘evaluation

influence.’’ This decision resulted in the exclusion of ‘‘evaluation capacity building’’ studies

that examined organizational learning through the evaluation process—one form of use—but

did not include the keywords ‘‘use’’ or ‘‘utilization.’’ Finally, the sample sizes of some of the

studies included in this review are rather small. Of the 41 studies included in the review,

approximately 65% (19 of 41) have sample sizes of 12 or fewer. The remaining studies rangedin sample size from 26 to 540.

Conclusion

In summary, the findings from this literature review support Cousins’ (2003) conceptual

framework that outlines dimensions of ‘‘evaluation context’’ (similar to evaluation implemen-

tation characteristics) and ‘‘decision/policy setting.’’ Additionally, the findings support the addi-

tion of one new category—stakeholder involvement—and one new characteristic—evaluator

competence (under the category of evaluation implementation). Findings point to the importance

of stakeholder involvement in facilitating evaluation use and suggest that engagement, interac-

tion, and communication between evaluation clients and evaluators is critical to the meaningful

use of evaluations.




Ap

pen

dix

Su

mm

ary

of

Em

pir

ica

lS

tud

ies

of

Ev

alu

ati

on

Use

an

dIn

flu

ence

(19

86

–2

00

5)

Stu

dy

Type

of

Use

Focu

sof

Stu

dy

Sam

ple

Cat

egory

of

Use

Key

Fin

din

gs

Fin

din

gs

Use

ProcessUse

Instrument

Conceptual

Symbolic

Alt

schuld

etal

.

(1993)

pp

Rel

atio

nsh

ipbet

wee

nat

titu

des

tow

ard

nee

ds

asse

ssm

ent,

invol-

vem

ent

inpro

cess

,bac

kgro

und

char

acte

rist

ics,

and

report

ing

char

acte

rist

ics

and

the

conce

ptu

al

and

inst

rum

enta

luti

liza

tion

of

nee

ds

asse

ssm

ent

concl

usi

ons

Hig

her

educa

tion

adm

inis

trat

ors

(n¼

62)

Dec

isio

nor

poli

cyse

ttin

g;

sta-

keh

old

erin

volv

emen

t

Use

of

nee

ds

asse

ssm

ents

wer

e

infl

uen

ced

by

coll

ege

adm

inis

-

trat

ors

’at

titu

des

and

level

sof

involv

emen

t.T

he

adm

inis

trat

ors

’

bac

kgro

und/t

rain

ing

and

char

ac-

teri

stic

sof

the

nee

ds

asse

ssm

ent

report

sw

ere

not

found

tobe

rela

ted

touse

Ayer

s(1

987)

pp

Rel

atio

nsh

ipbet

wee

nuse

of

a

‘‘st

akeh

old

erco

llab

ora

tive’

’ev

a-

luat

ion

appro

ach

and

inst

rum

en-

tal

and

conce

ptu

aluse

Guam

publi

csc

hool

dis

tric

t(n¼

1)

Dec

isio

nor

poli

cyse

ttin

g;

sta-

keh

old

erin

volv

emen

t

Ayer

sin

terv

iew

edfo

ur

of

the

sta-

keh

old

ers

who

par

tici

pat

edin

all

phas

esof

the

eval

uat

ion,

asw

ell

astw

om

ajor

use

rsof

the

eva-

luat

ion,

toso

lici

tper

cepti

ons

of

the

pro

cess

and

of

subse

quen

t

use

.P

arti

cipan

tsre

port

edposi

-

tive

atti

tudes

tow

ard

the

pro

cess

,

but

dir

ect

use

of

the

report

was

low

.H

ow

ever

,al

though

use

,as

mea

sure

dby

imple

men

tati

on

of

reco

mm

endat

ions,

was

low

,th

e

findin

gs

trig

ger

edpla

nnin

gdis

-

cuss

ions

and

neg

oti

atio

ns

bet

wee

nunio

nan

dag

ency

adm

inis

trat

ion

(conti

nued

)


390



Bam

ber

ger

(2004)

pp

Char

acte

rist

ics

of

hig

hly

cost

-

effe

ctiv

eev

aluat

ions

of

inte

rna-

tional

dev

elopm

ent

pro

ject

s

Dev

elopm

ent

pro

ject

eval

uat

ions

(n¼

8)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g;

stak

ehold

erin

volv

emen

t

Iden

tifi

edfi

ve

fact

ors

that

incr

ease

d

the

impac

tof

anev

aluat

ion:

(a)

a

conduci

ve

poli

cyen

vir

onm

ent—

eval

uat

ion

addre

sses

curr

ent

conce

rns

and

ther

eis

aco

mm

it-

men

tby

dec

isio

nm

aker

sto

use

resu

lts;

(b)

tim

ing

of

eval

uat

ion—

eval

uat

ion

launch

edw

hen

ther

e

are

clea

rly

def

ined

info

rmat

ion

nee

ds;

(c)

role

of

eval

uat

ion—

eval

uat

or

mu

stun

ders

tand

eva-

luat

ion

ison

eso

urc

eof

data

wit

hin

adeci

sion

-mak

ing

con-

text;

(d)

bu

ildin

ga

rela

tion

ship

wit

hth

ecl

ient

and

effe

ctiv

ely

com

mun

icat

ing

find

ings;

and

(e)

eval

uat

ion

cond

ucte

dby

eith

er

the

evalu

atio

nun

itof

the

man-

agin

gor

fund

ing

agen

cyor

by

ou

tsid

eag

ency,

or

join

tly,

asth

e

conte

xt

dic

tate

s

Bar

rios

(1986)

ppp

Rel

atio

nsh

ipbet

wee

nte

chnic

alan

d

org

aniz

atio

nal

var

iable

san

d

inst

rum

enta

l,co

nce

ptu

al,

and

per

suas

ive

use

of

eval

uat

ion

info

rmat

ion

Sta

te-l

evel

soci

alse

rvic

eag

ency

(n¼

1)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g;

stak

ehold

erin

volv

emen

t

Rec

om

men

dat

ions

requir

ing

poli

cy

chan

ges

or

inte

rpro

gra

mor

inte

ragen

cyac

tion

wer

em

ore

infl

uen

tial

inte

rms

of

the

dec

i-

sions

toim

ple

men

tth

emin

com

par

ison

wit

hre

com

men

da-

tions

that

sugges

ted

only

acti

on

by

pro

gra

mm

anag

ers.

The

fol-

low

ing

var

iable

sar

eal

sore

late

d

touti

liza

tion:use

rin

volv

emen

tin

the

form

ula

tion

of

the

study

and

eval

uat

or

cred

ibil

ity

inte

rms

of

pro

gra

mknow

ledge

(conti

nued

)


391



Bober

and

Bar

tlet

t

(2004)

pp

Eval

uat

ion

imple

men

tati

on

fact

ors

and

dec

isio

nan

dpoli

cyse

ttin

g

fact

ors

affe

ctin

gth

euse

of

trai

n-

ing

eval

uat

ion

resu

lts

atco

rpora

te

univ

ersi

ties

Corp

ora

teuniv

ersi

ties

(n¼

4)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g

Corp

ora

teuniv

ersi

tym

anag

ers

use

d

eval

uat

ion

findin

gs

ina

var

iety

of

way

sw

ith

inst

rum

enta

luse

s

dom

inat

ing.

Eval

uat

ion

imple

-

men

tati

on

fact

ors

wer

em

ore

import

ant

than

dec

isio

n-

or

poli

cy-s

etti

ng

fact

ors

inim

pac

t-

ing

use

.T

he

most

hig

hly

ranked

fact

or

was

com

munic

atio

nqual

-

ity.

Use

of

mult

iple

met

hods

of

report

ing

dat

aw

asef

fect

ive

for

incr

easi

ng

use

Boyer

and

Lan

gbei

n

(1991)

pF

acto

rsre

late

dto

the

use

of

hea

lth-

rela

ted

eval

uat

ion

rese

arch

resu

lts

by

mem

ber

sof

congre

ss

and

congre

ssio

nal

staf

fers

Congre

ssio

nal

hea

lth

and

hea

lth-

rela

ted

staf

fm

ember

s(n¼

100)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g;

eval

uat

or

com

pet

ence

Congre

ssio

nal

mem

ber

san

dst

affe

rs

bel

ieved

eval

uat

ion

report

sto

be

rele

van

t,ti

mel

y,

clea

r,m

ethodo-

logic

ally

rigoro

us,

and

pro

duce

d

by

reputa

ble

pra

ctit

ioner

s.T

he

rela

tive

import

ance

of

fact

ors

affe

ctin

guse

var

ied

dep

endin

gon

what

type

of

report

(Gen

eral

Acc

ounti

ng

Off

ice

[GA

O]

vs.

non-G

AO

)an

duse

r(m

ember

of

congre

ssvs.

staf

fer)

.O

ver

all,

tim

elin

ess

of

GA

Ore

port

sw

asth

e

stro

nges

tfa

ctor,

wit

hcr

edib

ilit

y

of

met

hodolo

gy,

and

clar

ity

of

report

ing

also

bei

ng

import

ant.

Pre

sence

of

anad

voca

teor

abse

nce

of

adet

ract

or

of

the

eva-

luat

or

also

pla

yed

aro

lein

use

Bro

wn-

McG

ow

an

(1992)

pE

ffec

tof

know

ledge

use

syst

em

(KU

S)

on

use

of

eval

uat

ion

find-

ings;

rela

tionsh

ipbet

wee

nev

a-

luat

ion

pro

cess

,si

gnif

ican

ceof

the

dec

isio

n,

per

ceiv

edim

pac

ts

of

the

dec

isio

n,

and

pre

fere

nce

s

tow

ard

eval

uat

ion

outc

om

esan

d

uti

liza

tion

of

eval

uat

ion

resu

lts

Sen

ior

hig

her

educa

tion

adm

inis

-

trat

ors

(n¼

8)

Dec

isio

nor

poli

cyse

ttin

g;

stak

ehold

erin

volv

emen

t

Dec

isio

nm

aker

sre

port

edso

me

incr

ease

inth

eir

par

tici

pat

ion

and

inte

rest

inth

eev

aluat

ion

pro

cess

bec

ause

of

usi

ng

the

KU

S.

The

uti

liza

tion

of

eval

uat

ion

findin

gs

was

impro

ved

.T

he

eval

uat

ion

qual

ity

and

uti

liza

tion

of

resu

lts

wer

eal

soen

han

ced

by

dec

isio

n

mak

ers’

per

sonal

stak

esin

the

eval

uat

ion

(conti

nued

)


392



Cai

(1996)

ppp

Rel

atio

nsh

ipbet

wee

nte

acher

s’

per

cepti

ons

of

thei

rin

volv

emen

t

inpro

gra

mev

aluat

ion

and

report

edle

vel

sof

inst

rum

enta

l,

conce

ptu

al,

and

sym

boli

cuse

New

York

stat

eK

-12

publi

csc

hool

teac

her

s(n¼

207)

Sta

keh

old

erin

volv

emen

tC

urr

ent

opport

unit

yfo

rin

volv

e-

men

tis

rela

ted

tow

illi

ngnes

sto

par

tici

pat

ein

futu

reim

ple

men

ta-

tion.

Lev

elan

dphas

eof

invol-

vem

ent

inev

aluat

ion

isre

late

dto

per

ceiv

edben

efit

sto

indiv

idual

and

toorg

aniz

atio

n.

The

ben

efit

s

of

such

involv

emen

tin

clude:

enhan

ced

uti

liza

tion

and

wil

ling-

nes

sto

be

involv

edin

futu

re

eval

uat

ions,

incr

ease

dknow

ledge

and

skil

lsre

late

dto

eval

uat

ion,

and

impro

ved

com

munic

atio

n

pro

cess

wit

hin

org

aniz

atio

ns

Cal

lahan

etal

.

(1995)

pF

acto

rsan

dpra

ctic

esre

late

dto

eval

uat

ion

uti

liza

tion

ingif

ted

educa

tion

pro

gra

ms;

exam

inat

ion

of

exem

pla

ryan

dnonex

empla

ry

eval

uat

ions

and

exte

nt

of

imple

-

men

tati

on

of

eval

uat

ion

reco

mm

endat

ions

Eval

uat

ion

report

sfr

om

dis

tric

t

gif

ted

educa

tion

pro

gra

ms

(n¼

12)

Dec

isio

nor

poli

cyse

ttin

g;

eval

uat

or

com

pet

ence

;

stak

ehold

erin

volv

emen

t

All

12

dis

tric

tsuse

dev

aluat

ion

info

rmat

ion

toen

act

som

ech

ange

ingif

ted

educa

tion

pro

gra

mm

ing.

The

‘‘w

ill

and

skil

l’’

of

key

per

-

sonnel

toev

aluat

eaf

fect

edth

euse

of

the

eval

uat

ion

resu

lts.

Key

condit

ions

affe

ctin

guse

:(a

)

dis

tric

t-w

ide

eval

uat

ion

poli

cy;(

b)

wri

tten

pla

ns

on

how

toim

ple

men

t

findin

gs;

(c)

mult

iple

stak

ehold

ers

wer

eco

nsi

sten

tly

involv

edin

pla

nnin

g,m

onit

ori

ng,an

dre

vie

w-

ing

eval

uat

ion

pro

cess

and

find-

ings;

(d)

stak

ehold

ers

pla

yed

role

of

advoca

ting

for

pro

gra

mch

ange

bas

edon

findin

gs;

and

(e)

key

per

sonnel

wer

eaw

are

of

rela

tion-

ship

bet

wee

ngif

ted

ed,e

val

uat

ion,

and

poli

tica

lpro

cess

es

(conti

nued

)


393



Car

pin

ello

(1989)

pp

Exam

ines

the

effe

ctof

the

pow

er

bas

eof

the

eval

uat

or

(leg

itim

ate,

refe

rent,

or

exper

t),

per

cepti

ons

of

dec

isio

n-m

akin

gco

nse

-

quen

ces,

and

eval

uat

ion

use

r

exper

ience

son

eval

uat

ion

use

in

term

sof

agre

emen

tw

ith

reco

m-

men

dat

ions,

per

cepti

ons

of

eva-

luat

ion

cred

ibil

ity,

nee

ds

for

info

rmat

ion,

and

inst

rum

enta

l

dec

isio

ns

Ger

onto

logy

nurs

esfr

om

New

York

(n¼

282)

Dec

isio

nor

poli

cyse

ttin

gC

onse

quen

ce,

pow

er,

and

exper

i-

ence

wer

efo

und

toaf

fect

how

eval

uat

ion

info

rmat

ion

isuse

d

and

pro

cess

edby

nurs

edec

isio

n

mak

ers.

Exper

ience

ddec

isio

n

mak

ers

indic

ated

anee

dfo

r

info

rmat

ion

when

infl

uen

ced

by

econom

icco

nse

quen

ces

and

refe

rent

pow

erbas

es,

wher

eas

less

exper

ience

ddec

isio

nm

aker

s

wer

eaf

fect

edby

affe

ctiv

eco

n-

sequen

ces

and

exper

tpow

er

bas

es

Chin

(2003)

pp

Impac

tof

usi

ng

cart

oons

and

poet

ry

inev

aluat

ion

report

son

pro

mpt

dis

cuss

ion

of

findin

gs

and

incr

ease

dunder

stan

din

gof

resu

lts

by

eval

uat

ion

stak

ehold

ers

Sch

ool

dis

tric

tev

aluat

ion

stak

e-

hold

ers

(n¼

26)

Eval

uat

ion

imple

men

tati

on

Alt

hough

cart

oons

and

poet

ryw

ere

wel

lre

ceiv

edby

eval

uat

ion

sta-

keh

old

ers,

eval

uat

ors

wer

enot

as

support

ive

of

thei

rin

clusi

on.

The

poet

ryan

dca

rtoons

convey

edan

emoti

onal

and/o

rvis

ual

repre

-

senta

tion

of

findin

gs;

how

ever

,

this

did

not

incr

ease

dis

cuss

ion

of

the

findin

gs

among

stak

ehold

ers

nor

did

iten

sure

that

report

read

ers

clea

rly

per

ceiv

edth

e

auth

or’

sin

tended

mes

sages

Com

bs

(1999)

ppp

Rel

atio

nsh

ipbet

wee

npre

exis

ting

posi

tive

atti

tudes

tow

ard

incl

u-

sive

educa

tion

and

the

per

sua-

siven

ess

of

pro

gra

mev

aluat

ion

findin

gs

asm

easu

red

by

Russ

on

and

Koeh

ly(1

995)

per

suas

ion

scal

e

Gen

eral

and

spec

ial

educa

tion

ele-

men

tary

teac

her

sin

Nort

hC

aro-

lin

a(n¼

76)

Dec

isio

nor

poli

cyse

ttin

gA

lthough

the

study

found

that

teac

her

s’at

titu

des

tow

ard

incl

u-

sion

wer

epre

dic

tive

of

the

per

-

suas

iven

ess

of

the

sum

mar

y

eval

uat

ion

report

,co

ncl

usi

ons

from

the

study

are

lim

ited

by

the

pec

uli

arit

ies

of

the

dat

a

(untr

eate

doutl

iers

inth

edat

ase

t)

(conti

nued

)


394



Cousi

ns

(1995)

pp

Exam

inat

ion

of

the

impac

tof

par

ti-

cipat

ory

appro

aches

use

din

one

mar

gin

ally

succ

essf

ul

and

one

hig

hly

succ

essf

ul

educa

tional

eval

uat

ion

Can

adia

ned

uca

tion

fiel

dce

nte

rs

(n¼

2)

Eval

uat

ion

imple

men

tati

on;

stak

ehold

erin

volv

emen

t

The

par

tici

pat

ory

pro

cess

enhan

ced

cred

ibil

ity

of

the

report

and

mad

e

the

findin

gs

more

rele

van

t,w

hic

h

intu

rnin

crea

sed

the

report

ed

use

fuln

ess

of

the

eval

uat

ion

Cousi

ns

(1996)

pE

ffec

tsof

rese

arch

erin

volv

emen

t

level

son

exte

nt

and

type

of

rec-

om

men

dat

ion

imple

men

tati

on

Can

adia

nsc

hool

dis

tric

ts(n¼

3)

Eval

uat

or

com

pet

ence

Des

pit

eth

evar

yin

gle

vel

sof

rese

arch

erin

volv

emen

t,docu

-

men

ted

use

was

rela

tivel

yst

able

.

Use

appea

red

tobe

more

affe

cted

by

tim

epre

ssure

san

dad

min

is-

trat

ive

support

than

by

level

of

rese

arch

erin

volv

emen

t.In

the

low

est

involv

emen

tca

se,

pote

n-

tial

for

use

was

hig

her

than

actu

al

use

,giv

enth

eti

mef

ram

eof

the

eval

uat

ion

Cro

tti

(1993)ppp

Use

of

pro

cess

and

end

pro

duct

sof

Pen

nsy

lvan

ia’s

long-r

ange

pla

ns,

asper

ceiv

edby

school

adm

inis

-

trat

ors

;re

lati

onsh

ipbet

wee

n

hum

anan

dco

nte

xt

char

acte

ris-

tics

and

per

cepti

ons

of

use

fuln

ess

Pen

nsy

lvan

iasc

hool

dis

tric

ts

(n¼

11)

Dec

isio

nor

poli

cyse

ttin

gD

iffe

rent

adm

inis

trat

ive

level

s

emphas

ized

dif

fere

nt

form

sof

eval

uat

ion

use

.L

oca

lco

nst

rain

ts

had

min

imal

infl

uen

ceon

the

eval

uat

ion

uti

liza

tion

pro

cess

.

The

acti

ve

org

aniz

ing

effo

rts

of

school

adm

inis

trat

ors

report

edly

pro

mote

dlo

ng-r

ange

pla

nuti

li-

zati

on.

Fac

tor

clust

ers

com

pri

s-

ing

hum

anan

dev

aluat

ion

var

iable

sre

ceiv

edhig

her

over

all

import

ance

than

conte

xt

var

iable

s

(conti

nued

)


395



Ear

l(1

995)

pp

Exam

inat

ion

of

the

impac

tof

two

par

tici

pat

ory

eval

uat

ions

on

the

incr

ease

dunder

stan

din

g,

com

-

mit

men

t,an

duti

liza

tion

of

the

eval

uat

ion

by

Documents

Volume 30 Number 3 Research on Evaluation Use 2009 ......Alkin (2003). Finally, the team conducted a manual review (looking for relevant research based on titles) of the following