Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
1
From Research to Real World Settings:
Factors Influencing the Successful Replication of Model Programs
Sharon Mihalic University of Colorado, Boulder
Center for the Study and Prevention of Violence Institute of Behavioral Science
UCB 442 Boulder, CO 80309
(303) 492-2137 (303) 449-8479 (fax)
Katherine Irwin
University of Hawaii, Manoa Department of Sociology
Saunders Hall 247 Honolulu, HI 96822
(808) 956-7693 (808) 956-3707 (fax)
Sharon Mihalic works at the Center for the Study and Prevention of Violence as the Director of the Blueprints for Violence Prevention Project. Katherine Irwin, is an Assistant Professor in the Department of Sociology at the University of Hawaii at Manoa.
ABSTRACT
2
As science-based programs become more readily available to practitioners, the
need for identifying and overcoming problems associated with the process of
implementation becomes critical. A major goal of the Blueprints for Violence Prevention
initiative has been to enhance the understanding of program implementation by studying
the influence of human and systems-level factors that challenge the successful
implementation of programs. This paper describes the results of a process evaluation
focused on discovering common implementation obstacles faced by 42 sites
implementing eight of the Blueprints programs. This evaluation revealed that most sites
involved in the project faced many challenges when implementing in “real-world”
settings. Using regression analyses to identify the most important of these factors,
findings revealed that the quality of technical assistance, ideal program characteristics,
consistent staffing, and community support were important influences on one or more
measures of implementation success.
3
INTRODUCTION
Many of us probably remember the age-old saying “it is not what you do, but how
well you do it that counts.” Usually expressed by the older and wiser members of our
families or communities, this statement tended to be a gentle way of telling us that we
have emphasized the outcome of our efforts over the process. The inherent wisdom was
that process and product are intimately connected. One should not be emphasized over
the other.
This lesson seems particularly poignant given the developments in criminal
justice programs in the late twentieth and early twenty-first centuries. Emerging from a
pessimistic “nothing works” perspective in criminal justice (Martinson, 1974; Regnery,
1985; Sechrest et al, 1979), the field of evaluation research flourished in the 1990s and
early 2000s and seemed to give birth to a never-ending hunger to find programs
demonstrating “positive outcomes.” The 1970s pessimism developed when few of the
expensive criminal justice innovations at the time significantly reduced crime and
delinquency (see Petersilia, 1990). By the late 1990s, however, several crime and
delinquency prevention and treatment programs boasted that they produced significant
results. Once evaluation results proclaimed that some programs “worked” (Botvin, 1990;
Catalano et al., 1998; Davis and Tolan, 1993; Dryfoos, 1990; Elliott, 1997; Gendreau and
Ross, 1980; Greenberg et al., 1999; Hawkins, Catalano, and Brewer, 1995; Hawkins,
Farrington, & Catalano, 1998;Lipsey, 1992; Lipsey and Wilson, 1997; Mendel, 2000;
Sherman et al. 1997; Tobler, 1992; Tolan & Guerra, 1994; Van Voorhis, 1987;
Wasserman and Miller, 1998), federal, state, and private organizations supported the
efforts to conduct outcome evaluations of a variety of criminal justice innovations and
4
promoted the adoption of effective programs. By the late 1990s the emphasis on
“program outcomes” became so diffuse that many practitioners found they could not
secure funding unless they conducted or submitted to outcome evaluation studies.
With the quest to ferret out “what works” from “what doesn’t” in the treatment
and prevention of crime and delinquency, the program evaluation field has tended to
ignore the intimate relationship between process and product. Funders and evaluators
have carefully focused upon “outcomes” and, as a result, have produced numerous
research findings outlining the programs that demonstrate significant positive results.
What tends to be missing in this literature is a discussion of “the process” of
implementing programs (Fagan, 1990; Greenberg et al., 2001). For example, in a review
of over 1,200 published prevention studies, only 5 percent provided data on
implementation (Durlak, 1997). Domitrovich and Greenberg’s (2000) review of 34
rigorously evaluated mental health programs revealed that only 11 studies considered
implementation factors in their analyses. Similarly, Dane and Schneider (1998) found
that only 39 of the 162 preventive interventions they examined contained information on
program integrity, and only 13 of these studies considered the impact of fidelity on
outcomes.1 Because of the emphasis on outcome over process, practitioners are left with
a list of model or best practices programs that produced favorable outcomes during
research trials, but there is very little information available regarding how to implement
these programs – what is often called “transporting” programs to real world settings. In
other words, we now know what to implement, but we know very little about how.
The importance of the process of implementation can not be overstated. In fact, as
the adage suggests, the process of implementation influences the product. For example,
5
meta-analyses demonstrate that programs receiving implementation monitoring (DuBois,
Holloway, Valentine, and Cooper, 2002) or that were implemented “better” produced
more change (Gresham, Cohen, Rosenblum, Gansle, and Noell, 1993; Wilson and
Lipsey, 2000). In his study of juvenile delinquency treatment programs, Lipsey (1999)
found that the most outstanding programs reduced recidivism rates by 40 percent.
Whether programs were thoroughly implemented and how long programs were
implemented significantly affected recidivism rates. Moreover, process evaluation studies
almost consistently show that programs implemented with fidelity to the original design
demonstrate superior outcomes (Fors and Doster, 1985; Gray, Emshoff, Jakes, and
Blakely, draft; Gresham et al., 1993; McGrew, Bond, Dietzen, and Salyers, 1994).
Therefore, it becomes clear that reducing crime, delinquency, and other problem
behaviors requires us to pay close attention to how programs are implemented.
In a response to the importance of implementation and the relative lack of
information available, researchers have begun to outline the “lessons learned” while
implementing programs. Gleaned from case studies, field experiences, and, in a few
cases, randomized studies, these findings outline some of the forces that seem to
influence implementation success and note that characteristics of prevention and
treatment programs, host organizations, staff, leaders, communities, and training and
technical assistance (TA) seem to make a difference. However, these case studies and
“tales from the field” are not able to identify which factors are most significant.
Furthermore, few studies have analyzed the replication of research based programs in
numerous settings. Those that have included large samples (Ellickson and Petersilia,
1983; Gottfredson and Gottfreson, 2002) have tended to focus on the implementation of
6
one type of program (i.e., law enforcement or school based) in one type of setting
(criminal justice systems or schools). What is missing is a systematic analysis of the
factors that can make or break implementation efforts across a variety of contexts and
programs.
This study fills this gap in our knowledge by looking at 8 programs implemented
in 42 sites across the United States. The 8 programs represented a range of modalities
including prenatal and postpartum, school based, mentoring, family therapy, and foster
care interventions. Each of the eight programs were considered “model programs” in that
they were all proven to significantly reduce violence and other problem behaviors during
research trials. In addition, the agencies responsible for implementing these programs
included schools (elementary, middle, and high), state supported health agencies, private
treatment organizations, and non-profit community agencies. Combining quantitative
and qualitative data collection techniques, researchers conducted a “process evaluation”
study with each site as they adopted one of 8 model programs and implemented it for two
years. During the two years of the study, researchers assessed how successfully the
programs were implemented and began to identify the common factors contributing to or
detracting from implementation success. At the end of the study, researchers tested their
hypotheses gleaned during the qualitative portion of the study as well as those from the
implementation literature to test the factors that seemed to significantly contribute to
implementation success.
Conceptual Framework for the Study of Implementation
Diffusion theory provides a conceptual foundation for examining dissemination,
adoption, implementation, and maintenance of innovations (Rogers, 1983; 1995).
7
Diffusion is defined as the process by which an innovation is communicated through
channels over time among members of a social system. An innovation is an idea,
practice, service, or other object perceived as new by an individual or other unit of
adoption. Diffusion of innovations occurs in four stages: (1) awareness is gained through
the provision of information (dissemination); (2) adoption, when the innovation is
selected for use, (3) implementation, when the innovation is used, and (4) maintenance,
when the innovation is in place and becomes institutionalized (Rogers, 1983; 1995).
According to Rogers (1995), once knowledge of a new innovation has been
acquired, and motivation and support for it has developed, a decision may then be made
to try the innovation. These mental processes are followed by the implementation stage,
which involves putting the idea into practice. It is at this point in the diffusion process
that overt behavior change is required and problems in how to use the innovation occur.
If all goes well during implementation, there will be some recognition of the benefits of
the innovation, and the integration of the innovation into organization or ongoing routine
will occur.
However, the decision to adopt a program does not always guarantee success in
implementation. In reality, there are a host of obstacles that can undermine the
implementation of effective programs, and these factors are only recently being addressed
in the research literature through implementation studies (Domitrovich and Greenberg,
2000; Greenberg, Domitrovich, Graczyk, and Zins, 2001; Mihalic, Irwin, Elliott, Fagan,
and Hansen, 2001). Implementation research can provide an understanding of the
organizational and human capacities and motivation necessary to successfully adopt,
implement, and sustain programs.
8
Implementation Research
Although primarily anecdotal, the implementation literature notes that replicating
programs is a complex process riddled with a number of challenges and barriers.
Generalizing from the many case studies, there seems to be five key factors that play a
role in how well a program can be implemented in a new setting. These include features
of the organization adopting the new program, staff characteristics, strong leadership,
adequate training and technical assistance (TA), and community support. There is also
some evidence that aspects of the program being adopted and preplanning also play a role
in implementation success.
Organizational features are understood as the characteristics of the organization
implementing the program and are the most commonly documented factors determining
implementation success. Such characteristics as the nature, structure, history, philosophic
traditions, economic standing, and stability of organizations are thought to play a key role
in how well new programs are implemented. Gendreau, Goggin, and Smith (1999) have
argued that organizations need to remain flexible, efficient, and non-conflictual when
solving problems. In addition, the presence of strong administrative support for a
program (Coolbaugh and Hansel, 2000; Dunworth, Mills, Cordner, and Greene, 1999;
Ellickson and Petersilia, 1983; Gager and Elias, 1997; Gendreau, Goggin, and Smith,
1999; Petersilia, 1990), open and clear communication channels within an organization
(Harris and Smith, 1996; Kegler, Steckler, Malek, and McLeroy, 1998), and agency
stability indicated by lack of staff turnover (Gendreau et al., 1999; Gottfredson and
Gottfredson, 2002; Lynch, Geller, Hunt, Galano, and Dubas, 1998; Mears and Kelly,
2002; Petersilia, 1990) play important roles in successfully implementing a program.
9
There is also some evidence that linkages between multiple agencies (i.e., collaboration)
can bolster implementation quality (Ellickson and Petersilia, 1983; Freudenburg, 1998).
Petersilia (1990) alludes to the importance of inter-agency linkages by arguing that
implementation efforts fare better when “larger systems” are receptive to the program.
While the nature and structure of organizations seem important to implementation
success, staff members have also been cited as influential agents during program
replication. Staff who are motivated to implement and supportive of the program,
generally, help successfully replicate programs (Elias and Clabby, 1992; Ellickson and
Petersilia, 1983; Gendreau et al., 1999; Hunter, Elias, and Norris, 2001; McCormick,
Steckler, and McLeroy, 1995; Petersilia, 1990; Taggart, Bush, Zuckerman, and Theiss,
1990). Gendreau et al. (1999, pg. 184) argued that staff need to “participate directly in
designing the new program,” while other researchers have noted that any mechanism to
get “staff involved” in the program will be beneficial. Staff skills have also been cited as
an important variable in success (Kegler et al. 1998; Lynch et al., 1998; Taggart et al.,
1990). Often staff skills are measured in terms of education, experience, or credentials.
There is also some evidence to suggest that adequate financial reimbursement
provided to direct line staff members makes a difference during implementation
(Coolbaugh and Hansel, 2000). This is an especially relevant point for prevention
programs, as many are implemented in schools and use teachers to implement programs
in the classroom. Thus, these programs constitute “add-ons” to teachers’ other duties,
and, more often than not, teachers are not paid for their extra time. While reimbursing
instructors for teaching a prevention program may not be feasible for many schools, and
some may not want to initiate such a practice, there are many other small gestures that
10
can be done to show support and appreciation (e.g., public acknowledgements, thank you
notes, providing extra classroom supplies). Another “add-on” barrier to implementation is
time to implement a program. This is especially true in school settings. In their survey of
380 coordinators of safe and drug free schools programs, King, Wagner, and Hendrick
(2001) discovered that time was one of the most consistently noted barriers to program
implementation (see also, Kramer, Lauman, and Brunsen, 2000).
School-based staff barriers, according to the literature, can be overcome by
integrating programs into schools’ daily operations (Gager and Elias, 1997; Gendreau et
al., 1999; Gottfredson and Gottfredson, 2002; Kramer et al., 2000). When teachers and
administrators recognize that new programs serve a larger purpose for the school, they
are not only more likely to adopt the initiatives, but also feel ownership for them (Fullan,
1992).
According to the literature, successful implementation efforts need strong
leadership and key personnel – often called program champions – to coordinate and
conduct the program (Ellickson and Petersilia, 1983; Farrell et al., 2001; Gager and Elias,
1997; Gottfredson and Gottfredson, 2002; Kegler et al. 1998; Petersilia, 1990). The
program champion is the motivator behind the new innovation, guiding its day-to-day
operations, fostering communication, and serving as a base of support for program
implementers. Projects that have highly committed champions have greater success
(Ellickson and Petersilia, 1983).
Training has frequently been noted as a key resource for success (Connell, Turner,
and Mason, 1985; Fors and Doster, 1985; Gager and Elias, 1997; Gingiss, 1992;
Gottfredson and Gottfredson, 2002; Hunter et al., 2001; Parcel, Ross, Lavin, Portnoy,
11
Nelson, and Winters, 1991; Perry, Murray, and Griffin, 1990; Ross, Luepker, Nelson,
Saavedra, and Hubbard, 1991; Taggart et al., 1990). Practitioners who receive thorough
training are more prepared to implement, more likely to implement, complete a greater
percentage of the program, adhere to the program more closely, and implement more
effectively (Connell et al., 1985; Flay, 1999; Fors and Doster, 1985; McCormick et al.,
1995; Ross et al, 1991; Parcel et al., 1991; Taggart et al., 1990). The most successful
types of training tend to be those that include knowledgeable and enthusiastic trainers and
site administrators (McCormick et al., 1995). On-going formal or informal training,
sometimes called technical assistance, can re-invigorate implementation efforts or
provide necessary on-going support (Durlak, 1998; Gager and Elias, 1997; Gingiss, 1992;
Parcel et al., 1991).
There is some evidence to suggest that the community in which a program is
implemented also influences success. Inter-agency cooperation as well as support from
citizens have been characterized as “community support.” Regarding agency
collaboration, Ellickson and Petersilia (1983, pg. 35) found that “innovations that require
but do not obtain cooperation across different organizations typically attain only low or
moderate levels of effectiveness; those that achieve high levels of external support are
much more likely to realize their full potential.” Barriers to community support are often
turf conflicts between agencies (Coolbaugh and Hansel, 2000; Stoil, Hill, Jansen,
Sambrano, and Winn, 2000) and a lack of citizen understanding or support for programs
(Freudenburg, 1998).
Community support is also important to school-based implementation of
programs. Differing community philosophies regarding the mission of the school can
12
easily undermine program efforts. In fact, in one survey, poor parental attitudes were
listed by Safe and Drug Free School Coordinators as one of the most strongly agreed
upon barriers to improving programs (King et al, 2001; see also Wenter et al., 2002).
Program integration in schools cannot occur unless the culture and attitude of the school
and community are receptive to new programs and innovations (Gendreau et al., 1999;
Kramer et al., 2000). Differences can be overcome when the community is made aware
of the problems that exist and the need for intervention, and are involved in the planning
process (Fullan, 1992). Institutionalization can be enhanced when a program is given
high visibility in the school (Gager and Elias, 1997), suggesting that all school staff,
parents, and the surrounding community should be aware of and involved in school
program efforts.
There is also some evidence to support the idea that the characteristics of the
program, such as complexity and structure, are important (Blakely, Mayer, Gottschalk,
Schmitt, Davidson, Roitman, and Emshoff, 1987; Ellickson and Petersilia, 1983;
Pankratz, Hallfors, and Cho, 2002; Petersilia, 1990). Freudenburg (1998) suggests that
adolescent substance abuse programs that have the potential for successful
implementation will be theoretically-driven models that have proved to be effective.
Additionally, they should be tailored to fit and assess specific needs of clients, link
adolescents to larger resources, and use strengths-based models and aftercare. In school
contexts, such aspects of programs as the standardization of materials (Gottfredson and
Gottfredson, 2002), and materials that are tailored to fit students’ needs (Lytle et al.
1994) are associated with success. Manuals or a set curriculum with designed activities
that are viewed as relevant, attractive, and easy to use enhance program adoption in the
13
classroom (Perry et al., 1990; Taggart et al., 1990). Also, programs that are flexible are
said to be superior during implementation efforts (Colorado Healthy Communities
Initiative, 1998).
Preplanning is believed to be beneficial to the implementation process, but there
is little information about this. The information that is available suggests that
interventions that are adopted based upon an empirically documented need (Gendreau et
al., 1999; Petersilia, 1990), after a more exhaustive information search (Gottfredson and
Gottfredson, 2002), and with activities that are initiated within the organization rather
than by forces external to it (Ellickson and Petersilia, 1983), are implemented with higher
quality. Adoption decisions are most successful when there is sincere motivation at
initiation, early top leadership support, and project director commitment (Ellickson and
Petersilia, 1983). However, successful adoption does not always lead to successful
implementation. Decisions made by community leaders are not always supported by
front-line staff charged with delivering the service. When motivation and support are
lacking, projects are sure to fail. Therefore, it is advisable that all stakeholders agree on
the need for change and the relevance of the intervention for the school or agency
(Fullan, 1992). The best method for ensuring consensus is to include all key players in
planning for program adoption (Diebold, Miller, Gensheimer, Mondschein, and Ohmart,
2000).
Combining the findings in this growing body of implementation literature, we are
left with the impression that there are many factors that influence implementation quality.
In fact, almost everything seems to matter, according to this literature. It is not easy to
discern from these findings the most important of these factors. Taking the sheer
14
frequency that particular factors have been mentioned, it would seem that organization
and staff characteristics, training, and community support are the most important features
of implementation. Program characteristics and preplanning, conversely, have not
received as much attention in the literature. Given that this literature is mostly comprised
of case study approaches, it is important to acknowledge that the findings from random
samples may provide a more reliable foundation for understanding the factors that are
most important.
The factors contributing to implementation success emerging most frequently in
the random sample or population studies that we examined are organization
characteristics (Gottfredson and Gottfredson, 2002; Gager and Elias, 1997), staff
characteristics (McCormick et al., 1995; Wenter et al. 2002), and training and/or TA
(Gottfredson and Gottfredson, 2002; Gager and Elias, 1997; McCormick et al., 1995).
Community support (Wenter, et al. 2002) and program characteristics (Gottfredson and
Gottfredson, 2002) also emerged as significant. Two multivariate regression studies
(McCormick et al., 1995;Wenter et al., 2002) have been conducted to assess which
among multiple factors have direct and significant effects. Wenter et al. (2002) found that
the most significant factors influencing substance abuse prevention practice in schools
are type of school (private vs. public, urban vs. rural), type of staff (primary prevention),
adequate resources, and school board and parent support. McCormick et al. (1995) found
training and awareness of curricula to be the most consistent correlates across multiple
measures of implementation success. Organizational size and climate also had significant
effects. Although, as noted, there were several correlates of implementation, these factors
failed to predict implementation in regression analyses. While providing more credible
15
results than case studies or random sample studies that use only bivariate statistical
analyses, these two studies do not test for the gamut of factors influencing replication
efforts. Therefore, our understanding of the factors affecting implementation success
among all those mentioned in the literature remains extremely limited to date.
METHODS
Study Design
These data come from the Center for the Study and Prevention of Violence (CSPV)
Blueprints for Violence Prevention initiative. The project, initially beginning in Colorado
and then expanding throughout the United States, was designed to reach two broad goals:
identify effective, research-based violence prevention programs (see Table 1 for a list of
these programs) and replicate these programs nationally (i.e., implement these programs
in non-research settings). The findings presented in this paper come from a replication
project in which the Office of Juvenile Justice and Delinquency Prevention (OJJDP),
through CSPV, funded two years of training and technical assistance to 42 sites interested
in implementing one of eight Blueprints for Violence Prevention model programs.2
During the two years of the Blueprints training and technical assistance funding, CSPV
conducted a “process evaluation” of the replication. Process evaluations assess the
delivery of a program by describing and documenting how well the program is being
implemented, or the “integrity” or “fidelity” of the implementation in comparison with
the program’s stated intent.
The primary goals in conducting the process evaluation were to discover: 1)
whether the program was reaching the intended target population (i.e., the appropriate
population, the number of recipients reached and their characteristics, e.g., age group,
16
gender, socioeconomic status etc.); 2) whether the program was being implemented as
designed (i.e., whether the program was delivering the services it intended to deliver in
the way it was designed to deliver them, in terms of number, frequency, and quality of
program components); 3) what were the obstacles and barriers that prevented a complete
implementation; and 4) whether the program was perceived by the implementers as
worthwhile. By examining these aspects of implementation, we were able to assess, both
qualitatively and quantitatively, some of the factors that contribute to implementation
success.
Site Selection
The primary intent of the replication portion of the Blueprints initiative was not to
study the process of implementing research programs in “real world” settings, but to
encourage and assist a national dissemination of highly effective violence prevention
programs. The chance to study and contribute to our knowledge of implementation was a
secondary goal. Therefore, site selection decisions were made to ensure that sites would
successfully implement the programs for at least two years and preferably longer. To
meet this goal, CSPV conducted thorough “site assessments” before granting funds to
each site. Site assessments, designed to determine sites’ “readiness” to implement
programs, included collecting information about the organizational capacity, funding
stability, commitment, and resources available at each site. In addition, CSPV
encouraged sites to demonstrate that the program fit the needs of the community or
school by conducting a needs assessment. Priority was given to sites that could
demonstrate that they were “ready” to implement.
17
Given this, the implementation study, in general, was conducted with a very select
group of sites that may or may not represent “real world” implementation contexts. In
addition, CSPV closely monitored sites during the two-year grant to help them progress
towards full implementation, thus, further mitigating the representativeness of the study.
Data Collection
After selecting the 42 sites for inclusion in the Blueprints replication grant (i.e.,
the grant to fund training and technical assistance for two years), CSPV conducted site
visits three times a year for two years with each site (typically 5 visits were on-site and
one was conducted by phone). Unless a site needed immediate guidance (i.e., they were
experiencing problems that threatened the implementation effort), CSPV staff visited or
telephoned the sites every four months, thus allowing comparability of implementation
progress across the sites. During visits, CSPV administered questionnaires and conducted
informal interviews with staff members to assess how the implementation of these
research-based programs was faring. The questionnaires combined quantitative questions
and open-ended questions. In addition, CSPV staff summarized their observations after
conducting a site visit.
For the quantitative portion of questionnaires, CSPV staff met with staff members
at the site to rate the progress of the implementation in terms of serving the appropriate
population, administering all of the program components, securing adequate funding and
resources, hiring and training all staff, making necessary key contacts and interagency
linkages, and achieving the proper treatment or “dose” (the frequency and intensity with
which the program was being provided). We created these questions by examining the
prescribed program delivery in the Blueprints books (written by the designers of each
18
program and CSPV to describe each program in-depth) and in efficacy trials of each
program. Program designers were also consulted. If the sites implemented all of the
components of the program at the dose that was prescribed, then the sites received an
“achieved” rating for each of these measures. In addition, CSPV counted the number of
components of each program that were achieved or not achieved, thus allowing us to
assess the percentage of core components achieved. Similarly, sites that complied with
funding, target population, staffing, and training criteria also received “achieved” ratings
for each measure.
Surveys administered during the course of the study also included questions
regarding the challenges individuals confronted during implementation and sites’
satisfaction with the program, training, and technical assistance (TA). We also contacted
sites six months after the end of the grant to discover whether the program had been
sustained.
The open ended questions asked site members to discuss the problems that arose
during implementation and how they were handled. Through the administration of these
open ended questions, informal conversations, and on-site observations, CSPV staff
developed a more subjective understanding of the barriers and obstacles commonly
confronted when research based programs are implemented in naturalistic settings.
Measures
The measures of implementation factors (independent variables) and success
outcomes (dependent variables) were derived from the literature as well as from the
hypotheses emerging during the qualitative portion of the study. When compiling the
independent variables for analyses, we reviewed all of the possible factors that have been
19
said to influence or seemed to play a role in implementation success including the
characteristics of programs, communities, training and TA, agencies, staff, and leaders.
To assess the role of program, community, staff, leadership, and agency, we
developed a set of scales designed to include many of the features of each concept that
seemed to affect implementation success. For example, when assessing program
characteristics, we asked site coordinators at the end of the two-year period to rate, on a
five point scale, how much of an asset (score of 5) or barrier (score of 1) the quality of
the materials, the flexibility, the time required, complexity, and cost of the program were
during implementation. This ideal program characteristics scale had a reliability of .69,
based upon Cronbach’s alpha. The ideal agency characteristics scale (reliability of .84)
included staff participation, administrative support, communication, fit between program
and agency, cohesion and collaboration, clarity of goals, clear lines of authority,
structural stability, champion, facilities, financial support, resources for program, and
political climate. We also asked sites to rank the function of program champions (ideal
champion characteristics scale, reliability of .81) in regards to the program champions’
support of the program, motivation, skill and knowledge, and time to devote to the
project. The ideal staff characteristics scale (reliability of .88) included staff members’
buy-in/support for a program, motivation, skill and knowledge, prioritization of program,
communication with others as well as the hiring pool of available staff. The community
support scale (reliability of .79) included the agency’s communication with and support
for partnering agencies, support of community leaders, and support of parents.
Training and TA were measured in two ways. First, for number of TA visits, we
created a ratio of the number of training sessions and TA visits received by sites by the
20
number of months of implementation. This allowed us to keep the failed sites in our
analysis. Four sites had stopped implementing the program before the end of the two-year
study. Second, we also assessed the quality of TA visits by asking sites to rank the overall
quality of the training and TA that they had received. We assessed this at the end of the
two years. For the failed sites, this score was taken during their last site-visit.
We assessed the failure of sites to hire and retain staff (inconsistent staffing) and
secure funding (inconsistent funding) similarly. These variables are comprised of ratios
of the number of times sites had fallen out of compliance with hiring staff and securing
funding by the number of months the program had been implemented, thus, again,
allowing us to include the four failed sites in our analysis.
Time to implement the program emerged as a consistent theme in our qualitative
findings. Therefore, we felt that it was necessary to determine how powerful time, as an
independent variable, was to success. Although it was originally part of the staff
composite measure in which sites were asked how much of a barrier or an asset time was
to implement the program, we excluded it from the composite staff variable and analyzed
it separately.
Measures of the dependent variables – implementation success – demanded
careful consideration. For example, every site considered success differently. Some were
happy to have introduced a new modality to their community, whereas others were eager
to implement every aspect of the program at the appropriate dose. To assess the range of
definitions of success3,4, we used four dependent variables including adherence to core
components, percentage of core components, dosage, and sustainability. Adherence is a
dichotomous variable indicating whether or not the site was able to implement every core
21
component of the program that was prescribed during research trials. Percentage of core
components is a continuous variable measuring the percentage of the core components
that were implemented per the number of components that should have been
implemented for each program. Thus, it is a ratio representing “percentage of adherence”
across all sites. The variables that comprise adherence and percentage of core
components are the same; they are merely calculated as a dichotomous or continuous
variable.
We also wanted to examine whether sites were able to implement the intervention
as frequently as it was prescribed during research trials. This is generally called the
“dosage” of the treatment and, depending upon the type of program, included everything
from the number of meetings between therapists and clients to the number of lessons
taught during the school year. Like adherence, dosage is a dichotomous variable,
indicating whether or not the site was able to implement all the dosage elements
prescribed. For example, there were five dosage elements for the PATHS program: (1)
lessons taught three times per week for a minimum of 20 minutes, (2) biweekly phone
consultation with program designers, (3) weekly or biweekly classroom observations by
the consultant, (4) biweekly individual or team meetings led by the consultant, and (5)
quarterly whole school staff discussions. Sustainability was also a dichotomous variable
representing whether sites were still implementing their program six months after the end
of the two-year grant (i.e., 30 months after startup).
Analysis
We used bivariate correlations to look at the significance and strength of the
relationships between each of the independent and dependent variables. To be
22
conservative in our bivariate analyses, we used two-tailed tests of significance. We then
conducted multiple regression analyses for each of the dependent variables (adherence,
percentage of core components, dose, and sustainability). For the dichotomous variables,
we conducted stepwise logistic regression, and for the continuous measure of percentage
of core components achieved, linear regression was used.5 This allowed us to examine
which factors emerged as most significant when the all independent variables were taken
into account. Because these analyses were exploratory and we did not want to omit
variables that might be important, we indicated significant relationships equal to or lower
than the .10 level for our regression analyses, and we included in the models relationships
between .11 and .20. This allowed us to include variables that might have been
significant if we had a larger sample size and more representative study population. Also,
this allowed us to identify the variables that might be important for future studies.
Given the general lack of multivariate analyses in implementation studies (for
exception see McCormick et al., 1995; Wenter et al., 2002), it seemed necessary to
compile a preliminary sketch of the factors that might be significant had our research
design conformed to experimental criteria. From these analyses, we can at least assume
that the significant relationships identified merit further analysis at a future date. In
addition, these regression analyses provide more support than case studies or correlation
analyses alone.
FINDINGS
Univariate and Bivariate Analyses
Overall, the 42 replication sites demonstrated high adherence to the programs. In
fact, 74% of the sites had implemented all core components of their program
23
(Adherence). Among the 11 sites that failed to implement all core components, four were
the failed sites that ended implementation prematurely, thus receiving implementation
scores of “0”. One site was slow in the start-up and had not implemented any of the core
components, and another site chose to implement only a few of the core components. The
other five sites implementated from from 75-92% of the core components of their chosen
programs (percentage of core program components achieved). Taken as a whole, the 42
sites achieved an average of 85% of the core program components.
Dosage was more difficult to achieve, with only 57% of the sites implementing all
dosage elements in the prescribed amounts. Most sites continued to implement the
program throughout the two years (n = 38). Six months beyond the termination of the
grant, an additional four sites, for a total of eight, had stopped implementing their
program. Thus, all of the effort that went into choosing sites carefully, providing training
and TA, and monitoring the process paid off in high rates of sustainability.
Table 2 shows the results of the correlation analysis, in which several bivariate
relationships appeared as significant. This was surprising given the fact that our number
of sites was relatively small and that most sites seemed to perform similarly on a couple
of the dependent variables. Ideal program and ideal staff characteristics were significantly
related to all four dependent variables, demonstrating their consistency in these analyses.
Significant on three of the dependent measures were: community support, time, ideal
champion characteristics, ideal agency characteristics, and number of TA visits (the only
variable whose effects were opposite to what had been predicted). The number of TA
visits had a significantly negative relationship with the adherence measures, suggesting
that the more TA visits that a site received, the less likely they were to implement with
24
high quality. This was due to the fact that the failed and floundering sites needed and
received more TA visits. TA quality, however, was significantly and positively associated
with both adherence and percentage of core program components achieved.
Although most of the sites worried about funding and suggested that lack of
resources was a major barrier to success, inconsistent funding did not appear to be
significantly related to any of our outcome measures. Inconsistent staffing was significant
on only one of the measures, indicating that more inconsistent staffing patterns resulted in
lower implementation of core components.
To summarize, 9 of the 10 independent variables were significantly associated
with either one or both of the adherence measures (adherence and/or percentage of core
program components achieved). Factors associated with dosage included ideal program
characteristics, community support, time to implement, and ideal staff characteristics.
Factors associated with sustainability included those same four variables plus ideal
champion characteristics, ideal agency characteristics, and number of TA visits, with
those sites receiving more TA visits more likely to fail.
Regression
We created four multiple regression models to identify the independent effects of
all of our independent variables upon adherence, percentage of core program components
achieved, dosage, and sustainability. For adherence, our dichotomous variable, only two
variables, ideal agency characteristics and TA quality, had a significant direct effect
(Table 3). Ideal agency characteristics was only significant at the .10 level. The
likelihood ratio R2 of .35 indicates that the three variables in the model had a moderate
effect. Once all of the other independent variables were taken into account, the variables
25
that were significant in the correlation analysis (number of TA visits; the characteristics
of staff, champions and program; and inconsistent staffing) had no direct effect on
Adherence.
For the continuous adherence variable, percentage of core program components
achieved, regression analysis revealed that three of the 10 independent variables had a
significant and independent effect: community support, inconsistent staffing, and TA
quality (Table 4). TA quality had a marginal effect at the .10 level. With an R2 of .48,
these variables were moderate predictors of adherence. Interestingly, program, staff,
champion, agency characteristics, number of TA visits, and time to implement, which
were significantly related at the bivariate level, did not have a direct effect once all
independent variables were taken into account.
The regression model for dosage had the greatest number of significant direct
predictors (Table 5). Ideal program characteristics and TA quality had significance levels
of .05, whereas number of TA visits and inconsistent staffing had only marginally
significant relationships at the .10 level. Interestingly, TA quality had a negative effect,
suggesting that as sites’ evaluations of TA quality decreased, dose increased. This
relationship at the bivariate level, although not significant, was also negative. Combined,
the six variables in the model had a likelihood ratio R2 of .42, which suggests that this
model had a moderate predictive power. Once other variables were taken into account,
time and community support failed to demonstrate any direct effects. Staffing
characteristics entered the model, but not at the traditional level of significance.
Interestingly, number of TA visits and TA quality, which had no significant relationships
at the bivariate level, appeared as significant at the multivariate level.
26
Sustainability, the last model, was significantly influenced only by program
characteristics (Table 6). As with other measures of success, many bivariate relationships
disappeared at the multivariate level for sustainability. More precisely, community
support, number of TA visits, time, staff, champion, and agency proved to have no
significant direct relationships with the ability of programs to sustain for 30 months (six
months after the grant ended). Also interesting is the fact that TA quality, which played a
role in the other success models, did not seem to emerge as a significant predictor in the
bivariate or the multivariate analyses. The likelihood ratio R2 of .56 suggests that this
variable had a moderate to high predictive power for sustainability.
Table 7 summarizes the significance of these factors across the four regression
models. Four of the factors were predictive in the various models at the traditional level
of significance (.05 or lower); these factors are signified by “As” in the table. These
factors were TA quality, ideal program characteristics, inconsistent staffing, and
community support. All but one (TA quality on dosage) were in the expected direction.
This variable was significant in two models and marginally significant (signified by a
“B” in the table) in another model. Two other factors were significant at levels between
.06 and .10 in one model each. These two factors, number of TA visits and ideal agency
characteristics also emerged in a couple of other models at levels of significance between
.11 and .20, as did ideal staffing characteristics and ideal champion. Interestingly, time
and inconsistent funding had no effects in any of the models.
DISCUSSION
One of the common assumptions in the literature is that “everything matters” to
implementation success. Based largely on case studies, this literature fails to identify
27
what factors are most important. This is an especially relevant point to practitioners who
face numerous financial and time constraints, but who are interested in doing the best that
they can in the face of limited resources. For them, it becomes important to identify the
most efficient pathways to implementation success. The findings presented here hint at
what these pathways might be.
The purpose of these analyses was to identify the factors that played a role in
implementation success. Throughout the last twenty years, researchers and practitioners
have begun to identify, often in a “tales from the field” format, some of the factors that
help or hurt the implementation of programs in both research and real world settings. In a
recent effort to disseminate research-based violence prevention programs for two years in
a variety of sites throughout the United States, we examined how the factors identified in
the literature affected our sites.
The findings from this study corroborate the overarching arguments in the
literature. At a bivariate level, the characteristics of staff implementing programs, the
quantity and quality of training and technical assistance, community support, having time
to implement programs, possessing strong leadership (i.e., program champions), and the
characteristics of the agency in which a program is implemented influenced whether or
not a program was successfully implemented in a new environment. Although not
mentioned frequently in the literature, the characteristics of the program being
implemented also contributed to success in our study. Our findings at a multivariate level,
however, suggest that some of these relationships may be spurious or work through other
variables.
28
Just as there are different ways of defining and measuring “implementation
success,” there are also different factors that lead to the various types of success. Dose
seemed to be directly influenced by program characteristics, TA quality (although
negatively), number of TA visits, and the ability of sites to hire and retain all staff.
Adherence was influenced by TA quality and agency characteristics, and the Percentage
of Core Program Components that were implemented was directly influenced by
community support, TA quality, and inconsistent staffing. Sustainability was significantly
influenced by program characteristics. Thus, “successfully implementing” a program
will, in large part, depend upon what practitioners are hoping to achieve and how they
define success for themselves. One of the most common measures of success, at least
among the programs analyzed in this study, has been achieving successful behavioral
outcomes in youth, in other words, reducing or preventing delinquent, violent, or
substance using behavior. If individuals are interested in reproducing the program effects
that were achieved during research trials, then they should pay attention to the factors that
influence all of the success measures identified in this study.
Because TA quality emerged in three of the four regression models, the quality of
TA seems to be the most consistent, direct factor across multiple success measures. In
fact, TA quality seemed to be a more consistent predictor than the number of TA visits,
although number of TA visits had a marginally significant and positive relationship with
dosage. The number of TA visits negatively predicted Adherence and Sustainability.
These relationships were not significant at the standard level; however, we had allowed
variables to enter or remain in these models through the .20 level of significance so as not
to overlook variables of importance that may not have entered the models at the
29
traditional level. This initially suggests that more is not always better, at least in terms of
adherence and sustainability, and that the content and quality of TA packages remains
extremely important. It is also important to note throwing TA at floundering and failing
sites does not always help. Many of our sites failed despite the extra effort and TA that
was utilized on them. Interestingly, although high TA quality helped sites to achieve
adherence to all core components, it had a negative effect on achieving all the dosage
elements of a program. This is somewhat confusing. School programs had the greatest
difficulty achieving the dosage elements (analyses not provided in this paper), and the
most commonly given reason for this, in open-ended format, was lack of time (Mihalic et
al., 2002). It is possible that many TA packages do not address the problems, such as lack
of time, that prohibit implementing at the appropriate dosage. Additionally, the rating of
TA quality was assessed by the coordinator of each project. Ultimately, the success of a
program is dependent upon the implementers, and it is possible that implementers of the
program might have had a different rating of the TA quality. It is also possible that since
little information has been gleaned scientifically about the dosage thresholds necessary to
achieve program results, some TA providers may have been hesitant to require that their
programs be implemented at some required frequency, fearing disillusionment among the
implementers.
Because of its exploratory nature, this study was not able to discern the precise
components of TA quality that influenced the different types of success. For example, is
TA quality a measure of whether TA providers were likable, easy to reach, informative,
experienced, good listeners, or helpful in overcoming problems? This study does,
however, imply that providing ongoing technical assistance to agencies and schools
30
throughout the nation requires as much coordination, care, and scientific rigor as was
required during the research trials to prove that programs are successful. Given the
consistently powerful direct relationship between TA and implementation success, future
studies should carefully assess the exact characteristics of TA quality that play a role in
implementation success. For example, how long of a training period is appropriate, what
are the best methods for presenting the program in training, is ongoing TA assistance as
important as the initial training and for how long, can TA be accomplished over the
phone or are on-site visits necessary, how often is TA required for maximum program
benefits, etc.
In addition to the importance of TA and the need for more careful and systematic
analyses of TA quality, these data also suggest that characteristics of various programs
influence success, at least in terms of dosage and sustainability. Based on our findings,
practitioners may want to pay close attention to selecting the appropriate program. It is
most important to select programs that match the local needs and that are consistent with
the stated goals or mission of the school, agency, or community. Adopting agencies
should take into account program flexibility, availability of materials and manuals, time
necessary to implement and complexity of a particular program, and costs. If any of these
program characteristics do not match well with the funding, resources, and mission of the
implementing agency, they might be better off finding alternatives.
We also found that inconsistent staffing directly influenced dosage and percentage
of core components achieved, suggesting that failing to hire and/or retain a full staff may
influence the extent to which the program will be fully implemented. During CSPV site
visits, we found implementation to be set back on various occasions as sites struggled
31
with maintaining a full staff. Particularly relevant to implementation was the time needed
to hire and train new staff members, as well as the overload on other staff during these
periods that sometimes prevented them from carrying out many of their own duties.
Having the support of the community was also important in achieving a higher
percentage of core program components. Considering that many of the Blueprints
programs are complex and require extensive community and interagency linkages to
obtain client referrals (all treatment programs) and volunteers (BBBS and Treatment
Foster Care), the importance of this variable is not surprising. Others have also suggested
that programs seem to fare better when the larger community is involved and supportive
of the efforts.
Ideal agency characteristics demonstrated itself as a relatively important factor in
adherence to all program components and sustainability, although neither attained the
traditional level of significance. It is interesting to note that while other factors may
influence implementation of a larger percentage of core components, it may require an
exceptional agency with “ideal” characteristics (e.g., administrative support, open lines of
communication, clear lines of authority, structural stability, financial support, etc.) to
succeed in implement all elements of a program as designed and to prolong program life.
Another interesting finding emerging from this study is the failure to find
significant effects for the staffing scale, leadership scale (i.e., program champion), time,
and funding variables in our regression analyses. This is particularly interesting given
that the literature is liberally peppered with tales about how each of these factors has been
a consistent problem during past implementation efforts. In fact, a qualitative analysis of
these data (Mihalic et al., 2002) concurred with the literature and suggested that these
32
were important factors. The emergence of significant relationships between these
variables and the different success measures in the bivariate analysis provided further
proof of the importance of these factors. These variables, however, disappeared in our
multiple regression analyses, indicating that, when everything is taken into account, they
do not have independent, significant effects and do not play a direct role in any type of
success identified here. These variables may indirectly influence success (i.e., they may
influence other variables, which then influence success). In addition, they may have a
spurious relationship to success; meaning that they have no relationships within the
model and that the bivariate relationships observed completely disappear once other
variables are taken into consideration (i.e., the relationship observed is dependent upon a
third variable so that the association disappears when the third variable is controlled).
Future studies are needed to assess the direct and indirect effects of all factors
identified here. In fact, given the non-random and largely exploratory design of this
study, the findings presented here are best interpreted as a road map for other more
systematic analyses. We have suggested that there are several models of success and that
TA quality, program characteristics, and inconsistent staffing are the three most
consistent predictors across varying success measures. The Blueprints data was cross-
sectional, limiting causal inferences that can be attributed to these implementation
characteristics. Additionally, although Blueprints staff met with and gathered data from
various people during site visits, many of the measures used for this paper were created
from a single respondent (usually the program coordinator) from each site who
characterized the overall climate of the agency and staff. The insights gathered from this
33
person may not reflect those of other staff members. Thus, more research is needed to
support these findings.
Studies examining the implementation of a variety of programs, implemented in
diverse contexts, using random sample techniques and multiple respondents are needed to
definitively state what are the most or least important ingredients for the successful
adoption of new programs. In addition, future research would do well to identify alternate
measures of success and the variables that directly and indirectly influence all models of
“successful implementation.”
_______________________________ Support for this project was provided by the Office of Juvenile Justice and Delinquency
Prevention through grant #98-MU-MU-K005. The authors also wish to acknowledge the
hard work and dedication demonstrated by all the members of the research team involved
in this project.
1 The lack of attention to implementation is interesting given that when program evaluations are conducted,
findings indicate that prevention and intervention initiatives are often inadequately implemented (Silvia and
Thorne, 1997). In their study of the implementation of school-based prevention programs, Gottfredson,
Gottfredson, and Czech (2000) found that the average prevention initiative managed to implement only
57% of the criteria necessary to create behavior change. This led them to conclude that the quality of
prevention activities in the nation’s schools is too poor to produce desired outcomes (i.e., prevention of
juvenile delinquency among students). Similarly, a study to test the effectiveness of the Life Skills Training
Program in 56 New York State schools (Botvin, Baker, Dusenbury, Tortu, and Botvin, 1990) showed that
only 27% to 97% (mean of 68%) of the program material was covered. Only 75% of the students in the
study were exposed to 60% or more of the prevention program.
34
2 At the time, ten Blueprints programs had been identified, but only eight were included in this grant. One
of the programs was being implemented more widely by CSPV under a different OJJDP grant, and the
other program was not ready for widespread dissemination.
3To diversify our categories of success, we also initially considered the inclusion of staff satisfaction with
the program as a dependent measure. Because this variable could also be operationalized as an independent
measure of staff buy-in, and because there were no significant bivariate relationships with the other
independent variables, we did not include this in the final study.
4 We would argue that the outcome of primary importance is achieving reductions in the antisocial
behaviors targeted. To accomplish this, attention must be paid to both adherence to program components
and dose. Sustainability of programs is important; however, program life can be achieved absent of
effectiveness on behavioral outcomes. For instance, DARE has been demonstrated to be quite sustainable,
but has also shown to have little effect in reducing drug use.
5 For dosage, adherence, and percentage of the core, backwards elimination procedures were used. For
sustainability, forward inclusion was used after backward elimination procedures failed. With only 6 sites
in the analysis failing to sustain implementation six months after the grant’s end, the backwards elimination
procedure may have failed because some combination of independent variables resulted in the condition
described as “complete separation,” in which case the maximum likelihood logistic regression model
cannot be estimated.
35
REFERENCES
Blakely, C. H., Mayer, J. P., Gottschalk, R. G., Schmitt, N., Davidson, W. S., Roitman,
D. B., & Emshoff, J. G. (1987). The fidelity-adaptation debate: Implications for
the implementation of public sector social programs. American Journal of
Community Psychology, 15, 253-268.
Botvin, G. J. (1990). Substance abuse prevention: Theory, practice, and effectiveness. In
M. Tonry & J. Q. Wilson (Eds.), Drugs and Crime (pp. 461-519). Chicago:
University of Chicago Press.
Botvin, G. J., Baker, E., Dusenbury, L., Tortu, S., & Botvin, E. (1990). Preventing
adolescent drug abuse through a multimodal cognitive-behavioral approach:
Results of a 3-year study. Journal of Consulting and Clinical Psychology, 58,
437-446.
Catalano, R. F., Arthur, M. W., Hawkins, J. D., Bergland, L., & Olson, J. J. (1998).
Comprehensive community- and school- based interventions to prevent antisocial
behavior. In R. Loeber and D. P. Farrington (eds.), Serious and violent juvenile
offenders: Risk factors and successful interventions (pp. 367-388). Thousand
Oaks, CA: Sage Publications.
Colorado Healthy Communities Initiative (1998). Lessons from the field: The Colorado
Health Communities Initiative. Denver, CO: The Colorado Trust.
Coolbaugh, K., & Hansel, C. J. (2000). The Comprehensive Strategy: Lessons learned
from the pilot sites. OJJDP Juvenile Justice Bulletin. Washington D.C.: U.S.
Department of Justice, Office of Justice Programs, Office of Juvenile Justice and
Delinquency Prevention.
36
Connell, David, Turner, Ralph, & Mason, Elaine (1985). Summary of findings of the
School Health Education Evaluation: Health promotion effectiveness,
implementation, and costs. Journal of School Health, 55, 316-321.
Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early
secondary prevention: Are implementation effects out of control? Clinical
Psychology Review, 18, 23-45.
Davis, L. & Tolan, P. H. (1993). Alternative and preventive intervention. In P. H. Tolan
& B. J. Cohler (Eds.), Handbook of clinical research and practice with
adolescents. New York: John Wiley and Sons.
Diebold, C. T., Miller, G., Gensheimer, L. K., Mondschein, E., & Ohmart, H. (2000).
Building an intervention: A theoretical and practical infrastructure for planning,
implementing, and evaluating a metropolitan-wide school-to-career initiative.
Journal of Educational and Psychological Consultation, 11, 147-172.
Domitrovich, C. E., & Greenberg, M. T. (2000). The study of implementation: Current
findings from effective programs that prevent mental disorders in school-aged
children. Journal of Educational and Psychological Consultation, 11, 193-221.
Dryfoos, J. G. (1990). Adolescents at risk: Prevalence and prevention. New York:
Oxford Univesity Press.
DuBois, D. L., Holloway, B. E., Valentine, J. C., & Cooper, H. (2002). Effectiveness of
mentoring programs for youth: A meta-analytic review. American Journal of
Community Psychology, 30, 157-197.
37
Dunworth, T., Mills, G., Cordner, G., & Greene, J. (1999). National evaluation of Weed
& Seed cross-site analysis. Washington, D.C.: U.S. Department of Justice, Office
of Justice Programs.
Durlak, J. A. (1998). Why program implementation is important. Journal of Prevention
& Intervention in the Community, 17, 5-18.
Durlak, J. A. (1997). Successful prevention programs for children and adolescents. New
York: Plenum.
Elias, M. J., & Clabby, J. F. (1992). Building social problem solving skills: Guidelines
from a school-based program. San Francisco: Jossey-Boss.
Ellickson, P., Petersilia, J., with Caggiano, M., & Polin, S. (1983). Implementing new
ideas in criminal justice. Santa Monica, CA: The Rand Corporation.
Elliott, D. S. (1997). Blueprints for Violence Prevention (Vols. 1-11). Boulder, CO:
Center for the Study and Prevention of Violence, Institute of Behavioral Science,
University of Colorado.
Fagan, J. A. (1990). Treatment and reintegration of violent juvenile offenders:
Experimental results. Justice Quarterly, 7, 233-263.
Farrell, A. D., Meyer, A. L., Kung, E. M., & Sullivan, T. N. (2001). Development and
evaluation of school-based violence prevention programs. Journal of Clinical
Child Pschology, 30, 207-220.
Flay, B. R. (1999). Effects of Positive Action Program and Positive Action training on
outcomes. Twin Falls, ID: Positive Action, Inc.
Fors, S. W., & Doster, M. E. (1985). Implication of results: Factors for success. Journal
of School Health, 55, 332-334.
38
Freudenburg, N. (1998). Barriers to implementing promising systemic and programmatic
approaches for working with substance-abusing youthful offenders. Annapolis,
MD: Robert Wood Johnson Foundation.
Fullan, M. (1992). Successful school improvement: The implementation perspective and
beyond. Buckingham, England: Open University Press.
Gager, P. J., & Elias, M. J. (1997). Implementing prevention programs in high-risk
environments: Application of the resiliency paradigm. American Journal of
Orthopsychiatry, 67, 363-373.
Gendreau, P., Goggin, C., & Smith, P. (1999). The forgotten issue in effective
correctional treatment: Program implementation. International Journal of
Offender Therapy and Comparative Criminology, 43, 180-187.
Gendreau, P., & Ross, R. R. (1980). Effective correctional treatment. Toronto:
Butterworths.
Gingiss, P. L. (1992). Enhancing program implementation and maintenance through a
multiphase approach to peer-based staff development. Journal of School Health,
62, 161-166.
Gottfredson, D. C., & Gottfredson, G. D. (2002). Quality of school-based prevention
programs: Results from a national survey. Journal of Research in Crime and
Delinquency, 39, 3-35.
Gottfredson, G. D., Gottfredson, D. C., & Czeh, E. R. (2000). Summary: National study
of delinquency prevention in schools. Ellicott City, MD: Gottfredson Associates,
Inc.
39
Gray, D. O., Emshoff, J., Jakes, S., & Blakely, C. (draft). ESID and dissemination
research: A case study and critique of a change model’s fidelity. College Station:
Texas A&M University.
Greenberg, M., Domitrovich, C., Graczyk, P., & Zins, J. (2001). The study of
implementation in school-based preventive interventions: Theory, research, &
practice. Washington D.C.: Center for Mental Health Services, Substance Abuse
and Mental Health Administration, U. S. Department of Health and Human
Services.
Greenberg, M., Domitrovich, C., & Bumbarger, B. (1999). Preventing mental disorders
in school-aged children: A review of the effectiveness of prevention programs.
Washington D.C.: Center for Mental Health Services, U. S. Department of Health
and Human Services.
Gresham, F. M., Cohen, S., Rosenblum, S., Gansle, K. A., & Noell, G. H., (1993).
Treatment integrity of school-based behavioral intervention studies: 1980-1990.
School Psychology Review, 22, 254-272.
Harris, P., & Smith, S. (1996). Developing community corrections: An implementation
perspective. In F. X. Hartmann (Ed.), Choosing correctional options that work:
Defining the demand and evaluating the supply. Thousand Oaks: Sage
Publications.
Hawkins, J. D., Catalano, R. F., & Brewer, D. D. (1995). Preventing serious, violent, and
chronic offending: Effective strategies from conception to age 6. In J. C. Howell,
B. Krisberg, J. D. Hawkins, and J. J. Wilson (eds.), A sourcebook on serious
40
violent and chronic juvenile offenders (pp. 61-141). Thousand Oaks, CA: Sage
Publications.
Hawkins, J. D., Farrington, D. P., & Catalano, R. F. (1998). Reducing violence through
the schools. In D. S. Elliott, B. A. Hamburg, & K. R. Williams (Eds.), Violence in
American schools (pp. 188-216). New York: Cambridge University Press.
Hunter, L., Elias, M. J., & Norris, J. (2001). School-based violence prevention:
Challenges and lessons learned from an action research project. Journal of School
Psychology, 39,161-175.
Kegler, M. C., Steckler, A., Malek, S. H., & McLeroy, K. (1998). A multiple case study
of implementation in 10 local Project ASSIST coalitions in North Carolina.
Health Education Research, 13, 225-238.
King, K. A., Wagner, D. I., & Hedrick, B. (2001). Safe and drug-free school
coordinators’ perceived needs to improve violence and drug prevention programs.
Journal of School Health, 71, 236-241.
Kramer, L, Laumann, G., & Brunson, L. (2000). Implementation and diffusion of the
Rainbows Program in rural communities: Implications for school-based
prevention programming. Journal of Educational and Psychological
Consultation, 11, 37-64.
Lipsey, M. W. (1999). Can intervention rehabilitate serious delinquents? Annals, AAPSS,
565, 142-166.
Lipsey, M. W. (1992). Juvenile delinquency treatment: A meta-analytic inquiry into the
variability of effects. In T. D. Cook, H. Cooper, D. S. Cordray, H. Hartmann, L.
41
V. Hedges, R. J. Light, T. A. Louis, & F. Mosteller (Eds.), Meta-analysis for
explanation (pp. 83-127). New York: Russell Sage Foundation.
Lipsey, M. W., & Wilson, D. B. (1997). Effective interventions for serious juvenile
offenders: A synthesis of research. Nashville, TN: Vanderbilt University.
Lynch, K. B., Geller, S. R., Hunt, D. R., Galano, J. & Dubas, J. S. (1998). Successful
program development using implementation evaluation. Journal of Prevention &
Intervention in the Community, 17, 51-64.
Lytle, L. A., Davidann, B. Z., Bachman, K., Edmundson, E. W., Johnson, C. C., Reeds, J.
N., Wambsgans, K. C., & Budman, S. (1994). CATCH: Challenges of conducting
process evaluation in a multicenter trial. Health Education Quarterly, Supplement
2, S129-S141.
Martinson, R. (1974). What works? – Questions and answers about prison reform. Public
Interest. June: 22-54.
McCormick, L. K., Steckler, A. B., & McLeroy, K. R. (1995). Diffusion of innovations in
schools: A study of adoption and implementation of school-based tobacco
prevention curricula. American Journal of Health Promotion, 9, 210-219.
McGrew, J. H., Bond, G. R., Dietzen, L., & Salyers, M. (1994). Measuring the fidelity of
implementation of a mental health model. Journal of Consulting and Clinical
Psychology, 62, 670-678.
Mears, D. P., & Kelly, W. R. (2002). Linking process and outcomes in evaluating a
statewide drug treatment program for youthful offenders. Crime & Delinquency,
48, 99-115.
42
Mendel, R. A. (2000). Less hype, more help: Reducing juvenile crime, what works—and
what doesn’t. Wahington D.C.: American Youth Policy Forum.
Mihalic, S., Ballard, D., Michalski, A., Tortorice, J., Cunningham, L., & Argamaso, S.
(2002). Blueprints for Violence Prevention, Violence Initiative: Final Process
Evaluation Report. Boulder, CO: University of Colorado.
Mihalic, S., Irwin, K., Elliott, D., Fagan, A., & Hansen, D. (2001). Blueprints for
Violence Prevention. OJJDP: Juvenile Justice Bulletin. Wahington D.C.: U.S.
Department of Justice, Office of Juvenile Justice and Delinquency Prevention.
Pankratz, M., Hallfors, D., & Cho, H. (2002). Measuring perceptions of innovation
adoption: The diffusion of a federal drug prevention policy. Health Educational
Research, 17, 315-326.
Parcel, G. S., Ross, J. G., Lavin, A. T., Portnoy, B., Nelson, G. D., & Winters, F. (1991).
Enhancing implementation of the Teenage Health Teaching Modules. Journal of
School Health, 61, 35-38.
Perry, C. L., Murray, D. M., & Griffin, G. (1990). Evaluating the statewide dissemination
of smoking prevention curricula: Factors in teacher compliance.
Petersilia, J. (1990). Conditions that permit intensive supervision programs to survive.
Crime & Delinquency, 36, 126-145.
Regnery, A. S. (1985). Getting away with murder. Policy Review, 34:1-4.
Rogers, E. M. (1995). Diffusion of innovation. New York: The Free Press.
Rogers, E. M. (1983). Diffusion of innovation. New York: The Free Press.
43
Ross, J. G., Luepker, R. V., Nelson, G. D., Saavedra, P. & Hubbard, B. M. (1991).
Teenage Health Teaching Modules: Impact of teacher training on implementation
and student outcomes. Journal of School Health, 61, 31-35.
Sechrest, L., White, S., & Brown, E. (1979). Rehabilitation of criminal offenders:
Problems and prospects. Washington D.C.: National Academy of Sciences.
Sherman, L. W., Gottfredson, D. C., MacKenzie, D., Eck, J., Reuter, P., & Bushway, S.
(1997). Preventing crime: What works, what doesn’t, what’s promising: A report
to the U.S. Congress. University of Maryland.
Silvia, E. S., & Thorne, J. (1997). School-based drug prevention programs: A
longitudinal study in selected school districts. Research Triangle, NC: Research
Triangle Institute.
Stoil, M. J., Hill, G. A., Jansen, M. A., Sambrano, S., & Winn, F. J. (2000). Benefits of
community-based demonstration efforts: Knowledge gained in substance abuse
prevention. Journal of Community Psychology, 28, 375-389.
Taggart, V. S., Bush, P. J., Zuckerman, A. E., & Theiss, P. K. (1990). A process
evaluation of the District of Columbia “Know Your Body” project. Journal of
School Health, 60, 60-66.
Tobler, N. S. (1992). Drug prevention programs can work: Research findings. Journal of
Addictive Diseases, 11, 1-28.
Tolan, P., & Guerra, N. (1994). What works in reducing adolescent violence: An
empirical review of the field. Boulder, CO: University of Colorado, Institute of
Behavioral Science.
44
Van Voorhis, P. (1987). Correctional effectiveness: The high cost of ignoring success.
Federal Probation, 51:56-62.
Wasserman, G. A., & Miller, L. S. (1998). The prevention of serious and violent juvenile
offending. In R. Loeber and D. P. Farrington (eds.), Serious and violent juvenile
offenders: Risk factors and successful interventions (pp. 197-247). Thousand
Oaks, CA: Sage Publications.
Wenter, D. L., Ennett, S. T., Ribisl, K. M., Vincus, A. A., Rohrbach, L, Ringwalt, C. L.,
& Jones, S. M. (2002). Comprehensiveness of substance use prevention programs
in U. S. middle schools. Journal of Adolescent Health, 30, 455-462.
Wilson, S. J., & Lipsey, M. W. (2000). Effects of school violence prevention programs
on aggressive and disruptive behavior: A meta-analysis of outcome evaluations.
Nashville: Center for Evaluation Research and methodology, Vanderbilt Institute
for Public Policy.