Upload
holly
View
39
Download
1
Tags:
Embed Size (px)
DESCRIPTION
The Importance of Assessing Program Implementation Fidelity. Finn-Aage Esbensen, Ph.D. University of Missouri-St. Louis Law Enforcement Planners Meeting St. Louis, MO September 2009 - PowerPoint PPT Presentation
Citation preview
Finn-Aage Esbensen, Ph.D.University of Missouri-St. Louis
Law Enforcement Planners MeetingSt. Louis, MO
September 2009This research is funded by the National Institute of Justice, Office of Justice Programs, U.S. Department of Justice, Award # 2003-JN-FX-0003 (October 2003 – June 2009) and Award # 2006-JV-FX-0011 (July 1, 2006 – December 2012). The opinions, findings, and conclusions or recommendations expressed in this presentation are those of the author and do not necessarily reflect the views of the Department of Justice.
OverviewComprehensive process evaluations are essential
components of outcome evaluations.
How and why to assess fidelity of program implementation.
Results from two NIJ funded evaluations will be described.
Both programs are law enforcement, in-school prevention programs.
Program Implementation Program Fidelity
(1) Dosage- is enough of the program actually implemented to expect the anticipated effect.
(2) Program Adherence- to what extent are program components actually taught or delivered.
(3) Quality of Delivery- if delivered, what is the quality of the delivery?
Why Document Program Fidelity?It Compliments Outcome Analyses:
One can confidently tie outcome effects to the program/policy under study.
- Or –When no effect is found
(1) the program, as designed and implemented, has no effect Often Assumed
(2) the program, as designed, is not well suited for the given setting
(3) implementation failure
Research on Program Fidelity
Research to date: “about half of school-based prevention activities are of such poor quality that they cannot reasonably be expected to make a difference…”
(Gottfredson et al., 2000: 7).
Low levels of program effect …are more likely attributable to improper implementation practices than inept program content (DuBois et al. 2002…)
Common Hurdles to Program Fidelity
The idea that “something is better than nothing.”Reality: Half-hearted implementation of programs
designed for high-risk youth fair no better than no implementation at all (Kovaleski et al., 1999).
Insufficient “buy in” on the part of providers.Effect: Program providers deviate from the
program, and administer the lesson(s) differently than intended (Ruiz-Primo, 2005).
Case StudiesTeens, Crime, and the Community and
Community Works (CW)National Crime Prevention CouncilVictimization reduction program
G.R.E.A.T. – Gang Resistance Education and TrainingBureau of Justice Assistance Gang prevention program
Common Program CharacteristicsUtilize law enforcement officers (or probation
officers)School-based and general prevention modelIncorporate interactive and active learning
strategiesEmphasize skills buildingRequire implementers to be trained
CW is a two-day sessionG.R.E.A.T. is one week for experienced officers or
two weeks for officers who have not previously taught
Program componentsSkill-based curriculum consisting of:
A 31-Lesson CurriculumCore program consists of 11 lessons that, if
taught with fidelity, requires approximately 30 hours of class time
The use of community resource people (CRP) to deliver the program
A group service learning project, referred to as Action Projects (AP)
Taught by teacher &/or SRO/JPO
CW Program GoalsIncrease teen awareness of their vulnerability to
crimeEducate teens on how to protect themselves and
their communitiesMotivate teens to take action in their communitiesHelp teens understand the costs of crime and what
can be done to prevent itFoster community relations by bringing community
resource people into the sessionsBond youth to the community through increased
self-esteemIncrease teen empathy toward crime victims
G.R.E.A.T. Program
Taught by uniformed law enforcement officerSkills building program13 lessons that take 13-14 hours to complete
Goal-setting; Decision-making; Communication skills; Conflict resolution
Student workbookG.R.E.A.T. project
G.R.E.A.T. Program Goals
Prevent youth from becoming involved in gangs, violence, and delinquency
Develop positive relationships between youth and law enforcement
Evaluation DesignsSimilar design for evaluation of both projectsPurposive selection of cities (G.R.E.A.T.) or
schools (CW)Observation of trainings
Observation of program delivery
Surveys of teachers, officers, &/or implementers
Outcome component (longitudinal survey of students)
CW Site Selection15 middle school programs in 9 cities/towns in 4
states
98 classes – 49 CW & 49 Comparison
2 – 10 classes per school (half CW, half comparison)
1,686 students34 to 229 students per school6th – 9th grade
G.R.E.A.T. Site Selection31 schools –
4 to 6 in each of 7 cities (representative of districts)195 classes –
102 G.R.E.A.T. & 93 Control3 - 12 classes in each school (half receive
G.R.E.A.T.) 24 - 35 classes per city
3,820 StudentsBetween 67 and 186 students per school6th or 7th (5 schools) grade
Training: Research QuestionsAre trainees adequately prepared to teach
the program after completion of training?Are trainees provided the opportunity to
practice program delivery?Are teaching strategies modeled during
training?Are trainers professional and knowledgeable?Do trainers adhere to suggested time frames?Is the time allotted to training adequate?
Community Works TrainingObserved 7 training sessions (6 two-day, 1
one-day refresher training).
Typical Training Scenario3 trainers (1 primary, 2 assistants)25-30 traineesTrainees represented a mixture of professions
(SROs, teachers, juvenile justice workers, …)
Community Works TrainingTrainers
Appeared comfortable when speaking from a script.
When asked questions, trainers sometimes provided vague or inaccurate information e.g., misdemeanor vs. felony
Emphasized the program was “adaptable and flexible” Is it a program or a resource?
Community Works TrainingTeaching Strategies
Training staff made use of many group activities and teaching strategies. e.g., icebreakers were modeled frequently
Activities were seldom linked to program content, even though a clear link exists.
Community Works TrainingProgram Content
Training provided attendees with an overview of the program, as well as key components of the program
More time is needed to allow for more modeling of program delivery.
More time needs to be devoted to explaining the program as a whole, not just individual lessons (e.g., skills building approach).
Little time was devoted to explaining the substantive content of the lessons. (i.e., not all in attendance had a law background)
Community Works TrainingProgram Content (Continued)
Trainers did not seem to alter trainings according to their audience. (e.g., English teachers may need more attention paid
to substantive knowledge, while SROs may need more time devoted to teaching strategies.)
Trainers did not provide remedies to common problems faced when implementing the program (e.g., substitutes, short classes).
Community Works TrainingTime Management
Posted timelines for trainings were seldom followed (e.g., long lunch, leave early).
Demonstration of a 45 minute lesson lasted 90 minutes. Coincidently, this lesson was never implemented
correctly in the classroom, per our observations.When trainees voiced concern over the time
needed to implement the program, trainers stated that one can either “use” the curriculum or “implement” the program; both could be used effectively (Flexibility).
Community Works TrainingBased on observation of training, what are your
expectations for program implementation fidelity?
G.R.E.A.T. TrainingObserved 7 G.R.E.A.T. Officer Trainings (GOTs)
(5 one week sessions and 2 two week session)
Typical Training Scenario (8+ trainers)Seminar Supervisor & logistical support from
Regional Administrator and staff6 trainers (Team Leaders)Gang Specialist and Education Specialist30 - 36 traineesTrainees primarily local law enforcement officers
G.R.E.A.T. TrainingTraining consists of multiple teaching blocks,
including:Transition from law enforcement to preventionGangsAdolescent development and instructional
issuesIssues of G.R.E.A.T. concern
Team leader models each lesson (although a few lessons are taught as an overview)
G.R.E.A.T. Training Team leaders typically adhere to training
rules and structure (e.g., time frames)Trainers stress importance of program
adherence, including historical development and program revision, skills-based program in which each lesson builds on prior lessons
Trainers model lessons so that trainees have experience of observing different teaching styles and different personal approaches to teaching standard curriculum
G.R.E.A.T. TrainingTeam leaders are “debriefed” by Supervisor
after each modeling sessionGroup discussion of lesson content and/or
delivery issuesTrainers utilize multiple teaching strategiesTeam leaders meet with team of 5 – 6 officers
for more in-depth discussion of program
G.R.E.A.T. TrainingTrainees (officers) are required to develop
one lesson overview and present to groupTrainees are subsequently required to model
their lesson to their teamEach trainee has experience with one lesson
and observes five others modeled by their team members
Trainees have considerable exposure to program content in this manner
G.R.E.A.T. Training ConcernsTrainings sometimes adjust schedule to
accommodate gang or education specialists – suggest consistency to enhance “flow”
Consider more emphasis on classroom management issues
Gang block has tendency to regress to discussion of stereotypical media images of gangs
Given observations of training, what are expectations for program implementation fidelity?
Observation of Program Delivery
Observed more than 100 CW classrooms
Observed more than 500 G.R.E.A.T. classrooms
SESSION 1 CHECKLISTCreating a Community Vision
Part 1 (55 minute estimate)
Instructions: Please place a check mark after each area that is covered in the session. Leave the line blank when certain areas are not covered. Also, please note the approximate time spent on each part and steps within each part. After the checklist is completed, please complete the subsequent qualitative session evaluation.
Start Time: ________
Step A: Warm up (15 minute estimate)
1. Introduction and welcome _______a. teens introduce themselves and state their favorite activity ______b. name tags distributed and used _______
2. Icebreaker: Shapes (kids with same shapes get together and form shape) ________3. Overview of program and today's session _________
a. teens put together puzzle messages (small group activity) _______
Actual time spent on Step A: ______
Step B: Creating a Community Vision (30 minute estimate) 1. Brainstorm what does it mean to be safe? _________2. Defining community ________a. help teens to come up with a definition similar to this: "a group of people that have something important in common" ______
b. teens consider what makes a school a community ______c. ask teens what things we have in common to make school a community _____
3. Think about what makes a community safe and secure _______
CW Observation Results
Overall, out of the 110 class periods observed, only 18 (16%) lessons were found to be delivered in the manner intended by program developers.
Common Deviations Skipping Steps in the LessonPoor Time Management Covering Multiple Parts in One Class Period
CW Observation ResultsThe program, as designed, is not well suited for
implementation in the given setting.
One school classified as high fidelity.
Three schools classified as medium fidelity. Could conceivably link program outcome to program
delivery
Eight schools classified as low fidelity. If program effect found, it could not be attributed to
program delivery.
G.R.E.A.T. Classroom ObservationsPurpose: determine extent to which program
is delivered as intended with regard to: Dosage Adherence to intended lessons Quality of instruction
518 observations of G.R.E.A.T. program delivery
33 different officers
Each lesson observed at least 30 times
G.R.E.A.T. ObservationsTime frame adherence
Did not appear to be a problem if there were no behavioral problems
When problems occurred, generally due to external factors (shortened day, substitute teacher, fire drill)
Topical areaOverall, officers covered all lessons
G.R.E.A.T. ObservationsLesson Adherence
Overall close adherenceSome officers skipped around, teaching
components out of orderOverall quality
Generally good qualityExternal factors most notable in terms of
affecting lesson quality (time of day, substitute teacher, some particularly “bad” classrooms)
G.R.E.A.T. Overall Program Fidelity27 officers (80 classes) implemented with
good fidelity (adhered to lesson components, topical areas, time frame, and “quality” of instruction)
3 officers (11 classes) had low fidelity (implemented much of lesson but with low quality)
3 officers (11 classes) failed to implement (did not cover lesson components and poor quality)
Implementer SurveysA complimentary technique to address program
fidelity is through implementer surveys.
Allows us to document lessons taught, CRP used, and how Action Projects were implemented.
Note: Research shows that this technique tends to overestimate degree of program fidelity.
CW Implementer Surveys
Impressions of the CurriculumOverall positive assessment.82% agreed that the program raised awareness of
programs that assist victims of crime.Most agreed that the lessons were relevant.59% felt the materials were appealing to students.
This may signal that materials should be modified to make them more appealing.
CW Implementer Surveys Program Fidelity
Programs were categorized as implementers and non-implementers based on the following standards:1) At least 70% of the “core 11” lessons taught2) At least 20 hours of program delivery time3) CRP had to be used4) AP had to be at least planned and initiated
Based upon the implementer surveys, 6 of the 14 schools could have been classified as having implemented the program.
G.R.E.A.T. Officer SurveySurvey of G.R.E.A.T. officers in 7 cities
(anonymous questionnaires)
137/205 = 67% response rate
G.R.E.A.T. Officer AttitudesProgram Goals (% agreeing or strongly agreeing)
G.R.E.A.T. teaches skills to avoid gangs and violence (80%)
G.R.E.A.T. has improved police/youth relations (90%)
G.R.E.A.T. strengthened police/community relations (90%)
G.R.E.A.T. has reduced community gang problem (29%)
G.R.E.A.T. has reduced community crime problem (29%)
G.R.E.A.T. Officer AttitudesProgram
G.R.E.A.T. is appropriate for age and comprehension levels (83%)
G.R.E.A.T. lessons address risk factors for gangs and delinquency (71%)
Enough time to cover important topics (72%)
G.R.E.A.T. School Personnel SurveyPurpose: obtain opinions about school-
based prevention programs in general and G.R.E.A.T. in particular.
All G.R.E.A.T. grade-level teachers and administrators at 31 participating schools (n=231/373, 62%).
G.R.E.A.T. School PersonnelProgram Assessment (% who agreed or strongly
agreed)
88% in favor of having G.R.E.A.T. in their school
90% - curriculum age appropriate
82% - gives students skills to avoid gangs and violence
86% - improved students attitudes about police
52% - reduced gang participation at school
46% - reduced gang participation in community
G.R.E.A.T. School PersonnelG.R.E.A.T. Classroom Teachers
80% - officer adequately trained in content
70% - officer adequately trained in classroom management
81% - students responsive to officer
78% - officer punctual (or rescheduled if needed
48% - officer incorporated teacher into lesson
14% - officer strayed from lesson plan
16% - officer had difficulty controlling class
G.R.E.A.T. Bottom LineGenerally favorable opinions expressed by
officers and school personnel
However, it is noteworthy that both officers and teachers more positive about mediating effects than on program objectives
Summary CommentsEvidence-Based Prevention Programming has
become a reality.Funding, Adoption in Schools/Communities
Attention to program fidelity is necessary.Tale of Two Programs
CW- Implementation Failure and No Program Effect
Is this a bad program, or one that was not implemented correctly?
With G.R.E.A.T., we have program fidelity and preliminary analyses indicate positive program effect.
Thank you for listening.