Community Plan Implementation Training - SDRGsdrg.org/ctcresource/community plan implementation training/trainer... · Trainer’s Guide (90 minutes) Ensuring High-Fidelity Implementation

Mod

ule

3

Trainer’s Guide(90 minutes)

Ensuring High-FidelityImplementation

Community PlanImplementationTraining

Communities That Care

3—2

Slides for Module 3

Module 3 ......................................................................... 3-1

We are here. ................................................................... 3-2

Module 3 goal................................................................. 3-3

Objectives ....................................................................... 3-4

What is implementation fidelity?..................................... 3-5

Implementation fidelity is key to program success! ...... 3-6

Adapting programs to local audience needs................. 3-7

Keys to successful implementation................................ 3-8

Ensuring administrative support .................................... 3-9

Selecting implementers.................................................. 3-10

Ensuring support of implementers................................. 3-11

Purposes of pre-implementation meetings.................... 3-12

Implementer trainings..................................................... 3-13

Planning for trainings...................................................... 3-14

Implementation plans ..................................................... 3-15

Implementation and evaluation process ........................ 3-16

Why evaluate implementation? ...................................... 3-17

Possible implementation problems................................ 3-18

Begin evaluating implementation beforeprogram delivery............................................................. 3-19

3—3

Data-collection methods ................................................ 3-20

Monitoring planning meetings ....................................... 3-21

Monitoring implementer training .................................... 3-22

Monitoring program delivery .......................................... 3-23

Designing an implementation evaluation....................... 3-24

Next steps....................................................................... 3-25

Next up ........................................................................... 3-26

Mouse-Click Icon(for a computer-based presentation)The mouse-click icon shows you what information will come up onthe slide when you click. (Some slides use several clicks.)


Notes

Slide 3-1

Module 3

3—4

Activity: Replicating a planMaterials: Unlined paper

Time: 10 minutes

Instructions:

1. Distribute a sheet of paper to each participant.

2. Ask participants to draw a house. Don’t give them any other instructions.

3. After a minute or two, ask them to compare their pictures with one another. Ask what they notice about the drawings. [They all look different.]

4. Now, ask them to draw another house. This time, tell them the houseshould be a 2-story colonial with four front windows on each floor, acenter chimney and a “welcome” sign on the front door.

5. After a couple of minutes, ask participants to compare pictures again.Ask if the pictures look more similar.

Debrief.

When everyone had a detailed plan and followed it, the pictures looked moresimilar. If your goal is to build a certain kind of house, you need to developplans and follow them—or you might end up with a different kind of housefrom what you wanted. It’s the same when implementing tested, effectiveprograms. To ensure that they have the same level of effectiveness as in theoriginal studies, you need to have a complete implementation plan and takesteps to ensure that the plan is followed as closely as possible.

Notes

Slide 3-2

3—5


In this module, we’ll discuss steps you can take to ensure that implementershave the necessary skills and expertise to implement the tested, effectiveprograms in your Community Action Plan, and other things you can do toensure high-quality—or high-fidelity—implementation.

Notes

Slide 3-3

Module 3

3—6

Review the goal.

Notes

Slide 3-4

3—7


Review the objectives.

Notes

Slide 3-5

Module 3

3—8

Objective 1: Explain the elements of successful, high-fidelity implementation.High-fidelity implementation means implementing a program according to theoriginal design. Each tested, effective program has many elements that cometogether to make the program work. If all of the key elements aren’t implementedas designed, the program may not have the same level of effectiveness as it did inthe original trials.

Review the slide.

High-fidelity implementation generally refers to:

• Reaching the target audience. This includes ensuring that the target audienceparticipates in the program in expected numbers, as well as conveying the keymessages in a meaningful way to program participants.

• Delivering all components of the program according to the original design. Forexample, the PATHS curriculum for elementary school students involves aseries of structured lessons addressing three components: self-control,emotional understanding and interpersonal problem solving. Teachingmethods include visual, verbal and hands-on approaches. All threecomponents need to be delivered using the prescribed methods to ensurethat the program has the desired effect on participants.

• Delivering the program at the intended dosage. Dosage refers to duration andfrequency of program delivery. For example, the Nurse-Family PartnershipTM

program requires that nurses visit a mother’s home beginning before the 28thweek of her pregnancy and for two years after the birth of her first child. If visitsstop when the child is born, the program isn’t being delivered for the durationnecessary to ensure effectiveness. Other programs may require that a specificnumber of sessions are delivered in a certain time period.

The implementation outcomes you wrote for each program in Phase Four of theCommunities That Care process should specify each of these conditions.

Notes

3—9


Activity: Reviewing implementation outcomesTime: 15 minutes

Instructions:

1. Have participants form groups according to people/agencies who will be collaborating to implement selected programs identified in theCommunity Action Plan.

2. Assign each group a program from the Community Action Plan. Explain that they will be working with this program for the remainder ofthe training.

3. Have each group look up its assigned program in the Community ActionPlan and locate the implementation outcomes and any implementationplans that are included in the plan.

4. Check for understanding by asking a few groups to summarize theimplementation outcomes and implementation components for theirassigned program.

Notes

Slide 3-6

Module 3

3—10

Objective 1: Explain the elements of successful, high-fidelity implementation.Review the slide.

Implementing tested, effective programs without taking steps to ensurefidelity can result in a program that has little effect on participants. That’s awaste of time and resources. On the other hand, carefully planning andmonitoring implementation can help ensure the program’s success.

Notes

Slide 3-7

3—11


Objective 1: Explain the elements of successful, high-fidelity implementation.It is very important to make every effort to implement tested, effectiveprograms with the highest possible degree of fidelity. Studies have shownthat adapting, deleting or adding program components can decrease or evennullify the effectiveness of a tested, effective program.

However, there are certain implementation decisions local implementers will need to make based on their knowledge of the target audience. These include:

• Recruitment—While the eligibility criteria for tested, effective programsmust be followed, local implementers should use recruitment strategiesthat they know to be effective with the target audience.

• Scheduling—Implementers should schedule program sessions at timesthat are realistic for participants to attend. For example, it may be best toschedule parent-training sessions in the early evening. As long as allcomponents of a program are delivered at the correct dosage,scheduling decisions should be based on local needs.

• Communication—Local implementers often know how to communicatemost effectively with the target audience. While the teaching techniquesprescribed by a tested, effective program must be used to ensure fidelity,implementers should use language and motivational techniques that willbe most effective at motivating participants and helping them understandwhat is being asked of them.

Notes

Slide 3-8

Module 3

3—12

Objective 1: Explain the elements of successful, high-fidelity implementation.According to research, certain local conditions are important for successfulprogram implementation.

Review the slide.

• The support of site administrators is key to ensuring that programs areimplemented with fidelity.

• It’s important that the individuals who deliver the program have theappropriate qualifications, experience and training. Implementers alsoneed to support the program approach.

• Holding planning meetings with everyone involved in implementationand training implementers in program methods are two vital steps thathelp ensure successful implementation.

• Written plans detailing every aspect of implementation help ensure thateach program is implemented as designed.

• Ongoing monitoring of the implementation process and technicalassistance from program developers are also important toimplementation fidelity.

Now let’s discuss these points in more detail.

Notes

Slide 3-9

3—13


Objective 2: Develop implementation plans.Site administrators—school principals, agency directors, etc.—are thegatekeepers to implementing a program at their site. They have the power toprovide the staffing and other resources needed for implementation. They alsoplay an important role in motivating staff to implement a program with fidelity.That’s why you need to obtain their support as early as possible. Here are somestrategies for ensuring administrator support.

Review the slide.

• Provide administrators with information about the program’s purpose,approach and expected benefits to the target population as well as thecommunity as a whole. Emphasize how the program will help the communityreach its vision for healthy youth by reducing priority risk factors andenhancing protective factors.

• Anticipate possible objections or concerns, and plan how to address them.Can you think of reasons a site administrator might object to theimplementation of a tested, effective program? Take responses and allowtime for discussion. Be sure to review the topics outlined below.

- Resource issues: Administrators may have concerns about how tobalance staff time needed to implement the program with other prioritiesand activities. For example, let’s say you’re implementing a classroom-based social competency curriculum that requires two 30-minutesessions per week. With public schools under pressure to meetstatewide academic standards, administrators may be reluctant to takeclassroom time away from academics to deliver a social skills program.To address this concern, you could work with the principal to tie thesocial skills curriculum in with the academic curriculum standards.

Module 3

Notes

3—14

- Turf issues: Hopefully most turf issues have been resolvedearlier in the Communities That Care process. However, turfconflicts can still arise among collaborating agencies. Forexample, if a lead agency will be overseeing the implementationactivities at another agency, or if two agencies must shareresources to implement a program, you may encounter turfconflicts between administrators. Involving both agencies in theplanning process, and ensuring open communication betweenadministrators throughout implementation, can help resolveturf conflicts.

• It’s a good idea to have site administrators sign an agreementsolidifying their commitment to implementing the program with fidelity.The agreement should specify what responsibilities the administratorand agency as a whole will have in implementation.

Activity: Identifying site administrator support issuesTime: 15 minutes

Instructions:

1. Ask the groups to spend about 10 minutes discussing the questionson the Site Administrator Support Worksheet in relation to theprogram they were assigned earlier in this module. Each groupshould have one person fill out the worksheet, which will be the basisof a plan for securing site administrator support.

2. After about 10 minutes, debrief. What kinds of problems areexpected? How did the groups decide to address these problems?

Activity

Text Activity w/ Directions

3—15


Site Administrator Support Worksheet

Program:

Implementing agencies:

DirectionsAs a group, discuss any concerns you expect site administrators to have about implementing this program, and how you plan to address them. Have one person fill in your ideas in the tables below.

Plan to report at least one major concern and how you propose to address it to the larger group.

Use the results of your discussion as a basis for securing site administrator support after the training.

Turf conflicts

Other Expected issues Plans to address

Expected issues Plans to address

Expected issues Plans to address

Resources/staff commitment

Notes

Slide 3-

Module 3

3—16

Slide 3-10

Objective 2: Develop implementation plans.Once you have the site administrator’s support, work with this person toselect the staff members who will deliver the program. Keep theseconsiderations in mind as you select implementers:

Review the slide.

• Implementers need the appropriate qualifications and experience todeliver the program effectively and with fidelity. Ideally, they should havethe same minimum qualifications that the implementers in the originalprogram trials had. For example, if a tested, effective family therapyprogram used only master’s-level social workers in the trials, then yourcommunity should use only master’s-level social workers whenimplementing the program.

• It’s important to select implementers who support the program’sphilosophical approach. For example, a therapist trained in andcommitted to a specific therapeutic method may resist adopting a newapproach. Select implementers who either already agree with theprogram approach or are open to new approaches.

• It’s also important to consider implementers’ understanding of the targetaudience. This is key to their ability to recruit, communicate with andmotivate participants. This might mean selecting bilingual staff toconduct parent training, or selecting teachers who have a particularlygood rapport with youth for a school-based program.

Notes

3—17


Activity: Identifying implementer selection criteriaTime: 20 minutes

Instructions:

1. Have participants work in the same groups and with the same program asthe previous activity.

2. Ask them to fill in the Implementer Selection Criteria Worksheet using theprogram information in the Community Action Plan.

3. After about 15 minutes, ask each group to summarize its findings.

4. Debrief.

Implementers need to be carefully selected based on their qualifications,knowledge of the target audience and support for the program.Implementing agencies should use these worksheets as a guide forselecting and/or hiring implementation staff.

Activity

Text Activity-Blank

3—18

Implementer Selection Criteria WorksheetProgram:

Agencies:

1. How many implementers need to be involved?

2. What are the minimum qualifications for implementers?

Credentials:

Knowledge/experience:

3. What special qualities/skills should implementers have to help them reach the target audience?

4. List any potential philosophical conflicts or compatibility issues with the program:

5. Note any other factors to consider when selecting implementers:

Notes

Slide 3-11

3—19


Objective 2: Develop implementation plans.

Mental SetImagine that one day you come to work and learn that your office has switched toa new computer program that changes the way you do your job. Nobody tells youwhy or provides any information or training on using the new program. How doyou feel? [Answers might include apprehensive, resentful, frustrated.] What arethe chances that you’ll do a good job using the new program under thesecircumstances? [Chances are slim.] What could the organization have done toincrease the likelihood that you’ll use the new program effectively? [Answersmight include providing advance notice about the new program, explaining thereasons for the switch, providing training in the new program, etc.]

The support of implementers is a key to the successful implementation andmonitoring of tested, effective programs. Here are some things you can do toensure the support and commitment of implementers:

Review the slide.

• Involve implementers in the planning process so they don’t feel that aprogram has been pushed on them without their input. Implementers mayhave already been involved in assessing community needs and resourcesand/or selecting tested, effective programs, policies and practices toimplement. At the implementation stage, invite them to planning meetings sothey can be involved in developing implementation plans.

• Provide clear information about the program early on to help easeapprehension. Explain what the program is, why it was selected and how itwill benefit the community.

• Provide implementers with training in the program’s approach, methods andcomponents so they have the knowledge, skills and confidence to implementthe program effectively.

• Ask implementers to sign an agreement that outlines their responsibilities andconfirms their commitment to implementing with fidelity.

Notes

Slide 3-

Module 3

3—20

Slide 3-12

Objective 2: Develop implementation plans.A good way to get administrators, implementation staff and other stakeholders onboard with implementation is to schedule planning meetings for each program.Here are some things these meetings can accomplish:

Review the slide.

• Meetings can introduce the program to everyone involved and build support forimplementation. For example, you might hold an initial informational meeting topresent the program’s goals, why it was selected and how it is expected tobenefit the target audience as well as help the community achieve its vision forhealthy youth.

• Planning meetings may be needed to negotiate the logistics of training and implementation—for example, dates that sessions will be delivered,location, etc.

• Meetings should result in detailed implementation plans and signedparticipation agreements from everyone involved. We’ll discuss these in moredetail shortly.

You may also want to schedule periodic meetings during implementation, to helpyou monitor progress and address problems as they arise.

Optional Activity: Planning for pre-implementation meetingsTime: 10-15 minutes

Instructions:

Review the Meeting Planning Worksheet on the next page. If the groups are ready,give them about 10 minutes to draft a plan for their first implementation meeting. Ifthey are not ready, they can use this worksheet as a tool for planning meetingsafter the training. (Note: The Agenda section includes suggested agenda items.Participants can place a check mark next to items they plan to cover and write inadditional items as needed.)

Activity

Text Activity-Blank

3—21


Meeting Planning Worksheet

Program:

Meeting date:

Time:

Location:

Goals and objectives:

Who will lead the meeting?

Who will take minutes?

Invitees:

Agenda:

o Program overview

o Description of level of administrator commitment needed

o Implementation staff criteria

o Logistics related to program delivery and evaluation

o Signed agreements

o Other agenda items:

Notes

Slide 3-

Module 3

3—22

Slide 3-13

Objective 2: Develop implementation plans.Review the slide.

Tested, effective prevention programs require some form of training forimplementers. As I mentioned earlier, trainings are important because theyprovide implementers with the information, skills and confidence toimplement the program effectively and with fidelity.

The program developer or other experts in the program should provide thetraining. Find out if training is provided as part of the program you areimplementing or if you can contract with the developer or distributor toprovide training. Once you know who will be providing the training, work withthis person to determine the training content and logistics.

Notes

3—23


Slide 3-14

Objective 2: Develop implementation plans.Here are some practical considerations to keep in mind when planningimplementer trainings:

Review the slide.

• Timing is important. Be sure to schedule trainings after you hireimplementation staff. Also, don’t schedule the training too far in advance ofimplementation, to avoid having to train new staff as a result of turnover. Forexample, if you schedule a training for teachers in the spring or earlysummer, you can’t be sure all of those teachers will return to school in thefall. Also, schedule trainings at a time when implementers are most likely tobe able to attend. For example, avoid scheduling trainings during the winterholiday season or school vacation week.

• Arrange for an appropriate location and facilities for the training sessions.

• Consider the need for staff coverage while implementation staff areattending a training. For example, if a training for teachers is held on aschool day, the principal will need to arrange for substitutes. You can seewhy the support of site administrators is so important!

• Make plans for training new implementers who may come on board after theinitial training sessions. This could involve one-on-one technical assistance,or another group training.

The Training Planning Worksheet in your guide can help you plan the agendaand schedule for training events.

Review the Training Planning Worksheet on the next page. This is a toolparticipants can use after the training when they begin planning trainingsessions for implementers.

Program:

Agencies:

Training will be provided by:

Training topics:

Training schedule:

Session Topics to be covered Date Location

Plans for training new implementers:

Module #

Text Activity-Blank

3—24

Activity

Training Planning Worksheet

Notes

3—25


Objective 2: Develop implementation plans.The result of all these meetings and trainings should be a detailedimplementation plan that includes:

• the target audience and expected number of participants

• plans for recruiting participants

• the individuals who will coordinate and deliver the program, and eachperson’s responsibilities

• a schedule for delivering the program’s components—for example, datesfor workshops or classroom sessions.

This information will be based on the implementation outcomes you reviewedearlier, as well as any preliminary implementation plans outlined in yourCommunity Action Plan. For example, while an implementation outcome maycall for teachers to implement a program, the implementation plan will identifyexactly which teachers. Similarly, while an implementation outcome mayidentify the number of sessions to be delivered, the implementation plan willinclude a complete schedule for delivering each session or component.

Use the Implementation Planning Worksheet in your guide to help youdevelop implementation plans.

Review the worksheet with participants. They can use this as a tool fordeveloping implementation plans after the training.

Slide 3-15

Module 3

Text Activity-Blank

3—26

Implementation Planning WorksheetProgram:

Agencies:

Target audience:

Program outcome(s):

Program components to be implemented:

Plans for recruiting participants from the target audience:

Implementation team

Coordinator:

Implementers:

Activity

3—27


Imple

men

tati

onPl

anni

ngW

orks

heet

,co

nt’d

Impl

emen

tati

onSc

hedu

leR

esou

rces

Loca

tion

Expec

ted

atte

ndan

ceIm

ple

men

ter

Tim

e/D

urat

ion

Dat

eA

ctiv

ity

Notes

Slide 3-16

Module 3

3—28

Objective 3: Identify methods of monitoringimplementation for fidelity.So far in this module, we’ve discussed steps you can take to help ensurethat the tested, effective programs your community has selected areimplemented with fidelity. Now we’re going to discuss methods you canuse to evaluate implementation.

Review the slide.

This slide gives an overview of the evaluation process. When you selectedtested, effective programs to address your community’s priority risk andprotective factors, you also wrote desired outcomes to help you evaluatethe effectiveness of those programs. These outcomes are a key part ofyour evaluation design.

Evaluating implementation tells you how well the program is beingimplemented in your community.

Once the program is implemented, you’ll also evaluate participantoutcomes, which you wrote in Phase Four. Participant outcomes measurethe program’s effects on participants.

Every two years, you’ll evaluate community-level outcomes—changes inrisk factors, protective factors and problem behaviors throughout yourcommunity. You’ll use this data to modify the Community Action Plan as needed.

Notes

Slide 3-17

3—29


Objective 3: Identify methods of monitoringimplementation for fidelity.Evaluating implementation means monitoring each step in theimplementation process. Here are some reasons to evaluate implementation:

Review the slide.

• Evaluating implementation helps ensure that the program is implementedwith fidelity to the original design. It forces you to develop detailedimplementation plans and holds implementers accountable for followingthose plans.

• Evaluating implementation helps you identify and correct implementationproblems along the way, before they result in program failure.

• Many funders require that you document all program activities to showthat you did what they gave you money to do. Implementation evaluationprovides that documentation.

• Evaluating implementation helps you identify obstacles andimplementation problems, which can serve as “lessons learned” forreplicating the program in other parts of your community or implementingsimilar programs in the future.

• By monitoring implementation efforts, you can identify and celebrate earlyprogram successes.

Notes

Slide 3-18

Module 3

3—30

Objective 3: Identify methods of monitoringimplementation for fidelity.Here are some possible implementation-related problems that can have anegative impact on program effectiveness.

Review each item as you click it onto the screen.

For example, if only 5 parents attend a parent-training session

whose implementation outcome called for 15 parents to

attend, you may not be reaching the target audience in great

enough numbers to be effective at reducing levels of priority

risk factors in the community. What are some possible

reasons for low attendance in a situation like this? [Lack of

transportation, bad timing, ineffective recruitment.] By

monitoring implementation carefully, you can identify the

problem early on, find out the cause and take steps to correct

it. For example, perhaps the invitation letter that went out to

parents didn’t adequately describe how they might benefit

from the program. Or by following up with invitees who did not

attend, you might find that they lacked transportation or child

care to attend at the scheduled time.

Notes

3—31


For example, monitoring efforts might reveal that a

teacher has only delivered 1 session per month of a

classroom program that is intended to be delivered once

per week. What are some possible causes for this type

of problem? [The teacher may not have been provided

with a clear implementation plan or understood the

importance of delivering the program at full dosage. Or

he or she may not have enough available classroom

time to deliver the intended number of sessions.]

Working with the principal to ensure that teachers have

the support and resources required for high-fidelity

implementation may help correct problems like these.

Evaluating implementation may reveal that the

components are not being delivered with the methods

identified in the program design. For example, perhaps

an implementer of a parent-training program isn’t using

all of the skill-building activities identified in the original

program. What are some possible reasons for a problem

like this? [Inadequate training; implementer could be

uncomfortable using the prescribed techniques.] You

may need to provide booster sessions to reinforce the

delivery methods implementers learned at the initial

program training.

Activity: Brainstorming implementation problems and solutionsTime: 10 minutes

Instructions:

1. Ask participants if they can think of any other possible problems with implementation. Write their responses on easel paper. Be sure to discuss:

• how the problem could impact program effectiveness

• possible causes

• possible solutions.

2. Debrief.

Any number of problems can arise during the planning andimplementation process that can decrease the effectiveness of yourprogram. Evaluating implementation can help you identify and solveproblems before they result in the failure of your program.

Notes

Slide 3-19

Module 3

3—32

Objective 3: Identify methods of monitoringimplementation for fidelity.Review the slide.

As we’ve been discussing, what happens before a program is actuallyimplemented may affect the quality of implementation and, subsequently, theprogram’s effectiveness. The quality of planning meetings, implementertrainings and other pre-implementation activities can have a significantimpact on the quality of implementation. That’s why implementationevaluation should include monitoring planning meetings, implementertrainings and program delivery itself.

Slide 3-20

3—33


Objective 3: Identify methods of monitoring implementation for fidelity.Before we get into specific methods of evaluating implementation, keep in mind that many programs may providetools to help you evaluate implementation and effectiveness. You should find out what your programs offer beforedeveloping tools yourselves.

Here are some common methods of monitoring implementation and conducting program evaluation:

Review the slide.

• A lot of implementation information can be obtained through basic record keeping. For example, keepingattendance records and minutes of planning meetings can tell you whether the right people attended themeetings and what was accomplished. You can also have implementers keep logs of all program-relatedactivities to compare against the implementation plans we discussed earlier. Attendance logs can also tell youwhether the target audience is receiving a program at the intended dosage.

• Observing program delivery is a common method of collecting implementation information. Observation is agood way to identify problems with the implementer’s skills, understanding of the program methods and abilityto communicate effectively with participants. Observation should be systematic—observers should usechecklists or other tools to keep track of topics covered, techniques used and other information.

• Questionnaires for implementers can provide information about implementers’ perceptions of the quality oftraining they receive as well as self-reported implementation activities. Questionnaires for program participantscan help you evaluate their perceptions of the program’s quality and whether the implementer is communicatingeffectively with participants.

• One-on-one interviews or focus groups with implementers can help you gain valuable insight into the kinds ofobstacles to high-fidelity implementation faced by implementers.

Which data-collection methods you use to monitor different aspects of implementation depend on the kind ofinformation you need. Let’s look at the kinds of information you’ll want to collect about planning meetings, trainingsand program delivery, and discuss what the best methods for collecting that information might be.

Notes

Notes

Slide 3-21

Module 3

3—34

Objective 3: Identify methods of monitoringimplementation for fidelity.Review the slide.

For planning meetings, you’ll need to have a record of who attended, whofacilitated the meeting, the agenda and topics discussed and what decisionswere made. Having this information will help you document the efforts madeto involve all stakeholders in the planning process, as well as the historybehind implementation decisions.

What would be the best way to collect this information? [Attendance logs,agendas, meeting minutes, post-meeting questionnaires/evaluations.]

Slide 3-22

3—35


Objective 3: Identify methods of monitoring implementation for fidelity.These are some of the questions about the process of training program implementers that you might needanswered to assess the cause of implementation problems.


You may need to verify that all current implementers have received appropriate training. This can

help you identify causes for problems with program delivery. What do you think the best way to

track this information would be? [Keeping attendance records for all training events.]

If you run into problems with the quality of implementation among trained staff, you also might need

information about the quality of the training provided. For example, you might need to verify that the

training covered the program’s research base, approach and core components as well as specific

delivery methods. What do you think is the best way to collect this information? [Observing all

trainings with a checklist, reviewing training materials, self-report questionnaires for the trainer

and/or trainees.]

As we discussed earlier, it’s important that implementers leave trainings with the confidence

that they can deliver the program effectively, as well as the skills and motivation to do so.

What are some methods for collecting data like this? [Post-training evaluation forms, focus

groups and interviews.]

Notes

Notes

Slide 3-23

Module 3

3—36

Objective 3: Identify methods of monitoring implementationfor fidelity.Here are some questions you’ll need to ask about program implementation.


Keeping track of how the target audience is recruited for a

program can help you identify reasons for low levels of

participation. For example, you may need to assess whether

recruitment methods were culturally appropriate. What do you

think is the best way to collect this information? [Maintaining files

of all recruitment efforts—flyers, invitation letters, etc.]

It’s important to keep track of the number of participants in the

program, because this tells you whether your implementation

outcomes are being achieved. If the program involves several

sessions, you’ll want a record of which participants attended

each session. A drop in participation levels may signal an

implementation problem. What’s the best way to keep track of

participation levels? [Attendance logs.]

Notes

3—37


You’ll also need to keep track of when all program

components are delivered to ensure that the dosage

matches that of the original design. Tracking this information

can help you identify problems such as a program not being

delivered frequently enough or for long enough periods. What

are some ways to track this information? [Implementation

logs or questionnaires kept by implementers.]

As we’ve discussed, delivering the components in the

correct dosage doesn’t guarantee implementation fidelity.

You also need to monitor the quality of program delivery to

assess whether implementers are properly using the delivery

methods called for in the program design. If they aren’t, they

may need more training or motivation. What are some ways

to collect information about the quality of program delivery?

[Observing program sessions with a checklist, self-report

questionnaires from implementers.]

You may also need to assess whether implementers are

communicating effectively with the target audience. What are

some ways to assess this during implementation?

[Participant evaluations of the program sessions, focus

groups, interviews, observing program sessions.]

Let participants know that examples of an observation form, implementationlog and training evaluation questionnaire are provided on the next threepages in their guides. If time permits, review the examples before moving on.

Module #

3—38

Module 3

Sample Observation Form

Program:

Teacher:

Date of Session Observed:

Time/Duration of Session:

1. What topic(s) did this session cover?

2. Did the teacher follow the lesson plan? Yes No

If not, what parts were not covered?

3. Did the teacher enhance or add to the lesson plan in any way? Yes No

If yes, please describe.

4. What activities were used to deliver this lesson?

5. What percentage of time was spent on the following techniques:

Lecture _______

Group discussion _______

Practice _______

6. Please describe any implementation problems you noticed that are not covered elsewhere on this form.

Note:This is an example of an observation feedback form for a classroom-based program. Use this as a guide todevelop program session observation forms that are appropriate for the types of programs you areimplementing. You may want to include checklists of specific topics or activities that should be covered in aparticular session. You may also want to work with the program developers to develop these forms.

Activity

3—39


Sam

ple

Imple

men

tati

onLo

gD

irect

ions

Use

this

form

tolo

gal

lofy

ouri

mpl

emen

tatio

nac

tiviti

es.I

fyou

have

devi

ated

from

the

orig

inal

impl

emen

tatio

npl

anin

any

way

,ple

ase

expl

ain

inth

e“c

omm

ents

”co

lum

n.

Ple

ase

retu

rnco

mp

lete

dfo

rmto

Jane

Do

e,P

rog

ram

Co

ord

inat

or,

on

the

first

of

each

mo

nth

.

Prog

ram

:

Imple

men

ter:

Co

mm

ents

Mat

eria

lsU

sed

Loca

tion

Att

end

ance

Tim

e/D

urat

ion

Dat

eA

ctiv

ity

Activity

3—40

Sample Training Evaluation FormProgram:

Training Date:

1. Overall, how would you rate this training?

1 2 3 4 5

Poor Excellent

2. How well do you feel you understand the program’s goals and theoretical approach?

1 2 3 4 5

Not at all Somewhat Very well

3. How useful were the training activities in helping you develop the skills needed to implementthis program?

1 2 3 4 5

Not useful Somewhat useful Very useful

4. How confident do you feel in your ability to implement this program with fidelity to the original design?

1 2 3 4 5

Not confident Somewhat confident Very confident

5. What additional support or training do you think you need to successfully implement this program?

6. Any additional comments on this training or your ability to implement this program?

Notes

3—41


Objective 3: Identify methods of monitoring implementationfor fidelity.As you can see, implementation evaluation methods are fairly straightforward. Alot of implementation information can be collected through careful recordkeeping and implementation logs. Even so, you should develop a formalimplementation evaluation plan for each program that identifies:

• the key questions you hope to answer by collecting implementation data

• the methods you will use to collect that data

• who will be responsible for collecting the data and who will be responsiblefor acting on it. For example, while implementers may be responsible forcollecting most implementation data, you also need to decide to whom that data should be reported, and who will analyze the data and makedecisions based on the results. For example, you may want to create animplementation-evaluation task force to oversee the implementation program.

• a schedule for reporting deadlines. This is important because you want tomake sure you collect implementation data at regular intervals during theimplementation process, so that you can catch any problems before theyresult in program failure.

Activity: Implementation evaluation designTime: 20 minutes

Instructions:

Have participants work in their groups to fill in as much as they can of theImplementation Evaluation Worksheet in their guides. After about 15 minutes,check for understanding by asking a few groups to report their chosen methodsfor one of the implementation steps. Remind participants that this worksheetshould be filled out for each new program being implemented.

Slide 3-24

Program: ________________________________________________________________________

Agencies: ________________________________________________________________________

Pre-implementation planning

Implementer trainings

Module #

Text Activity-Blank

3—42

Module 3

Implementation Evaluation Worksheet

Items to evaluate Methods Who will collect Time line


Activity

Text Activity-Blank

3—43


Implementation

Implementation Evaluation Worksheet, cont’d


Notes

Slide 3-25

Module 3

3—44

Review the next steps, as outlined on the slide. Explain that task forces willbe created in Module 5.

Notes

Slide 3-26

3—45


Review the modules and milestones. Tell participants that in the next moduleyou will discuss evaluating participant outcomes—changes in participants’knowledge, attitude, skills and behavior as a result of the programs thecommunity implements.

Documents

Community Plan Implementation Training - SDRGsdrg.org/ctcresource/community plan implementation training/trainer... · Trainer’s Guide (90 minutes) Ensuring High-Fidelity Implementation