CAPP EvaluationAugust 2011 Update

Jane Powers, Amanda Purington, Jenny Jane Powers, Amanda Purington, Jenny PariseParise

ACT for Youth Center of ExcellenceACT for Youth Center of Excellence

OverviewOverview•Review Evaluation Plan: Documenting

EBP Implementation

•Ways to improve documentation

•Revisions to the Tools

•Next Steps

CAPP Initiative CAPP Initiative

Goal 1: Promote healthy sexual behaviors and reduce the practice of risky sexual behaviors among adolescents

Core Strategy 1: Provide Provide comprehensive, age appropriate, comprehensive, age appropriate, evidence-based and medically evidence-based and medically accurate sexuality education to accurate sexuality education to promote healthy sexual behaviorspromote healthy sexual behaviors

We know a lot about what works to We know a lot about what works to prevent teen pregnancyprevent teen pregnancy

EBP

•Increase age of first intercourse

•Increase use of condoms

•Decrease # sexual partners

•Decrease frequency of sex

DecreaseTeen Pregnancy

PromoteAdol SexualHealth

What does it take to What does it take to implement EBPs in real implement EBPs in real

world settings???world settings???

Research to PracticeResearch to Practice While many EBPs have yielded positive While many EBPs have yielded positive

outcomes in research settings, the record outcomes in research settings, the record at the local level of “practice” is mixed at the local level of “practice” is mixed (Wandersman, 2009; Lesesne et al, 2008; (Wandersman, 2009; Lesesne et al, 2008; Fixsen, 2005). Fixsen, 2005).

What do we know about What do we know about Implementation?Implementation?

• Implementation influences program outcomesImplementation influences program outcomes

• If EBPs are not implemented with fidelity and If EBPs are not implemented with fidelity and quality, not likely to result in outcomes observed quality, not likely to result in outcomes observed in researchin research

• Achieving good implementation increases Achieving good implementation increases chances of program success and stronger chances of program success and stronger benefits for participantsbenefits for participants

(Durlak and DuPre, 2008)(Durlak and DuPre, 2008)

Factors Affecting Factors Affecting ImplementationImplementation

• Community LevelCommunity Level

• Organizational Capacity Organizational Capacity

• Program CharacteristicsProgram Characteristics

• Training and TATraining and TA

• Facilitator CharacteristicsFacilitator Characteristics

CAPPCAPPAn Opportunity to Learn!!!An Opportunity to Learn!!!

Opportunity to learn:Opportunity to learn:

• How do EBPs work in different settings?How do EBPs work in different settings?

• How do EBPs work with diverse youth How do EBPs work with diverse youth populations?populations?

• What adaptations are needed to make EBPs What adaptations are needed to make EBPs work more effectively?work more effectively?

• Are EBPs successful in getting to outcomes?Are EBPs successful in getting to outcomes?

Documenting Implementation is Documenting Implementation is critical!critical!

• Helps us learn how the intervention Helps us learn how the intervention was deliveredwas delivered

• Helps us understand challenges facedHelps us understand challenges faced

• Helps us learn about what does and Helps us learn about what does and does not work re implementing EBPsdoes not work re implementing EBPs

Tools to evaluate Tools to evaluate ImplementationImplementation

• Fidelity Checklist: document what you Fidelity Checklist: document what you did, adaptations made, did, adaptations made, successes/challengessuccesses/challenges

• Attendance Record: who, where, Attendance Record: who, where, when, how muchwhen, how much

Role of the Fidelity Checklists

Role of the Fidelity Role of the Fidelity ChecklistsChecklists

• Document:Document:– What you didWhat you did– TimingTiming– Adaptations made and whyAdaptations made and why– Challenges and successesChallenges and successes

Role of the Fidelity Role of the Fidelity ChecklistsChecklists

Help you Prepare!!Help you Prepare!!

Using Fidelity Checklist to PlanUsing Fidelity Checklist to Plan

• Review activitiesReview activities

• What equipment is needed?What equipment is needed?

• TimingTiming

• Anticipating issuesAnticipating issues

• Questions that might come upQuestions that might come up

• Prepare for next cyclePrepare for next cycle

What have we learned?What have we learned?

Ways to Improve Ways to Improve DocumentationDocumentation

1. Provide More Details!1. Provide More Details!

– WHAT WHAT was changed?was changed?

– WHYWHY was it changed?was it changed?

GREAT exampleGREAT example

“ “We took the entire class period to implement and We took the entire class period to implement and process this activity. Although it is timed at 15 process this activity. Although it is timed at 15 minutes, it took ten minutes alone to break the minutes, it took ten minutes alone to break the class of 32 students into groups, hand out cards class of 32 students into groups, hand out cards and explain the activity. We also needed more and explain the activity. We also needed more time to allow groups to respond as we physically time to allow groups to respond as we physically had more groups than if we were working in a had more groups than if we were working in a small group setting.”small group setting.”

Good exampleGood example““Due to the age of the participants and the Due to the age of the participants and the material in the Robert Townsend video clip, the material in the Robert Townsend video clip, the teacher did not find it to be appropriate for the teacher did not find it to be appropriate for the classroom. Instead we discussed the information classroom. Instead we discussed the information in the clip and used a short role-playing exercise in the clip and used a short role-playing exercise to make the information clear.”to make the information clear.”

– Why wasn’t it appropriate?Why wasn’t it appropriate?

– What was the role-playing exercise?What was the role-playing exercise?

Need more detailsNeed more details

““ran out of time”ran out of time”•Tells us WHY but not WHAT was changedTells us WHY but not WHAT was changed

•Was the activity shortened (if so, how?) or Was the activity shortened (if so, how?) or skipped altogether? skipped altogether?

““switched the order of these activities”switched the order of these activities”•Tells us WHAT the change was but not WHY Tells us WHAT the change was but not WHY

it was madeit was made

2. Document ALL 2. Document ALL adaptationsadaptations

Was Activity Carried Out According to Direction in the Facilitator’s Curriculum?

•Leave something outLeave something out

•Change orderChange order

•Add new contentAdd new content

•Shorten activityShorten activity

•Substitute activitySubstitute activity

•Modify activityModify activity

2. Document ALL adaptations 2. Document ALL adaptations cont.cont.

– Include info on pre-approved adaptationsInclude info on pre-approved adaptations– Were NO adaptations at all made??Were NO adaptations at all made??– ExampleExample

• NO adaptations madeNO adaptations made

• General Comment: “General Comment: “It would have been nice It would have been nice to have more time for each activity/moduleto have more time for each activity/module””

– Were activities completed according to the Were activities completed according to the curriculum?curriculum?– Were some activities omitted or shortened?Were some activities omitted or shortened?

3. Complete the Fidelity 3. Complete the Fidelity Checklist right away!Checklist right away!

– Capture this information when the Capture this information when the experience of the module and the experience of the module and the activities are fresh in your mindactivities are fresh in your mind

4. Complete the right number of 4. Complete the right number of toolstools

One One set of tools per cycle per set of set of tools per cycle per set of participantsparticipants

Class 1

Class 3

Class 4

Class 5

Fidelity Checklist Attendance Record

Class 2 Fidelity Checklist Attendance Record

Fidelity Checklist Attendance Record

Fidelity Checklist Attendance Record

Fidelity Checklist Attendance Record

Teacher 1

Teacher 2

Teacher 3

Fidelity Checklist Attendance Record

Common AdaptationsCommon Adaptations

• Common adaptations include:Common adaptations include:– Updating videos, or using something more Updating videos, or using something more

relevant in place of videosrelevant in place of videos– Changes to accommodate timing issuesChanges to accommodate timing issues– Changing language/names to be more inclusiveChanging language/names to be more inclusive

• How can the COE support these common How can the COE support these common adaptations?adaptations?– Providing/linking to updates on medical Providing/linking to updates on medical

informationinformation– Identifying alternative videos (updated)Identifying alternative videos (updated)– Developing alternatives to the condom Developing alternatives to the condom

demonstrationdemonstration

Documentation DetailsDocumentation Details• Block names of individual participantsBlock names of individual participants• Do not use acronyms – spell these outDo not use acronyms – spell these out• Write legiblyWrite legibly• Review your documentation for typos Review your documentation for typos

beforehandbeforehand• Bundle all evaluation materials for each Bundle all evaluation materials for each

cycle (fidelity checklists and attendance cycle (fidelity checklists and attendance records) and mail to:records) and mail to:

Amy BreeseAmy BreeseCornell UniversityCornell University

ACT for Youth Center of ExcellenceACT for Youth Center of ExcellenceBeebe HallBeebe Hall

Ithaca, NY 14853Ithaca, NY 14853

Submit tools electronicallySubmit tools electronically

• Save the fidelity check list and Save the fidelity check list and attendance record under a different attendance record under a different name and input the informationname and input the information

• Preserve the template for future usePreserve the template for future use

• Send the documents to Amanda Send the documents to Amanda Purington (Purington (ald17@cornell.eduald17@cornell.edu))

Revisions to the Fidelity Revisions to the Fidelity ChecklistsChecklists

1.1. Clarified the adaptations description field:Clarified the adaptations description field:““If Changed, If Changed, WHATWHAT was changed and was changed and WHYWHY? Please ? Please

be specific: describe things you left out, added, be specific: describe things you left out, added, or changed and WHY.”or changed and WHY.”

2.2. Added “Were changes (if any) pre-approved?” Added “Were changes (if any) pre-approved?”

3.3. Added a comments section after each Added a comments section after each module:module:

““Please use this space if you have comments on this Please use this space if you have comments on this module or any of its activities:” module or any of its activities:”

Revisions to the Attendance Revisions to the Attendance RecordRecord1. Added “Age” column for participants

Next StepsNext Steps• Additional tools:Additional tools:

– Participant Feedback FormParticipant Feedback Form– Facilitator Feedback ToolsFacilitator Feedback Tools– Pre/Post TestsPre/Post Tests– Other needs?Other needs?

• Tools and presentations on Tools and presentations on www.actforyouth.net - Search for - Search for “CAPP” “CAPP” in the site search box at the top of the in the site search box at the top of the home page.home page.

• Revise Tools for Year 2 Revise Tools for Year 2

Contact InformationContact Information

Amanda Purington: Amanda Purington: ald17@cornell.edu 607-255-1861 607-255-1861

oror

Jane Powers: Jane Powers: jlp5@cornell.edu

607-255-3993607-255-3993