Training and Technical Assistance Supporting Evidence-based Prevention and Intervention, and Juvenile Justice System Improvement September 9, 2014 Evidence-Based

Training and Technical Assistance Supporting Evidence-based Prevention and Intervention, and Juvenile Justice System Improvement September 9, 2014 Evidence-Based Prevention and Intervention Support Center (EPISCenter)

1.PCCDs Approach to Prevention 2.Implementation Quality & Monitoring 3.The Role of Technical Assistance Strategic Coordination PAYS, CTC, EBPs, EBIs Systems for Data Collection & Reporting Virtual Technical Assistance 4.Outcomes of a State Agency & University Partnership Presentation Outline 2

1.Using data to identify community risk and needs (CTC) 2.Identifying local services to match those needs (CTC) 3.Assessing additional programmatic needs; CTC, EBPs, fit & feasibility 4.High fidelity and quality implementation of EBPs 5.Roll-up and tracking of implementation and outcomes data 6.Estimating return-on-investment/cost-benefit analysis 7.Supporting sustainability from seed grants to post-funding 8.Developing statewide capacity for prevention across all levels Key Aspects and Activities 3

Support to Community Prevention Coalitions Improve Quality of Local Innovative Programs and Practices Support to Evidence-based Prevention & Intervention Programs Multi-Agency Steering Committee (Justice, Welfare, Education, Health) Intermediary and State-level Prevention Support System EPISCenter: Initiatives and Goals Resource Center Mission: To support the proliferation of quality prevention and intervention programs aimed at promoting positive youth development and preventing violence, delinquency, substance abuse and other problem behaviors in children and adolescents. Implementation Quality Broad-scale Dissemination Long-term Sustainability 4

PART I PCCDS APPROACH TO PREVENTION: A RESEARCH-INFORMED STRATEGY

Moving From Prevention Science... Problem Response Define the Problem Identify Risk & Protective Factors Develop & Test Interventions Implement & Evaluate Programs 6

We know a great deal about how youth problems develop, and how to effectively prevent them (& reduce prevalence) Known risk & protective factors Multiple domains of influence Community, family, school, peer, individual Multifinality and equifinality A public health approach to public safety Different trajectories (early vs. late starters) Criminogenic impact of intervention Prevention Science Background 7

Programs/services can be placed along a continuum of confidence based on their evidence or theory How confident are we that this program or practice is a good use of resources AND improves outcomes for children and families? Very Confident Evidence-based This program has been rigorously evaluated and shown to work Research-based This program is based on sound theory informed by research Promising Approaches We really think this will work but we need time to prove it Best Practices Weve done it and we like it EFFECTIVEHARMFUL Iatrogenic (Harmful) This program has been rigorously evaluated and shown to be harmful Ineffective This program has been evaluated and shown to have no positive or negative effect PromisingIneffective unknown Very Confident *Bumbarger & Rhoades, 2012 The Continuum of Confidence 8

To Prevention Service Provide Technical Assistance Set & Collect Performance Measures Monitor Quality of Program Implementation Assess Public Health Impact Response 9

We also know a great deal about factors that influence quality of program implementation Champions Stakeholder buy-in Program credibility Sites capacity for implementation Strong site coordinators Training in the program model Implementer confidence in delivery Proactive-ness and responsiveness of technical assistance Implementation Science Background 10

Prevent dependency, delinquency, and ATOD use to the greatest degree possible (primary prevention) Intervene effectively with youth for whom primary prevention is not sufficient Allow communities the flexibility to select strategies that best meet local needs Create community-level infrastructure for strategic prevention planning and coordination Provide accountability and use scarce resources efficiently ULTIMATELY. Pennsylvanias EBP Dissemination Model To move the needle on key indicators of (behavioral) health at the POPULATION level PA Youth Survey + CTC prev. infrastructure & prioritize RPFs + Targeted support for selected EBPs + Technical assistance to promote quality, dissemination, and sustainability = Population level impact 1989 1994 1998/2001 2001/2008 2003/2005 11

Risk Factors (Causes) Adolescent Problem Behaviors (Outcomes) Equifinality Key Point: There are multiple routes to address a targeted behavior problem. Multifinality Key Point: When we target risk factors (the underlying causes of behavior), we address more than one problem behavior. 12

Risk Factors (Causes) Adolescent Problem Behaviors (Outcomes) Implications: State and local agencies interested in efficiently and effectively addressing youth problem behaviors should collectively focus on underlying risk and protective factors that drive common and shared problems. 13

Measures risk and protective factors across multiple domains. A voluntary survey conducted in schools every other year for youth in 6 th, 8 th, 10 th, and 12 th grades. Adapted from the Communities That Care Youth Survey, additional questions added on gambling, prescription drug abuse, other antisocial behaviors, and experience of trauma and grief. All CTC Sites are essentially required to use it, and many additional schools volunteer to participate. 2013 PAYS: 200,000+ youth, 335 school districts, 70 other schools The Pennsylvania Youth Survey (PAYS): PAs Essential Tool for Prevention Planning 14

Creating Fertile Ground for EBPs: The Role of PAYS in Data-Informed Prevention Planning (The Communities That Care model) Collect local data on risk and protective factors Use data to identify priorities Select and implement evidence- based program that targets those factors Re-assess risk and protective factors Form local coalition of key stakeholders Leads to community synergy and focused resource allocation 15

A Review of Research Findings on PA Coalitions Connection to coalitions increases likelihood of program sustainability 2-yrs after funding ends. CTC more likely than non-CTC to be sustained. Connection to CTC coalitions reduces likelihood of making program adaptations over time (4-yr longitudinal study) Knowledge of the CTC model significantly predicts coalition providing downstream support for evidence-based programming. EBP implementers report CTC assistance in: Mobilizing the community Supporting actual prevention efforts Promoting evidence-based programming Assisting in impactevaluation 16

5 year Longitudinal Study of PA Youth % Change of CTC/EBP Youth Over Comparison Group 419 age-grade cohorts over a 5-year period Youth in CTC communities with EBPs: Lower rates of delinquency Greater resistance to negative peer influence Stronger school engagement Better academic achievement Feinberg, M.E., Jones, D., Greenberg, M. T., Osgood, W. D., & Bontempo, D. (2010). Effects of the Communities that Care model in Pennsylvania on change in adolescent risk and problem behaviors. Prevention Science, 11, 163-171. Delinquency Negative Peer Influence Academic Performance School Engagement 17

Impact on Juvenile Court Placement Rates: Comparison of Placement Rates for Counties* With and Without an EBI Bumbarger, B. K., Moore, J., & Rhoades, B. (2010). Impact of evidence-based interventions on delinquency placement rates. Presentation at 2011 Society for Prevention Research annual meeting. Washington, DC. 18

Evidence-based Programs are a wise investment of state resources Communities with EBPs embedded in the context of CTC have lower levels of delinquency and youth drug use * EBPs in PA produce an overall return of $5 for every $1 invested a statewide return measured in hundreds of millions ** Conservative estimates of the CTC model demonstrate a $5 return, more realistic estimates indicate a $10 return*** The Big Picture: * Feinberg, M.E., Jones, D., Greenberg, M. T., Osgood, W. D., & Bontempo, D. (2010). Effects of the Communities that Care model in Pennsylvania on change in adolescent risk and problem behaviors. Prevention Science, 11, 163-171. ** Jones, D., Bumbarger, B., Greenberg, M., Greenwood, P., and Kyler, S. (2008). The Economic Return on PCCDs Investment in Research-based Programs: A cost-benefit assessment of delinquency prevention in Pennsylvania. Prevention Research Center, Penn State University. *** Kuklinski, M. R., Briney, J. S., Hawkins, J. D., & Catalano, R. (2012). Cost-benefit analysis of Communities that Care outcomes at eighth grade. Prevention Science, 13, 150-61. 19

PART II IMPLEMENTATION QUALITY & MONITORING

Deterrent effect with a strong research design Sustained effect Multiple site replication Basis for Evidence in EBPs 21

When communities tweak a program to suit their own preferences or circumstances, they wind up with a different program whose effectiveness is unknown. Blueprints for Healthy Youth Development www.colorado.edu/cspv/blueprints ? 22 Deviating from Program Design

Berkel, C., Mauricio, A. M., Schoenfelder, E., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12(1), 23-33 The Connection Between Implementation & Outcomes 23

Unforeseen challenges (time, resources, population access) Lack understanding of the programs underlying theory Implementers lack necessary skills Programs that are not user friendly Lack of administrator support Status quo 24 Barriers to Fidelity

Based in understanding of programs theory of change/logic model Data collection Process measures (e.g., # of lessons delivered) Outcome measures (e.g., reduced antisocial attitudes) Implementer & participant feedback (e.g., how well did Session 6 go?) Use of valid, reliable survey tools (developmentally appropriate, actually measures program targets) Observation of program delivery By developer, or certified trainer/implementer Implementer & administrator review and reflection on data Adjustments to implementation made accordingly Elements of Implementation Monitoring 25

Implementation Monitoring Example 26

PART III THE ROLE OF TECHNICAL ASSISTANCE 27

Support to Community Prevention Coalitions Improve Quality of Local Innovative Programs and Practices Support to Evidence-based Prevention & Intervention Programs Multi-Agency Steering Committee (Justice, Welfare, Education, Health) Intermediary and State-level Prevention Support System EPISCenter: Initiatives and Goals Mission: To support the proliferation of quality prevention and intervention programs aimed at promoting positive youth development and preventing violence, delinquency, substance abuse and other problem behaviors in children and adolescents. Implementation Quality Broad-scale Dissemination Long-term Sustainability 28

Methods of Technical Assistance Strategic Plan & Outreach Resource development, distribution Fact sheets, how-to guides, webpages Trainings In-person, webinars, YouTube videos Networking meetings (quarterly) CTC & program-specific, cross-over meetings Connect with, learn from, and problem-solve with peers On-site consultations and implementation plan development Training, fidelity, outcome measurement, implementation, connection to coalition 29

STRATEGIC COORDINATION

Technical Assistance Cross-Systems Engagement Cross-Systems Engagement Policy Recommendations Research State Agencies Cross-agency coordination System-level barriers Gap analysis Program Developer Programmatic barriers to implementation/ sustainability PA-based Trainings Group discounts Implementing Site Recruitment Data collection Local stakeholder buy-in 31

Strategic Connections 32

Strategic Connections DDAP trainings in CTC needs assessment training For Single County Authorities to assess local program and service needs Connecting with Intermediate Units to discuss programmatic needs and availability of PCCD funding for EBPs Virtual web-meetings with PA stakeholders to review evidence- based intervention utilization and outcomes data, including JCJC, OMHSAS, CJJT&R, JJSES Stage 3 Discussions with developers of evidence-based programs that may meet PA needs identified by gap analysis; fit and feasibility EPISCenter prominently featured in the Prevention Research Centers strategic plan Consultation to other states on development of state-level infrastructure for prevention ( PA is a model!) 33

PA Infrastructure for Efficiencies: SFP 10-14 Training & Quality Assurance Trainer cadre trained to train up SFP facilitators and observers Improved turn-around time for scheduling training Decreased PA funds going to out-of-state trainer travel expenses Quality assurance designees Improved turn-around time for scheduling QA visit, and for receiving QA feedback and certification letter IYS Training & Curriculum Discounts In-state IYS planned training dates Improved ability to plan for training due to availability Increased access to timely training Decreased out-of-state travel expense for grantees Curriculum discounts to PCCD grantees EPIS as ordering & invoicing hub Scaled discounts according to quantity of curriculum ordered Decreased PCCD cost for materials Strategic Connections 34

Gap Analysis for PCCD: Phase I: Review state-level PAYS and Juvenile Justice data to identify risk & needs (PAYS 2011, 2013, Disposition 2012, Recidivism 2007-09). Phase II: Identify programs that address PA needs, determine level of evidence in program effectiveness, recommend programs for PA to support (what remains are true gaps). Phase III: Drill-down to county-level indicators of risk using PAYS and JJ data, draw in additional data from other sources (Dept. of Health, PDE, etc.), identify strengths (positive indicators, coalition presence, prior funding), recommend programs and strategies to address youth needs. 6 person team comprised of EPIS representation across CTC, prevention, intervention, and juvenile justice staff. Phase I and II presentation and program recommendations are posted online: http://episcenter.psu.edu/gaps Strategic Development 35

Cross-pollinating across CTC and SPEP: Raising awareness in community prevention coalitions of the JJSES activities, SPEP SPEP focus on quality improvement has implications for all youth-serving programs More widespread use of the YLS will result in coalitions ability to use that data in conjunction with PAYS to determine their youths risks and needs Strengthening the role and connection between local prevention efforts and local juvenile justice efforts Strategic Development 36

SUPPORT FOR THE PAYS

Presentations & Trainings PAYS 101 & 2013 Webinars PA Safe Schools Conference Commonwealth Prevention Alliance (CPA) CTC Regional Meetings Resources Short, online tutorial videos Guide for interpreting data Templates for sharing community data PDE Guide & Workbook PAYS 2013 Community/district recruitment Advisory Group Beyond 101: Advanced topics Using public health model Prevention planning Grant writing Connecting to coalitions/community Sustainability Support for Informed Use of the PAYS 38

Support for Informed Use of the PAYS 39

Support for Informed Use of the PAYS 40

41 Support for Informed Use of the PAYS

SUPPORT FOR CTC

Communities That Care (CTC) Drug-Free Communities (DFC) Strategic Prevention Framework (SPF) Integrated Services Plan Hybrid models Support to Community Prevention Coalitions Improve Quality of Local Innovative Programs and Practices Support to Evidence-based Prevention & Intervention Programs EPISCenter Initiative Areas 43

Creating Fertile Ground for EBPs Risk-focused Prevention Planning (the Communities That Care model) Collect local data on risk and protective factors Use data to identify priorities Select and implement evidence- based program that targets those factors Re-assess risk and protective factors Form local coalition of key stakeholders Leads to community synergy and focused resource allocation 44

Support for the Communities That Care Process 45

New grantee orientation Mobilizer training Milestones & Benchmarks assessment with the Board (grant start, middle, end) Co-development of implementation plan On-site trainings of mobilizer, board, and workgroups in CTC process Quarterly networking meetings (regional NW, SW, C, NE, SE) Quarterly site visits Monthly phone consults between TA provider and mobilizer The PCCD CTC Grantee: TA & Milestones 46

Assessing & Supporting Community Coalitions Web-based data collection from CTC board members Board membership, leadership, relationships, work style CTC process Programs implemented Barriers experienced Technical assistance accessed Provide feedback to sites on coalition functioning Summary report to TA consultant Report presented to CTC site Used for strategic planning 47

Virtual Technical Assistance: CTC Resources 48

SUPPORT FOR EBPs

ART - Aggression Replacement Training BBBS - Big Brothers Big Sisters IYS - Incredible Years (Parenting; Basic & Advanced) IYS Incredible Years (Youth; Classroom & Small Group) LST - Life Skills Training OBPP Olweus Bullying Prevention Program PATHS - Promoting Alternative Thinking Strategies PTNDA - Project Toward No Drug Abuse SFP 10-14 Strengthening Families Program 10-14 Support to Community Prevention Coalitions Improve Quality of Local Innovative Programs and Practices Support to Evidence-based Prevention & Intervention Programs EPISCenters Four Focal Initiatives 50

Functional Family Therapy (FFT) Multisystemic Therapy (MST) Multi-dimensional Treatment Foster Care (MTFC) Support to Community Prevention Coalitions Improve Quality of Local Innovative Programs and Practices Support to Evidence-based Prevention & Intervention Programs EPISCenters Four Focal Initiatives 51

New grantee orientation Initial site visit by Prevention Coordinator Co-development of implementation plan Spring and fall site visits, and site ratings Quarterly networking meetings Quality assurance visit, rating, and feedback by program developer Near-end of grant Outcomes Report Ongoing quarterly reporting of process & outcomes data The PCCD EBP Grantee: TA & Milestones 52

General and program-specific capacity for evidence-based programming Using data-informed decision making strategies Understanding the program components, underlying theory Coaching in implementation fidelity (general, program-specific, lesson- or component-specific) Identifying participant recruitment and retention strategies for site coordinators Supporting implementation and outcomes monitoring; data collection and evaluation Content and Skill Areas for Technical Assistance 53

General and program-specific capacity for evidence-based programming Developing sustainability strategies; education and support in other funding streams, needs-based budget process/planning, medical assistance funding, program transfer to local infrastructure (e.g., school) Connecting with other learning community members Facilitating communication with developers, researchers Building stakeholder buy-in, meaningful reporting on outcomes Content and Skill Areas for Technical Assistance 54

Virtual Technical Assistance: EBP Resources 55

SYSTEMS FOR DATA COLLECTION AND OUTCOMES REPORTING

Spreadsheet Tools PCCD-funded prevention programs (ART, BBBS, IYS, LST, OBPP, PATHS, SFP 10-14, PTNDA) INSPIRE Intervention Programs (FFT, MST, MTFC) Same goals: Generating process & outcome data, monitoring Increasing reliability & validity of data Reducing data reporting burden Increasing usability of data by variety of stakeholders Standardized Data Collection and Support 57

The Next Generation of EBP Spreadsheets 58

Quarterly and Annual INSPIRE Reporting 59

VIRTUAL TECHNICAL ASSISTANCE

Virtual Technical Assistance: YouTube & Webinars 61

Virtual Technical Assistance: Outreach 62

Virtual Technical Assistance: Outreach 63

PART IV OUTCOMES OF STATE AGENCY & UNIVERSITY PARTNERSHIP

History of Research-Based Prevention in Pennsylvania 1994: Key state leaders introduce Communities that Care (CTC) in PA Spearheaded by Pennsylvania Commission on Crime & Delinquency (PCCD) and Juvenile Court Judges Commission 1994-2002: Initiation of CTC funding by PCCD 16 cycles of CTC model introduced in ~120 communities 1996: PCCD Co-funding of research for Blueprints programs 1998: Process Study of CTC conducted by Prevention Research Center Resulted in creation of statewide TA infrastructure to support CTC Formalized connection between CTC and EBP Initiative 1998: Initiation of Evidence-based Program Initiative by PCCD 10 cycles of EBPs funded over 13 years, resulting in ~200 EBPs 2001: Narrowed list of supported EBPs, aka PA Blueprints 2008: PCCD created Resource Center for Evidence-Based Prevention and Intervention Programs and Practices Multi-agency Steering Committee Representing Justice, Welfare, Education, and Health 65

Development and support of communities of practice Including common public health language in RFAs Statewide surveillance system (PAYS) Focus on underlying causal mechanisms vs. narrowly defined behavioral outcomes Community coalitions as local prevention infrastructure Ongoing monitoring of implementation Requires tools, skills, and motivation Policy and Practice Innovations Bumbarger, B. K., & Campbell, E. M. (2011). A state agency-university partnership for translational research and the dissemination of evidence-based prevention and intervention. Administration and Policy in Mental Health and Mental Health Services Research, 39, (4), p. 268-277. 66

From Lists to Improved Public Health Synthesis and translation of research to practice, (and practice to research) EBP selection, dissemination, and uptake Ensuring sufficient implementation quality and fidelity Understanding adaptation and preventing program drift Measuring and monitoring implementation and outcomes Policy, systems, and infrastructure barriers Coordination across multiple programs and developmental stages Sustainability in the absence of a prevention infrastructure Bumbarger, B. and Perkins, D. (2008). After Randomized Trials: Issues related to dissemination of evidence-based interventions. Journal of Childrens Services,3(2), 53-61. Bumbarger, B., Perkins, D., and Greenberg, M. (2009). Taking Effective Prevention to Scale. In B. Doll, W. Pfohl, & J. Yoon (Eds.) Handbook of Youth Prevention Science. New York: Routledge. 67

68 From the Field to the Research Journals

Collaborative Policy Innovators: James Anderson Mike Pennington Linda Rosenberg Keith Snyder Clay Yeager Investigators and Authors: Brian Bumbarger Mark Greenberg Mark Feinberg Brittany Rhoades Louis Brown Wayne Osgood Michael Cleveland Damon Jones Jennifer Sartorious Julia Moore Brendan Gomez Richard Puddy Stephanie Bradley Elizabeth Campbell The EPISCenter and research described here are supported by grants from the Pennsylvania Commission on Crime and Delinquency. Special thanks to the staff of the Office of Juvenile Justice and Delinquency Prevention (OJJDP). 69

Documents

Training and Technical Assistance Supporting Evidence-based Prevention and Intervention, and Juvenile Justice System Improvement September 9, 2014 Evidence-Based