Brigitte Manteuffel, Ph.D.g ,ICF Macro

October 27, 2010

The cross-site evaluation will address ◦ The relationship between the HTI program structure

and inputs (e.g., the State–local organization collaborative relationship, the adult–child mental p,health system collaborative relationship), ◦ Process evaluation through fidelity assessment, and

Outcome evaluation of youth outcomes◦ Outcome evaluation of youth outcomes.◦ Comparative effectiveness studies

What was the effect of the intervention on participants?participants?

What was the effect of the HTI intervention on participants, compared to young adults participating in SOC programs?p g

What was the effect of the HTI intervention on participants, compared to a national sample of similar young adults not participating in the HTI intervention?intervention?

What program/contextual factors were associated with outcomes?

What individual factors were associated with What individual factors were associated with outcomes?

How durable were the effects?

Three studies will use data collected from◦ The HTI study◦ Existing national data◦ Data from the CMHI longitudinal outcome study◦ Data from the CMHI longitudinal outcome study CMHI data from previous and currently funded cohorts

consist of older youth (17–18 years old) who can be compared with young adults in the HTI programscompared with young adults in the HTI programs.

Several funded CMHI programs target transition-aged youth. D f h b d i h HTI Data from these programs can be compared with HTI programs

What is the comparative effectiveness of each of the What is the comparative effectiveness of each of the three HTI program models?

What is the comparative effectiveness of services received by youth in HTI programs versus morereceived by youth in HTI programs versus more traditional services received by a national sample of youth?

Using data from the National Longitudinal Transition S d 2 (NLTS2) h i h di bili i illStudy 2 (NLTS2) on youth with disabilities, we will compare outcomes of HTI youth with this national sample.

What is the comparative effectiveness of HTI What is the comparative effectiveness of HTI programs versus CMHI programs serving transition-aged youth?

Two-tier, longitudinal cohort design◦ Includes three embedded comparative effectiveness

studies. Data collected at intake and every 6 months Data collected at intake and every 6 months

for 24 months All youth receiving services eligible for y g g

longitudinal outcome study Youth self report Intensive tracking and follow up of the first

196 youth entering services.

NOMS measuresd l h d f Transition to Adulthood Program Information

System (TAPIS) ◦ E.g., life skills, employment, housing, education,E.g., life skills, employment, housing, education,

social support, and parenting CMHI youth measures (clinical, functional)

Possible other sources of measures Possible other sources of measures◦ Ansell-Casey Life Skills Assessment, the NLTS2, the

Post-School Data Collection Question Bank, the O R i S l ( d i h H d S l fOutcome Rating Scale (used in the Heart and Soul of Change Project), and the Peabody Treatment Progress Battery

Proposed to engage youth in evaluation ◦ Opportunity to obtain real-time results on their

progress◦ Providing additional data on change over time.Providing additional data on change over time. ◦ Can be conducted at any time, enabling young

adults to assess their progress according to their own timelineown timeline.

Annie E. Casey Foundation developed similar processprocess

Standard Protocol◦ Grantee will conduct in-person baseline interviews follow-up interviews with youth who remain in servicesfollow up interviews with youth who remain in services

(consistent with the TRAC data collection process and outcome evaluation protocol) To the extent possible grantees to track and follow upTo the extent possible, grantees to track and follow up

with youth who have left services. Training and TA on retention strategies Tracking system for monitoring all youth in services Tracking system for monitoring all youth in services.

Intensive Tracking Subsample◦ First 196 youth (28 per site) entering services◦ Informed consent for follow up by cross-site

evaluation staffevaluation staff◦ Follow up while in services by grantee evaluator

consistent with TRAC protocol, follow up by cross-site staff after youth leave servicessite staff after youth leave services

Descriptive statistics◦ Youth characteristics◦ Youth characteristics◦ Site and cross-site levels◦ Subgroup analyses

Multivariate statistics Multivariate statistics◦ relationship of site-level differences in program fidelity

on differences in youth-level outcomes using multiple regression analysis

C i l Comparative analyses◦ Outcomes for three HTI program models implemented by

the seven grantees◦ Outcomes for the sample of youth entering services in◦ Outcomes for the sample of youth entering services in

HTI sites, compared to similar youth served in CMHI programs and in the NLTS2 sample (samples of 157 youth in each) . R d ANOVA◦ Repeated measures ANOVA.

The purpose of this evaluation component is to fid li i hassess program fidelity, using the

Comprehensive Program for Transition-Age Youth Program Fidelity Assessment tool, andYouth Program Fidelity Assessment tool, and assess each program’s adherence to requirements of the grantee RFA. Th d i l d i d i The study involves document review and onsite in-depth site visits that include semistructuredinterviews with key constituents and case recordinterviews with key constituents and case record reviews to collect data on system fidelity, development, and sustainability.

How closely did implementation match the plan?

What types of deviation from the plan occurred?occurred?

What led to the deviations? What effect did the deviations have on the What effect did the deviations have on the

planned intervention and performance assessment?

Who provided what services, to whom, in what context, and at what cost?

Two program fidelity assessment site visits will be conducted in all seven grant communities across 3 years◦ One in the second year of grant funding after OMB◦ One in the second year of grant funding after OMB

approval has been granted and ◦ One at a 24-month interval from the first site visit.

Transition-Aged Youth WorkgroupI t◦ Input on Strategies to engage youth and relevant outcome measures Strategies for engaging and retaining youth in services and

the evaluation instruments and measures to assess relevantthe evaluation, instruments and measures to assess relevant outcomes for transition-aged youth

Recommendations for assessment method (e.g., self-assessment versus interview

◦ Participants Youth/young adult participants HTI local evaluators and project directors Experts in the fields of mental health, education, juvenile

justice, and child welfare. ◦ 2-day meeting in January

Review grantee materials Review and select measures Obtain input from experts and transition age

h kyouth workgroup Finalize evaluation design Prepare OMB package and obtain approval Prepare OMB package and obtain approval Evaluation training and TA Prepare data management & tracking systems Prepare data management & tracking systems