APHSA Members WebinarSeptember 26, 2013

The Innovative Strategies for Increasing Self-Sufficiency (ISIS) Evaluation

Introduction

Purpose of webinar Introduction of speakers Review of agenda

– ACF Perspective– Evaluation Summary– Evaluation Partner: Workforce Development Council

of Seattle-King County– Q & A

2

What Is ISIS? ISIS – the Innovative Strategies for Increasing Self-Sufficiency

evaluation – is an evaluation of promising strategies for increasing education, training, employment and self-sufficiency among low-income families.– Impact Study using Random Assignment– Implementation Study– Cost-Benefit Analysis

Evaluation by the Office of Planning, Research and Evaluation (OPRE) in the Administration for Children and Families (ACF)

Led by Abt Associates, in partnership with MEF Associates, APHSA, NCSL, NGA, consultants, Public Strategies, Inc. and the University of Michigan.

Support for programs provided by the Open Society Foundations, The Joyce Foundation, The Kresge Foundation, The Meadows Foundation and ACF/OFA Health Professions Opportunity Grants.

10-year project, currently in the random assignment phase (year 6)3

Why is ACF and OPRE evaluating career pathways? OPRE is the principal research and evaluation office of ACF.

– OPRE is to serve as the “principal advisor” to the Asst Secretary on research and evaluation matters.

– Our research covers the entirety of ACF’s program areas (though not evenly).

Our research and evaluation activities are guided by the principles of rigor, relevance, transparency, independence and ethics.

We have a long history of conducting experimental research, particularly in welfare to work and employment and training programs (see new book by Gueron and Rolston).

Our approach to research and evaluation has also taken on greater prominence and importance in the current Obama Administration and OMB’s emphasis on “evidence-based policy-making.”

4

Poverty and Pathways:The Motivation for ISIS 3 Key “Facts”

– Previous research has demonstrated effectiveness in moving families off cash assistance and into employment. (NEWWS, GAIN, New Hope, MFIP, SSP, state waiver experiments, etc.)

– Economic mobility is increasingly “skill-biased.”– Individuals with lower skills/lower income have fewer financial and

human capital resources from which to draw on and make productivity investments.

• Money is a major constraint, but time is also an important constraint. 2 Important Limitations of Existing Research:

– Education and training has not been as promising or financially beneficial for poor families as it has for the population as a whole.

– Families were not substantially better off even in the successful interventions (large numbers remain poor.)

1 Major Challenge– How can we integrate education, workforce development and human

services to create pathways out of poverty and increase self-sufficiency for a large and diverse set of American families? 5

Emerging Evidence: Career Pathways and Sector-Based Strategies

Year Up (Economic Mobility Corp, 2010)

• National org provides 6 months customized training + 6 month paid internship in IT and finance sectors w/local colleges, completers earn 14+ credits

• Results from small initial experiment (n=164)

6

Two key questions:1) Do future earnings recoup those foregone while in training?2) Do the benefits exceed the costs?

Emerging Evidence: Career Pathways and Sector-Based Strategies

Sectoral Training Strategies (Maguire et al. 2010)

3 experienced CBOs provide customized short-term training in varied high-demand fields

Disadvantaged adults with HS+ education

Careful screening Total sample n=1,014

7

The ISIS Evaluation ISIS aims to learn about promising strategies for increasing

employment and self-sufficiency among low-income individuals and families

Core principles– Collaborate with stakeholders and partners to Identify most relevant

interventions and strategies • Semi-structured discussions with over 250 individuals (key APHSA role)

Test a few things well– Experimental design with random assignment– Test one type of intervention in nine sites, rather than nine disparate

interventions Aim for positive influences on social policy and practice

8

Key Career Pathways Ideas A series of connected education and training programs and

support services – Addresses the wide range of skill and other needs of low-skilled

individuals– Provide credentials valued in high demand occupations/sectors

Enables individuals to secure employment within a specific industry or occupational sector, and to advance to successively higher levels of education and employment within that sector

Each step is designed to prepare the participant for the next level of employment and education

Build effective partnerships– Education and training providers, social service providers,

workforce 9

V. BA+ Programs Upper-Skilled Jobs

IV. 1-2-Year Certificate to AA Programs Mid-Level Skilled Jobs

III. Short-Term Certificate Programs Entry-Level Skilled Jobs

II. Sectoral Bridge Programs Semi-Skilled Jobs

I. Basic Bridge Programs

Pros

pect

s fo

r goo

d-pa

ying

, sta

ble

empl

oym

ent

Occupational, academic, and life skills

The Basic Career Pathways Model

Key Career Pathways Services Comprehensive assessment

– Academic and non-academic skills Basic and technical skills instruction

– Modularization, contextualization, acceleration, flexible delivery, active learning

Supports – Proactive advising and guidance, supplemental instruction,

social supports, supportive services, financial assistance Employment connections

– During and after training

11

Foundational Academic Skills

Intermediate Outcomes

Increase Performance & Persistence in Training

• Certificate/Diploma• 2- year, 4- year Degree

Improve Performance & Advancement in Jobs

• Earnings• Benefits• Job security

Improve Other Outcomes

• Income & assets• Child & adult well being• Local economic growth

Participant Characteristics•Demographic•Educational•Economic

Program Inputs Primary Outcomes

TO N

EX

T S

TEP

Occupational Skills

Psycho-social Factors

Career Orientation and Knowledge

Resource Constraints

Other Personal and Family Challenges

Contextual Factors: Institutional, Economic, Social

Initial Targeting & Placement Decisions

Theory of Change for Career Pathways

Take First/ Next Step In

Career Pathway/Latti

ce

Comprehensive Assessment

Supports

Employment Connections

Core Curriculum

The ISIS Sites

13

Year Up National Capital Region

Year Up Atlanta

Year Up Boston

Year Up Chicago

Year Up New York City

Year Up Providence

Year Up Puget Sound

Year Up San Francisco Bay

Area

San Diego Workforce Partnership Bridge to Employment

Program

Des Moines Area Community College

Prepared Learner Program

Valley Initiative for Development and

Advancement (VIDA)

Instituto del Progreso Latino Carreras en Salud

Madison Area Technical College Patient Care Pathways

Bellingham Technical College I-BEST

Pima Community College Pathways to Healthcare Program

ISIS HPOG site

ISIS site

Workforce Development Council of Seattle-King County Health Careers for All

Everett Community College I-BEST

Whatcom Community College I-BEST

Health Professions Opportunity Grants (HPOG) In 2010, the HPOG program was created as part of

the Affordable Care Act in order to provide TANF recipients and other low-income individuals with access to jobs that pay well and address workforce needs in health professions.

$67 million a year to 32 grantees across 23 states, including 5 tribal grantees.

Program is administered by the Office of Family Assistance (OFA).

Research and evaluation overseen by OPRE.

14

Health Professions Opportunity Grants (HPOG) To evaluate HPOG ACF has implemented a multi-

pronged research strategy that includes:– Implementation, systems and outcomes evaluation for

HPOG TANF/low income grantees– Evaluation of Tribal HPOG – Performance tracking: HPOG PRS collects uniform

data across all grantees– Impact studies in a subset of grantees

• HPOG Impact Studies• Innovative Strategies for Increasing Self-Sufficiency (ISIS)

– University-based field initiated research15

The Sample of ISIS Sites Career pathways is both a framework and a model

for particular programs – many variations incorporating basic ideas possible

9 ISIS programs exemplify central career pathways ideas, but vary greatly in:– Basic elements such as target populations (e.g.,

TANF), occupations and training steps, provider partnerships

– Lead organization—CBOs (3), Community/Technical Colleges (4) and WIBs (2)

– Key strategies included; their design, intensity, and duration

16

Who is participating in ISIS?

17

BTC DMACC Instituto Madison Pima SDWP VIDA WDC Whatcom Year Up NCR

Total Sample

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

48.3%

26.8%

6.9%18.4% 14.8% 13.0%

28.0%16.7% 14.3%

52.2%

22.0%

51.7%

73.2%

93.1%81.6% 85.2% 87.0%

72.1%83.3% 85.7%

47.8%

78.1%

Sex

Male Female

Who is participating in ISIS?

18

BTC

DMACC

Instituto

Madiso

nPim

aSDWP

VIDA

WDC

Whatcom

Year U

p NCR

Total Sam

ple0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

55.2%

19.6%28.7%

39.8%

19.0%29.5%

36.7%

14.6% 14.3%

98.6%

32.8%

31.0%

38.4%

55.4%45.8%

44.6%

42.5%

53.3%

58.3%

1.4%1.4%

44.9%

13.8%

42.0%

15.9% 14.4%

36.4%28.0%

10.0%

27.1%

85.7%

22.3%

Age at Baseline

Under 25 25 to 39 40 and older

Who is participating in ISIS?

19

BTC

DMACC

Instituto

Madiso

nPim

aSDWP

VIDA

WDC

Whatcom

Year U

p NCR

Total Sam

ple0%

10%

20%

30%

40%

50%

60%

70%

80%

44.8%

75.4%

54.4% 53.2%

68.1% 66.5%

53.5%

75.0%

28.6%

50.7%

60.4%

37.9%

9.4%

19.8%27.4%

7.1%13.5% 12.7%

6.3%

28.6% 31.9%

15.0%

Total Family Income in Prior Year

Below $15,000 $30,000 or Above

ISIS Program Diversity

Health Office/Business

IT Welding/Electrical

Customer Service

Building Maint..

DMACC X X X XI-BEST X X XInstituto XMATC XPCC XSDWP XVIDA X X X XWDC XYear Up X X

20

ISIS Research Questions

What is the impact of the program on persistence in education and achievement of credentials and degrees?

What is the impact of the program on career-track employment and earnings?

What is the impact on well-being?

21

Study Components

ISIS will include site-specific: Impact study

– Using baseline data, administrative data (UI, college records), surveys

Implementation study– Interviews, staff surveys and site-level administrative data

Cost-benefit studyStudy parameters: Randomly assign a minimum of 1,000 eligible

individuals to treatment and control groups– Control group can access other service in the community

22

Study Deliverables

Evaluation design Program profiles (2013) Cross-program implementation report Program-specific 15 month reports

– Implementation – Impact– Cost

23

ISIS Program

Health Careers for All Workforce Development Council of Seattle-King

County

24

What is HCA?

Health Careers for All is funded by a 5-year HHS HPOG grant

Healthcare training via cohorts or individual training support

Career and education “navigation” support Efforts to better align systems that serve the target

population (workforce, education, social service)

Outcome Targets for HPOG

Over the 5 years of the grant period: Enroll 920 TANF recipients and other low-income

adults and youth Achieve a training completion rate of 70%

(benchmark = 55%) Achieve a training-related placement rate of 60%

(benchmark = 25%) Achieve a return-to-training rate of 25% (benchmark

= 11%)

Outcomes to Date

Over life of the HPOG grant Nearly 600 customers enrolled Approximately 300 customers completed healthcare

training More than 150 customers employed Approximately 60 customers continued/returned for

next level training

TANF Participation

Project Goal to have TANF recipients comprise approximately 1/3 of enrollments over the HPOG project period

TANF participation has averaged 40-45% over the first 3 years of the project

Project designed in close partnership with TANF agency

TANF Partnership

Partners in developing project model and proposal Members of project steering committee overseeing

implementation of project model Clearly articulated outreach and referral protocol Regular check-in meetings in addition to steering

committee

HCA in ISIS

Implemented ISIS in summer 2012 (end of Year 2 of HPOG grant)

ISIS Goal: Randomly assign 700 individuals Outcomes to Date (end of Aug 2013):

– 267 individuals RA’d (51% TANF)– 134 assigned to treatment group (51% TANF)– 110 enrolled in HCA– 94 entered training

30

TANF Partnership Post ISIS

Re-doubled coordination efforts during implementation of ISIS study

Refined referral protocols Generated materials to support TANF staff Efforts to safeguard against over-enrollment in

control group during random assignment (ensure 50-50 treatment/control split among TANF referrals in addition to overall RA pool)

Customer Story Kim is a single mom of three boys who came to HCA in 2011 after nearly two years of training and job hunting in medical administration produced no employment. She had been trying to enter the healthcare field since 2008, was nearing the end of her lifetime TANF benefits, and had nearly exhausted her allotted training time through DSHS. Kim had 9 years of steady employment as a journeyman Industrial Maintenance Mechanic but in 2008 was terminated from employment because of excessive time off related to domestic violence issues. Due to the downturn in the economy, she was unable to find alternate employment in her profession. After a period of homelessness, she found transitional housing and subsequently entered subsidized housing. In addition to employment and housing challenges, Kim was busy managing 3 young boys all with behavioral issues and special needs. After some counseling to figure out the best path for her to enter a health career as quickly as possible, and to best utilize skills gained along the way, she enrolled in the HCA Phlebotomy cohort, which she completed with the highest scores in the class.  Aside from excellent grades, Kim demonstrated patience, bilingual skills, a wry sense of humor and excellent ability work calmly with diverse patients. Following the phlebotomy training, Kim received job search assistance through WorkFirst and the HCA job developer.   In September 2012, she was offered a position at LabCorp at $14.54 per hour, 25 hours per week to start.  Three months later, Kim called to say that she had received praise for the work she was doing and was offered a full-time position. Kim is very thankful for the ongoing services received from the HCA program and now wants to start nursing prerequisites so she can eventually become an RN.

Lessons Learned to Date

Importance of understanding the TANF program and how it is implemented in your area

Importance of cultivating relationships at both the local and regional/state level

Benefits and challenges of grappling together with barriers (e.g., participation requirements and 12-month limit on training as a “countable” activity)

Questions?

34

Additional Information

www.projectisis.orghttp://www.acf.hhs.gov/programs/opre

35

Karen GardinerAbt AssociatesKaren_gardiner@abtassoc.com

Brendan KellyOPRE/ACFBrendan.Kelly@acf.hhs.gov   

Seanna RuvkunWorkforce Development Council of Seattle-King CountySRuvkun@seakingwdc.org

Cynthia WoodsideAPHSAcwoodside@aphsa.org