NYC Center for Economic Opportunity: Using Data and Evaluation for Program Improvement and Budgeting

NAWRS 2013 Presentation

David S. Berman, MPA, MPH

About CEO

Center for Economic Opportunity– Established December 2006 to implement, monitor,

and evaluate New York City’s ambitious anti-poverty agenda.

Innovation Fund– $100 million public-private partnership.– Supports the implementation of CEO’s anti-

poverty initiatives and pilot programs.– Funds the monitoring and evaluation of programs.

Commitment to Data & Evaluation– All program outcomes tracked.– Program-specific evaluation strategies developed.– Evaluation products include early implementation

reviews, analyses by program area, and several long-term evaluations.

– Evaluation partners include City agencies and nine independent evaluation firms.

– Annual budget decisions based on performance:● CEO has budget authority; funds not committed to

specific agency or activity● Budget pressure – protect the good; free up

resources for new ideas

2

How We Work

Center for Economic Opportunity– Implement new programs in collaboration with

multiple City agencies.– Utilize strong internal and external evaluation

teams to measure results.– Base future funding decisions on program results.– Share best practices and lessons learned.

Agencies– Program design, management of contracts and

providers, and program implementation.

Providers– Provide direct services, recruit participants, and refer

to other services.

Independent Evaluators– Pool of nine independent evaluation firms contracted

to measure the impact of CEO programs.

3

Performance Monitoring Overview

CEO is a data-driven institution that uses data to inform:– Program design, implementation, and evaluation– Community and population needs – service gaps– Budgetary and program decisions

Performance Monitoring Process:– Rigorous Program Management: Check-ins, site visits, phone calls – Qualitative Reports: Monthly narratives outlining activities, challenges

and next steps– Quantitative Reports: Quarterly data reports on certain process,

outcome, and demographical metrics

Delivery Team:– The Center: Internal program managers with direct oversight over

program design, implementation, evaluation, and budget– CEO Data Team: Internal data team responsible for data collection,

management, and reporting

4

Metrics and Targets

Metrics and Data– Set of metrics established to track quarterly performance of

all programs– Some indicators standardized, some unique to individual

programsTargets

– Established for key indicators– Focus on outcomes– Reviewed annually

Data Shared– Public reports share high level data– Data shared back to providers

Performance Improvement Using Data5

Example of CEO Data Dashboard

Cohort 2 (Entered Fall 2009) Three-Year Graduation Rate2 55.0% >24.7% 54.9%

Cohort 3 (Entered Fall 2010) Two-Year Graduation Rate2 55.0% >22.3% 27.5%

Cohort 4 (Entered Fall 2011) One-Year Retention Rate 78.8% >7.2% 78.7%2 Three-year graduation rates are only available for Cohorts 1 and 2; two-year graduation rates are available for Cohorts 1, 2 and 3.

START DATE: 09/2007 | FY 2012 BUDGET: $6,800,000 | STATUS: Baselined1 | SITES: 6Assists students in earning associate’s degrees within three years by providing a range of academic and support services. Program features include advisement and tutoring, tuition waivers, free textbooks, and transportation assistance. The program offers consolidated course schedules to accommodate students’ work schedules, career and employment specialists to help students with job placement, and career development. ASAP targets students who need 1 or 2 developmental courses at the beginning of the program.

CUNY Accelerated Study in Associate Program (ASAP) (CUNY)

Outreach for New Members 899 450 908

Number of New Enrollees 423 315 347

Number Who Completed the Job Readiness Training 186 252 154

Number Placed in Jobs3 156 207 162

Number Retained Jobs at 90-days 90 109 1133 Job placement definition changed to 30-day placements in FY12

Offers NYCHA residents employment and training services, community-based support for work, and rent-based work incentives. In FY13, the program will expand to new sites through the Young Men's Initiative.

START DATE: 10/2009 | FY 2012 BUDGET: $881,000 (YMI & CEO Private) | STATUS: Implementation | SITES: 1

Jobs-Plus at Jefferson Houses (NYCHA/HRA/CUNY)

6

Providers

Our Process

7

Agencies

Feedback on ProgramPerformance: Systemic Changes Made If Needed

Feedback on ProviderPerformance: Provider Changes Made If Needed

Reports Reports

Program Design

Shared Externally

Lessons Identified & Shared

CEO

Evaluation Strategy

8

All CEO programs assessed

Program-specific evaluation strategies based on:– Availability of data– Implementation status– Timing of expected outcomes

Types of assessments– Routine performance monitoring (meetings, site visits, regular reporting,

data analysis) – Early Implementation / outcome studies – Program / topic-specific studies– Impact evaluations

Pilot: Young Adult Internship Program

9

Young Adult Internship ProgramProvides short-term subsidized internships and placements into jobs, education or advanced training, with follow-up services for disconnected youth ages 16 to 24 years.

Program Outcomes● 6,719 young adults enrolled since 2007.● 89% completion rate, 60% placement rate, and 40% retention rate.● Over $15 million in wages earned, with average earnings of $2,500 per

10-12 week internship.● Over 2 million hours worked, with average hours of 350.● 20 community partners, 60+ business partners.

Upcoming EvaluationYAIP will undergo a randomized control trial evaluation to be conducted by MDRC with funding from the U.S. Department of Health & Human Services.

Pilot: Young Adult Literacy Program

Program ModelOffers literacy and numeracy instruction, work readiness, support services, and stipends to disconnected young adults who are unemployed, out of school, ages 16 to 24 who are at the Pre-GED level. Program is delivered by 12 program sites composed of 5 community-based organizations contracted through DYCD and three Public Library systems.

Program Outcomes– Enrolled 750 youth in FY 2012, with over 410 literacy gains, 345 numeracy gains,

and 544 placed in internships. – Monitoring has been used to assess internship participation, and data was used

to change internship requirements to promote higher uptake

Evaluations❧ Summer 2009❧ 2009-2010 Follow-up: Longer term retention and educational gain❧ Upcoming Evaluations: Best practices of high performing sites❧ Findings have been used to change the program model, for example adding paid

internships to all sites

10

Next Generation : Social Innovation Fund - Project Rise

11

Project RiseA cohort-based initiative that provides educational and employment opportunities to 18-24 young adults who are out of work and school and lack a HS diploma or GED. Components include: Quality Education, Paid Internships, Case Management, and more. Project Rise builds on CEO’s Young Adult Literacy and Young Adult Internship programs.

ParticipantCharacteristics Evaluation Plans

Implementation study and analysis of participant outcomes, with particular emphasis on lessons relating to youth engagement and program participation. It will also explore the flow of interested participants into the program and include participant interviews.

Research Informs Expansion: Workforce Development

12

STRATEGY CEO PROGRAMSEVALUATION RESULTS

SIF PROGRAMS

Stable employment and career advancement to provide a clear pathway out of poverty

Sector-Focused Career Centers (SBS) are job placement and training centers that focus on a single economic sector. Each center strives to meet the specific employer needs of that sector and to provide low-income workers with access to jobs with career advancement opportunities.

Sector-Focused Career Centers (Transportation). Participants are placed at higher rates and have higher hourly wages and more weekly hours at placement, as compared to Workforce1 Center’s clients.

WorkAdvance is a blended model designed to assist adults obtain employment in targeted sectors that have room for advancement. The program will be offered in New York City, Cleveland and Youngstown, Ohio, and Tulsa.

Advance at Work (SBS) increases income for employed low-wage workers through job upgrades, access to work supports, and asset-building activities.

Advance at Work. Participants have higher placement rates, higher hourly wages, and more weekly hours, as compared to the traditional Workfore1 Centers clients.

Jobs-Plus at Jefferson Houses (CUNY, HRA, NYCHA) is an evidence-based employment program targeting public housing residents. This place- based program offers employment and training services, outreach, and incentives designed to help “make work pay.”

Jobs-Plus. MDRC’s study of the previous national pilot demonstrated increased earnings for residents for at least seven years after the program’s full implementation, relative to a control group.

Jobs-Plus seeks to raise and sustain the level of employment and earnings among residents of public housing developments. Services will be tailored to residents’ individual needs and draw from a menu of on-site and referral services. The program will be offered in New York City and San Antonio.

Fund What Works, End What Doesn’t

13

1 Advance at Work 1 Business Solutions Training 1 Advocate,Intervene,Mentor 1 AccessNYC Marketing and 2 Child Care Tax Credit Funds (AIM)* Outreach3 Community Partners 2 Employment Works 2 Arches: Transformative 2 CUNY Performance Based 4 CUNY ASAP 3 Food Handlers Certification Mentoring Scholarships5 NYC Training Guide 4 NYC Justice Corps* 3 Cornerstone Mentoring* 3 CUNY Young Adult 6 Office of Financial 5 Shop Healthy 4 CUNY ASAP Evening Employment Program

Empowerment 6 Teen ACTION 5 CUNY Fatherhood Academy* 4 e311 Language Access 7 School-Based Health Centers 7 Young Adult Internship 6 Cure Violence (Ceasefire)* Campaign8 Sector-Focused Career Program* 7 Digital Works* 5 Non-Custodial Parents

Centers 8 Young Adult Literacy 8 Every Child Has an Initiative9 EITC Mailings Programs/ Community Opportunity to Excel and

10 Jobs-Plus Education Pathways* Succeed (ECHOES)*11 CEO Poverty Measure 9 IMPACT Peer Mentoring for 6 Opportunity NYC: Family 12 Office of the Food Policy Young Adult Literacy* Rewards

Coordinator 10 Justice Community* 7 Opportunity NYC: SPARK11 Justice Scholars* 8 Opportunity NYC: Work 12 Learn as You Earn Rewards

Advancement Program 9 Youth Financial Empowerment

(LEAP)* 10 MillionTrees Training 13 Low-Income 11 Nurse Career Ladders

Entrepreneurship Program* 12 CUNY Preparatory

14 Pathways to Excellence 13 City Hiring InitiativeAchievement and Knowledge 14 LIFE Transitions Program(PEAK)* 15 Model Education:

15 Scholars at Work* GED/College16 Teen and Young Adult Health 16 Model Education: Mentoring

Program* 17 Model Education: Supportive 17 Work Progress Program* Basic Skills18 SIF: Family Rewards 18 Social Innovation Fund 19 SIF: Jobs-Plus Providers (4 of 18 for poor 20 SIF: Project Rise performance)21 SIF: SaveUSA22 SIF: Work Advance23 Expanded EITC for Single Adults

Pilot Project * Expanded by Young Men's Initiative

SUCCESSFUL STILL WATCHING EARLY IMPLEMENTATION

Poor Program M

odel/ Implem

entation

DISCONTINUED

One Tim

e Investments

Pilo

t En

ded

Context Changed

Impact and Next Steps

14

Expanding Opportunities– Expanding subsidized jobs programs through the Young Men’s Initiative. – Influencing CEO’s national program model through the Social Innovation Fund.– Developing new program models. – Address unemployment and low-wage work through testing expanded EITC for

Single Adults

Building Evidence– Program evaluations are underway for several subsidized jobs programs,

including Young Adult Internship Program, Work Progress Program, Young Adult Literacy Program, Justice Community, and Project Rise.

– Upcoming reports in 2013:● Data Comparison Study of Young Adult Internship Participants and Youth at

Workforce1 Career Center.● Evaluation of CEO Subsidized Employment Programs.

Sharing Best Practices– Internal/external conferences and forums.– Practitioners, policy makers, and funders.

For More Information

NYC Center for Economic Opportunity

David S. Berman,

Director of Program Management & Policy

212-341-0173

DBerman@cityhall.nyc.gov

www.nyc.gov/ceo

15

/NYCOPPORTUNITY

@NYCOPPORTUNITY