1

YOUTHBUILD EVALUATIONBuilding evidence about the effect

of YouthBuild on the young people it serves

August 17, 2011Washington, D.C.

2

Overview• Project overview and rationale for

study• Research questions and study

design• Roles and responsibilities of

organizations participating in the study

• Timetable for study activities• Questions

2

3

What is the YouthBuild Evaluation and who is

involved?• Large scale, random assignment study

evaluating the effects of program participation on youth enrolled in 77 DOL and CNCS funded programs nationwide

• Sponsored by the U.S. Department of Labor and the Corporation for National and Community Service

• Research team:– MDRC– Social Policy Research Associates (SPR)– Mathematica Policy Research (MPR)

3

4

Why a study of YouthBuild?• Reengaging young high school

dropouts is an important national policy challenge

• Rigorous studies of second chance programs like YouthBuild can help shape their services and policies for youth

• Previous YouthBuild studies very useful, but not designed to measure program effects

4

5

What are the study benefits?

• Distinguishes YouthBuild as a model youth program to policymakers and local program operators

• Provides evidence that supports national policy and public funding

• Opens up the possibility of enrollment for some applicants who might otherwise be screened out

5

6

What questions will the study answer?

• What are the effects or impacts of YouthBuild participation?

• How are YouthBuild services delivered in different settings and what are the characteristics of the youth who participate?

• What is the cost effectiveness of YouthBuild programs?

6

7

What will we learn about program effects?

How do the outcomes for youth enrolled in YouthBuild programs compare to similar youth?– On educational attainment, employment,

earnings, civic engagement, criminal justice involvement?

– Do the effects change over time?– Do some youth benefit more?

Data sources: public records data, enrollment forms, and surveys at 12, 30 and 48 months

7

8

What will we learn aboutprogram implementation?

What services are provided and to whom?– How are services delivered?–What kinds of organizations provide

services?–What are the characteristics of the youth?–What challenges are encountered and how

are they addressed?–What alternative programs are available?

Data sources: visits to programs, focus groups with participants, and program service data

8

9

What will we learn aboutcost effectiveness?

What is the cost of serving the average YouthBuild participant?

How does the cost compare to the impacts generated by the program?

Data sources: analysis of programs costs and participation

9

10

What is random assignment?

• Random assignment is a research method that determines the benefit a program has for the individuals it serves

• Uses a lottery-like process to place individuals into a program group and a comparison group

• Assures that groups have similar characteristics, so the differences over time between the groups is the impact of the program

10

11

Why use random assignment?

• Most reliable way to measure program effects

• Widely used in social service settings• Endorsed by OMB, DOL, DOE and

other federal and private agencies• Fair and equitable way to determine

who receives program services

11

12

80

70

0

20

40

60

80

100

Ready to Work

Per

cent

Pla

ced

In A

Jo

b

I Want A Job

Program Group

Comparison Group

40

0

20

40

60

80

100

Per

cent

Plac

ed In

A

Job

20

General example of random assignment results

12

13

Example from National Guard Youth ChalleNGe

ProgramEducational Attainment

Outcome (%) ChalleNGe Comparison Difference

Has High School diploma or GED 71.8 55.5 16.2***

HS Diploma 30.3 26.6 3.7

GED 56.9 34.5 22.4***

Earned any college credit 34.9 18.8 16.1***

Employment in the Past 12 Months

Employed 88.4 84.5 3.9*

Earnings ($) 13,515 11,248 2,266***

Number of months employed 8.1 7.2 0.9***

13

14

Summary of random assignment steps

Comparison group referred to other

services in the community

Youth receive study welcome letter and $10 gift card 14

15

What is the program’s role?• Designate a contact person to liaise with study

team on progress and challenges• Participate in training on the research procedures• Recruit a sufficient pool of eligible applicants to

make random assignment possible• Administer the research procedures– Ensure applicants are informed about the study

and consent to participate– Record enrollment data into research database

• Participate in survey of grantees and host the study team during the process study

15

16

What is the research team’s role?

• Schedule meetings and site visits• Tailor random assignment to your program• Train program staff on research procedures• Monitor applicant pool to ensure that there

are no crossovers between YB and Non-YB comparison groups

• Collect and analyze research data• Provide assistance throughout research• Report findings to funder• Disseminate findings to practitioners, policy-

makers and others16

17

What is the coach’s role?• Provide technical assistance and

clarification to programs as needed.

• Potential issues include:– Challenges regarding recruitment– Comparison group – Data entry requirements– DOL performance standards

17

18

When will programs be involved?

• Select 77 sites from pool of DOL and CNCS funded YouthBuild programs (2011)

• Visit all sites and begin to develop study enrollment plan (2011)

• Enroll approx. 4,600 youth in the study (2011-2012)

• Administer the Grantee survey (2012)• Study program implementation (2012-

2013)

18

19

QUESTIONS?

19

20

Need more information?Please contactSharon Rowser

sharon.rowser@mdrc.org Cynthia Miller

cynthia.miller@mdrc.orgor

Eileen Pedersonpederson.eileen@dol.gov

or visit www.mdrc.org