Embed Size (px)
Citation preview
IMPAQ INTERNATIONAL LLC.
PRESENTER: DR. NEHA NANDA
ADAPTING A QUASI-EXPERIMENTAL DESIGN: IMPLEMENTING AN INTENT-TO-TREAT DESIGN OF A WORKFORCE DEVELOPMENT GRANT TO THE OHIO-PENNSYLVANIA INTER-STATE REGION
AMERICAN EVALUATION ASSOCIATION
AMERICAN EVALUATION ASSOCIATION
AGENDA§ Description of the USDOL Workforce Innovation Fund (WIF) Grants &
Evidence Base Requirements
§ Description of the Pathways to Competitiveness (P2C) Grant Program
§ Ohio-Penn Inter-state Region
§ Evaluation Methodology
§ Quasi-Experimental Design
§ Discussion and Conclusion
AMERICAN EVALUATION ASSOCIATION
DOL WORKFORCE INNOVATION FUND (WIF) GRANTS
§ June 2012, ETA DOL awarded 26 grant programs to support innovative employment and training service delivery in the public workforce system.
§ Design innovative approaches to generate long-term improvements in the public workforce system
§ Improved outcomes for job seekers and employers
§ Build evidence-based practices in the workforce development field
AMERICAN EVALUATION ASSOCIATION
DOL WIF GRANTS – BUILDING EVIDENCE-BASE
§ All grant types required an Implementation study and an Outcome analysis
§ $6 million Workforce Innovation Fund (WIF) USDOL grant to address workforce challenges in the Ohio-Penn Inter-state region.
TYPE A $1-3 Million Dollars Outcome Analysis
TYPE B $3-6 Million Dollars Comparison Group Methodology such as Quasi-Experimental Design
TYPE C $6-12 Million Dollars Random Assignment Design
AMERICAN EVALUATION ASSOCIATION
OH-PENN PATHWAYS TO COMPETITIVENESS (P2C) PROGRAM
§ IMPAQ International – Third Party Evaluator to Conduct Independent Evaluation of P2C Program.
§ Five-county inter-state region consists of three counties in Ohio (Columbiana, Mahoning, and Trumbull) and two counties in Pennsylvania (Lawrence and Mercer).
§ Similar economic environment with common industry sectors and workforce development needs.
§ Significant job losses and chronic unemployment since 1970s, which were exacerbated during the recession of 2007–2009.
AMERICAN EVALUATION ASSOCIATION
OH-PENN PATHWAYS TO COMPETITIVENESS (P2C) PROGRAM§ Manufacturing sector in the Oh-Penn region now poised for resurgence due to the
expansion of the oil and gas industry.
§ Critical shortage of qualified workers and skill gap presented major recruitment challenges for manufacturers
§ Weak talent pipeline
§ Training programs not aligned with employers’ current skill needs
§ Surging retirements in skilled trades positions
§ Job candidates’ lack relevant, hands-on experience
AMERICAN EVALUATION ASSOCIATION
OH-PENN PATHWAYS TO COMPETITIVENESS (P2C) PROGRAM AIMED AT:
§ Addressing workforce challenges facing the Ohio-Penn inter-state region manufacturing sector
§ Increasing enrollment in manufacturing career pathway programs and attainment of nationally-recognized credentials
§ Improving employment and earnings outcomes for job seekers
§ Aligning manufacturing-related education and training curriculum with employers’ skills needs
§ Changing misperceptions about manufacturing
§ Increasing the availability of hands-on learning opportunities in real-world manufacturing environments
AMERICAN EVALUATION ASSOCIATION
OH-PENN PATHWAYS TO COMPETITIVENESS (P2C) PROGRAM
§ Over 700 outreach events including student fairs, youth job fairs
§ Manufacturing Workshops and Jobseeker presentations
§ Expansion of the ‘Industry Needs You’ website
AMERICAN EVALUATION ASSOCIATION
KEY RESEARCH QUESTIONS – IMPACT EVALUATION
§ Does the P2C program improve the employment outcomes for individuals in the Oh-Penn Inter-state Region’s public workforce system?
§ Does the P2C program improve the earning outcomes for individuals in the Oh-Penn Inter-state Region’s public workforce system?
AMERICAN EVALUATION ASSOCIATION
CHALLENGES - EVALUATION DESIGN§ Rigorous Comparison Group Design not
possible in a comprehensive outreach and marketing-based program.
§ No Data Collection involving jobseekers was conducted as part of grant activities.
EVALUATION DESIGN APPROACH
§ “Intent-to-treat” evaluation design based on Propensity Score Matching
§ All individuals residing in the five county Oh-Penn region were treated as program participants.
§ Comparison group, from matched individuals who reside in Ohio and Pennsylvania outside of the Oh-Penn region.
AMERICAN EVALUATION ASSOCIATION
DATA§ Obtained individual-level demographic information and employment and
earnings outcome measures from two administrative data sources from each state:
§ Workforce Investment Act and Workforce Innovation and Opportunity Act (WIOA/WIA) and Wagner-Peyser Act participation data, and
§ Unemployment Insurance (UI) Wage Record data.
§ Controlled underlying county characteristics based on American Community Survey (ACS) 2008-2012 data.
AMERICAN EVALUATION ASSOCIATION
EVALUATION DESIGN APPROACH
§ Separate analysis for WIA and Wagner-Peyser Services and by gender.
§ Step 1: Produce propensity score for everyone in the treatment and comparison group counties
§ Step 2: Match individuals in the treatment group with those in the comparison group
§ Step 3: Run regression to control for the effect of county characteristics on outcomes of interest use matched data within common support region.
§ Step 4: Report regression-adjusted estimates of outcomes.
AMERICAN EVALUATION ASSOCIATION
DISCUSSION & CONCLUSION
§ Program did not lend itself to rigorous impact evaluation
§ Results from an ITT-based design may not be entirely attributable to the program itself
§ Crucial to have a comprehensive implementation and outcome studies to align with and understand impact evaluation results.
