Upload
rosalyn-gibbs
View
222
Download
0
Tags:
Embed Size (px)
Citation preview
What Works? Evaluating the Impact of Active Labor Market Policies
May 2010, Budapest, Hungary
Joost de Laat (PhD), Economist, Human Development
Outline
• Why Evidence Based Decision Making?
• Active Labor Market Policies: Summary of Findings
• Where is the Evidence? The Challenge of Evaluating Program Impact
• Ex Ante and Ex Post Evaluation
3
Why Evidence Based Decision Making?
• Limited resources to address needs
• Multiple policy options to address needs
• Rigorous evidence often lacking to prioritize policy options and program elements
4
Active Labor Market Policies:Getting Unemployed into Jobs
• Improve matching of workers and jobs
• Assist in job search• Improve quality of labor supply
• Business training, vocational training• Provide direct labor incentives
• Job creation schemes such as public works
5
Active Labor Market Policies
Public Expenditure as % of GDP in OECD Countries, 2007 (OECD Stat) ACTIVE LABOR MARKET POLICIES 10: PES and administration 0.1520: Training 0.1430: Job rotation and job sharing 0.0040: Employment incentives 0.1050: Supported employment and rehabilitation 0.0960: Direct job creation 0.0570: Start-up incentives 0.01TOTAL ACTIVE 0.56 PASSIVE LABOR MARKET POLICIES 80: Out-of-work income maintenance and support (incl. unemployment insurance) 0.6490: Early retirement 0.11TOTAL PASSIVE 0.75
6
International Evidence on Effectiveness of ALMPs
• Active Labor Market Policy Evaluations: A Meta Analysis. By David Card, Jochen Kluve, and Andrea Weber (2009)• Review of 97 studies between 1995-2007
• The Effectiveness of European Active Labor Market Policy. By Jochen Kluve (2006) • Review of 73 studies between 2002-2005
7
Do ALMPs Help Unemployed Find Work?(Card et al. (2009), Kluve (2006))
• Subsidized public sector employment• Relatively Ineffective
• Job search assistance (often least expensive)• Generally favorable, especially in short run• Combined with sanctions (e.g. UK “New Deal”)
promising
• Classroom and on-the-job training• Not especially favorable in short-run• More positive impacts after 2 years
8
Do ALMPs Help Unemployed Find Work?(Card et al. (2009), Kluve (2006))
• ALMPs targeted at youth• Findings mixed
9
The Impact Evaluation Challenge
• Impact is difference in outcome with and without program for those beneficiaries who participate in the program
• Problem: beneficiaries have only one existence; they participate in the program or they do not.
10
Impact Evaluation Challenge: before – after comparison ok?
before after
$1000
$2000
Skills Training
Program Impact = $1000 extra income?
Income for beneficiary increases from $1000 to $2000 after training
11
Impact Evaluation Challenge: before – after often incorrect
before after
$1000
$2000
NO Skills Training
NO! Program Impact = $500 $1500
Income for the same person but without training would have increased from $1000 to $1500 because of improving economy
12
Impact Evaluation Challenge
•Solution: a proper comparison group
• Comparison outcomes must be identical to treatment group outcomes, if the treatment group did not participate in the program.
13
Impact Evaluation Approaches
Ex ante:1.Randomized evaluations2.Double-difference (DD) methods
Ex post:3. Propensity score matching (PSM)4. Regression discontinuity (RD) design5. Instrumental variable (IV) methods
14
Random assignment
before after
$1000
$2000
Skills Training
Program Impact = $500
$1500
Income comparison group is $1500
Income treatment group is $2000
15
Randomized AssignmentEnsures Proper Comparison Group
• Ensures treatment and comparison at start of program are the same (background and outcomes)
• Any differences that arise after program must be due to the program and not due to selection-bias
• “Gold” standard for evaluations; not always feasible
16
Examples Randomized ALMP Evaluations
• Improve matching of workers and jobs• Counseling the unemployed in France
• Improve quality of labor supply• Providing vocationally focused training for
disadvantaged youth in USA (Job Corps)
• Provide direct labor demand / supply incentives• Canadian Self-Sufficiency Project
17
Challenges to Randomized Designs
•Cost
•Ethical concerns: withholding a potentially beneficial program may be unethical
• Ethical concern must be balanced with:• programs cannot reach all beneficiaries (and randomization may be fairest)
• knowing the program impact may have large potential benefits for society …
18
Societal Benefits
• Rigorous findings lead to scale-up:
•Various US ALMP programs – funding by US Congress contingent on positive IE findings
• Opportunidades (PROGRESA) – Mexico
• Primary school deworming – Kenya
• Balsakhi remedial education – India
19
Ongoing (Randomized) Impact Evaluations:Ongoing (Randomized) Impact Evaluations:From MIT Poverty Action Lab Website (2009)From MIT Poverty Action Lab Website (2009)
20
World Bank’s Development Impact Evaluation Initiative (DIME)
• 12 Impact Evaluation Clusters:• Conditional Cash Transfers• Early Childhood Development• Education Service Delivery• HIV/AIDS Treatment and Prevention• Local Development• Malaria Control• Pay-for-Performance in Health• Rural Roads• Rural Electrification• Urban Upgrading• ALMP and Youth Employment
21
Other Evaluation Approaches
Ex ante:1.Randomized evaluations2.Double-difference (DD) methods
Ex post:3. Propensity score matching (PSM)4. Regression discontinuity (RD) design5. Instrumental variable (IV) methods
22
Non-Randomized Impact Evaluations “Quasi-experimental methods”
•Comparison group constructed by evaluator
• Challenge: evaluator can never be sure if behaviour of comparison group mimics that of treatment group without program: selection bias
23
Example: Suppose Only Very Motivated Underemployed Seek Extra Skills Training
• Data on (very motivated) under-employed individuals who participated in skills training.
• Construct comparison group from (less motivated) under-employed who did not participate in skills training.
• DD method: evaluator compares increase in average incomes between two groups
24
Double-Difference (DD) Method
Treatment group
Comparison group (non-randomization)
Program impact (positive bias)
25
Non-experimental design
•May provide unbiased impact answer•Relies on assumptions regarding comparison•Usually impossible to verify assumptions
•Bias always smaller if evaluator has detailed background variables (covariates)
26
Assessing Validity of Non-Randomized Impact Evaluations
• Verify pre-program characteristics are same between treatment and comparison
• Test ‘impact’ of program on outcome variable that should not be affected by the program
• Note: will always hold in properly designed randomized evaluations
27
Conclusion
•Everything else equal, experimental designs are preferred. Assess case-by-case.•Most appropriate when:
• New program in pilot phase• Not in pilot phase but receives large
amounts of resources and its impact is questioned
•Non-experimental evaluations often cheaper; interpretation of results requires more scrutiny
28
THANK YOU!
29
Impact Evaluation Resources
• World Bank (2010) “Handbook of Impact Evaluations” by Khandker et al.
• www.worldbank.org/sief • www.worldbank.org/dime • www.worldbank.org/impactevaluation • www.worldbank.org/eca/impactevaluation (last site coming soon)
• http://ec.europa.eu/regional_policy/sources/docgener/evaluation/evaluation_en.htm
• www.povertyactionlab.org• http://evidencebasedprograms.org/