Diapositive 1

Social Experimentation &Randomized Evaluations

Hlne GiacobinoDirectorJ-PAL Europe

DG EMPLOI, Brussells,Nov 2011World Bank Bratislawa December 2011ratThe Need for EvaluationWe have little hard evidence on key questionsWhat is the most cost-effective program to reduce unemployment? What is the real impact of microcredit?

Evidence is important:for maximizing the impact of limited resourcesto give donors and policy makers evidence to select better programs

Evidence provides an objective platform for debate

Evaluations sometimes demonstrate that conventional wisdom needs to be rethought

needs to be part of policy decisions,2A network of researchers at universities around the worldFocused on randomized evaluationsFounded in 2003 by Esther Duflo and Abhijit Banerjee, MIT Professors of Economics

J-PAL: Jameel Poverty Action LabBuild capacityImprove the lives of the poorDisseminate the resultsEvaluate social programsJ-PAL J-PAL today: 5 Offices & 64 Affiliates

306 randomized evaluations in 45 countriesAbout 150 are completed

J-PAL Evaluations around the WorldImplement it on the ground in the form of a real program, and see if it worksTo know the impact of a program must be able to answer counterfactual:How would individual have fared without the programBut cant observe same individual with and without the programNeed an adequate comparison groupindividuals who, except for the fact that they were not beneficiaries of the program, are similar to those who received the programCommon approaches:Before and after (But many things happen over time?)Participants vs. Non-participants (But are they different? More motivated? Live in a different region?)How to evaluate the impact of an idea? 7How do we evaluate whether an idea is good or not?The best way, since the devil is in the details is to implement it on the ground in the form of a program (or evaluate the effort of someone else who had the same bright idea). Need to compare what happened to someone in the program to what would have happened to him in the absence of the program - Key.This is difficult, as no one is observed in the same conditions with and without the program

8Based on Orr (1999)Target PopulationNot in evaluation SamplePopulationRandom AssignmentTreatment GroupControl GroupBasic set up of a randomized evaluationBefore the program starts : assign the program randomly to a subset of an eligible groupPilot programsWhen budget is limited When the program is being phased in

Groups are now strictly comparable

We can determine the impact of the program by comparing the treatment and the comparison group

8Randomized assignmentThis method works because of the law of large numbersBoth groups (treatment and control) have the same characteristics, except for the programDifferences in the outcomes can confidently be attributed to the program

To give useful results, you need to work with a large enough number of people (schools, etc..), because of the law of large numbers:Take 200 people and randomly split them into 2 groups of 100: the average height and weight will be very similar across both groups It would not work if you had only 10 people split into 2 groups

Because the 2 groups (control and treatment) are strictly comparable, except for the program, any systematic differences in the outcomes between them can confidently be attributed to the experimental program.

9The French ContextThe French context

J-PAL EuropeStarted in 2008 with 9 affiliatesToday: 16 affiliates (Belgium, Denmark, France, United Kingdom, Sweden)About 45 projects:Mainly in developed countriesMostly ongoingEvaluations in FranceFrance had almost no evaluation cultureFrench Association of EvaluationMostly process evaluations Endless debates about validity of resultsA very limited common knowledge accumulated about the main social problems and how to address themUrgent need for a method that can provide evidence to direct the public debate

The Experimental Fund for YouthIn 2008, creation of the Experimental Fund for Youth (Fond d'exprimentation pour la jeunesse) To encourage innovative youth policiesTo experiment and evaluate these policies$230 million for 2 years, a team of 12 peopleWithin 18 months, 350 projects funded

What's New?GovernmentEncourages initiatives and does not only support its own programsDecides to support projects only if they are evaluatedPrepares programs for scaled-up conditions from their inception

Knowledge against PovertyChair "Knowledge against Poverty" opens in 2009 at the Collge de FranceFunded by Agence Franaise de Dveloppement for 5 yearsFirst Professor: Esther DufloFighting Unemployment

Many ProgramsCounseling the unemployedCounseling and job placement for young graduate job seekersCounseling welfare recipientsDiscrimination in hiring and anonymous CVsFacilitating youth's access to apprenticeships and encouraging them to complete themSmall business training and loans for aspiring entrepreneurs in disadvantaged neighborhoodsSupporting 18-25 year-olds through long-term mentoring plus financial assistanceCounseling and Job Placement for Young Graduate Job Seekers

Policy IssuesHigh unemployment rate of young peopleEven for young educated peopleDespite government job placement services

Would private operators be able to provide better support?Would strong incentives improve the labor market outcomes?How does this affect employment rates in the overall job market?France has a large population of long-term job-seekers, especially among young generationsEducation helps a lot to find a job but still there is a growing population of young educated people experiencing long term unemployment (25 to 32% of the University graduates are unable to find stable work 3 years after graduation)Counseling them is the job of the Public Employment Service. This raises specific issuesThey have biased perceptions about their labor market value and the type of jobs they can get,There is excess supply for some occupations and deficit in othersThe young people have strong preference to take small jobs that provide cash in the short runThey get trapped in an endless succession of small jobsPublic Employment service may not be the best to address these specific needs: caseworkers have very large portfolios : 120 jobseekers each..Private operators work wit 30 jobseekers

19ContextProgram launched by French Ministry of Labor in 2007, for 2 yearsImplemented throughout 10 regions of FranceAmong 57,000 young job seekers Aged between 18 to 30, holding at least a 2 years vocational or university degreeNo stable work for at least 6 monthsThis evaluation measures the impact of supplementing public employment counseling with more intensive counseling from a private provider.20Description of the InterventionIntensive counseling and placement provided by private operatorsOperators may be for-profit or non-profitOperators have financial incentives to place the young graduates in a durable job10,000 young graduates in the programJob seekers were randomly selected at the local employment agencies levelThe proportion of eligible youth varied from 0%, 25%, 50%, 75% up to 100%, to examine the spillover effect

Intensive counseling and placement provided by private operators selected by call for tender (on the basis of the services they propose to provide and their prices)Operators may be for-profit operators (mainly temporary work agencies) or non-profit onesPrivate operators have financial incentives to place the young graduates in a durable jobPaid in three parts : 25% if the young graduate enters the program (not mandatory)40% if he finds a job within 6 months on a durable contract (of at least 6 month duration)35% if the youth is still employed after 6 months Altogether payment ranges from 1600 to 2100 depending on operators 10 000 young graduates in the program:In ten regions of France Job seekers were randomly selected to receive this intensive counseling.Researchers varied the proportion of job seekers receiving counseling in the different communities to examine whether increasing the number of counseled individuals within a community displaced people who did not receive it.

The program is costly because of both the portfolio size reduction and Private Operators

21Experimental Design50% participants75% participants100% participants0 participant25% participantsA specific design to identify displacement effectTake awayIndividuals who received intensive counseling were more likely to be employed in the short term but over the long term (18 months) these differences disappearedFirms were incentivized to find employment lasting at least 6 monthsthey focused on jobs which just met that conditionBenefits for the beneficiaries are obtained at the expense of non-beneficiariesrotation of the unemployed in the queuing process to access employmentmay be useful for people having no chance at all?

Individuals who received intensive counseling found work more quickly but there was no significant increase in their long term employment rates.18 months after, the difference disappearsNo evidence of stepping stone effects:No increase in durable employment in the long run Programs just speed up the return to employment

Job placement success was higher among for profit agencies than among non profit onesResults fit with what private operators were paid for:Performance of the operator exactly fits the requirement in its contract, does not exceed themPerformance of the operator exactly fits the requirement in its contract

Design chosen so as to measure displacement effectBecause it is a common criticism against counselingBut No displacement effect was foundIndividuals who were not offered this intensive counseling in the areas where a larger proportion of the population received this service were not less likely to find a job.This suggest that rather than crowding out people who did not receive counseling from the pool of available jobs employment counseling can actually increase the efficiency of job searchers.

23

"La Mallette des Parents"La Mallette des Parents (the parents' toolkit)Is it really possible to improve parents involvement ?Has increased parental involvement any effect on children?Does the effect on program participants spread out on other families?

The ProgramChildren and their parents in 6th grade in low-income schoolsThe toolkit: a booklet with thematic guidelines3 meetings with the parentsA DVD in 10 languagesCost: 1000 per school (7/child)

26Messages to the ParentsAll parents can help their childrenWork outside of school is extremely important for successParents should be involved in their children's homeworkChildren need to feel that their parents understand how school functions and care that they adhere to the demands of teachers and administrationExperimental Design: Four GroupsTreated Classes (randomized in)Control classes(randomized out)VolonteerNonvolonteersVolonteerNonvolonteersCompareCompareParents' InvolvementMore interactions with school:Meetings with teachersParticipation in parents' organizations

Helped more their children at home

Same magnitude as the initial gap in involvement between blue-collar and white-collar families

Children's BehaviorLarge improvementeven for classmates whose parents were not volunteers34% less likely to be sanctioned for disciplinary reasonsSimilar results in reduction of absenteeismLimited but significant impact in test scores in French, no impact in MathTake awayProgram demonstrated strong effects on parental involvement and child behaviorto a smaller extent on cognitive achievement

Behavior of all students in the selected classes improved, including those whose parents did not participatePolicy ImplicationSimple and inexpensive program

Rigorous evaluation: can convince schools or governments that such action is worth taking

Ongoing generalization in France

New program launched for children in the last class of compulsory schoolPolicy LessonsDirected parent discussion groups are an effective policy tool for increasing parental involvement, even in underprivileged areaIncreasing parental involvement and awareness of school structure improved student behavior and positively impacted learningEven though only a small fraction of parents choose to participate, the benefits of their involvement were felt by all children in the classroom.More informationwww.povertyactionlab.org34Many thanks !

LimitsNot appropriate when:the impact to measure is a macro impactthe scaling-up of the pilot will modify the impact a lotthe experiment modifies the realitythe beneficiaries are in a context of urgency

It takes a lot of time, sometimes difficult to combine with political needsWhen to do a randomized evaluation?As soon as there is an important question and the answer will help fighting povertyNot too early and not too lateThe program is not too special (can be used in many other contexts)To be really effective, we need time, talents and .. some moneyWhen NOT to do a randomized evaluationWhen it is too early, the program still needs to be ameliorateIf the project is too small (not enough participants)If it was already demonstrate that this program has a positive impact. Use the money for more beneficiaries!If the program has already startedThe question of ethicResources are very limited any wayThis random assignment is usually seen as very fairEvery project has to go through an ethic committeeJ-PAL never works on projects if the cost of the evaluation means less beneficiaries. After the evaluation is over, if necessary, the control group will also get the programDifficulties in the fieldWe face some resistances:some people just cannot accept the random assignment perceived as unjusttake up is much lower than previous,Not always easy to overcome the cultural differences between researchers and partnersNot easy to deal with very decentralized organizationsThe evaluation sometimes goes against some financial or political interests Be very careful in choosing the program!ConclusionsPolicy needs experimentationToo many errors are made all the time; too much time is lost; too much money is wastedWithout experimentation, the stakes are too high. What has been scaled up cannot afford to be wrong. No incentive to learn and progress

Experimentation needs to be seriousWe need to be rigorous in determining success or failureOtherwise, everyone is free to defend their pet project

Experimentation needs to be creativeIf we are just trying, and accept the possibility of failure, we do not need to think inside the box This mindset could revolutionize social policy