Field experiments and policy evaluation · Habiba Djebbari (U. Laval) Field experiments and policy evaluation Siem Reap, Dec 8th 2011 2 / 33. ... A distinct phone number for each

Field experiments and policy evaluation

Habiba Djebbari

U. Laval

Siem Reap, Dec 8th 2011

Habiba Djebbari (U. Laval) Field experiments and policy evaluation Siem Reap, Dec 8th 2011 1 / 33

Introduction

Experimental approach as a cornerstone of scientific method

disentangle cause and effectparticularly diffi cult with naturally occurring data in social sciences“correlation doesn’t imply causation”

Vigorous debate about the value of experimental methods forinforming policy (e.g., Deaton 2010, Heckman 2010, Imbens 2010)


Introduction

Experimental approach: a spectrum of methods

commonality: random assignment of a “treatment”from laboratory experiments to policy experimentsresearch with varying dose of economic theory

“Field experiments refers to the implementation and evaluation, bycomparing different treatment groups chosen at random, of anintervention or set of interventions specifically designed to test ahypothesis or a set of hypothesis” (Duflo 2006)


Contrasting field experiments to policy experiments

Policy experiments are designed to learn about the effectiveness of anintervention:

Objective is to find out whether program should be scaled-up, reformedor even terminatede.g.: PROGRESA cash transfer program in Mexico

Policy experiments are big scale, expensive endeavours

cost of impact evaluation as % of total program costs


Source : Gertler et al. (2011)

Contrasting field experiments to policy experiments

Policy experiments are generally limited to looking for whether theprogram works or not

program usually embeds a large bundle of incentives (e.g., conditionalon attendance of children to school, health talks, nutritionalcomplements to young children and breast-feeding women, cashtransfers, targeting women, etc..)do not assess the effectiveness of each set of incentives separatelydo not aim at explaining mechanisms through which effects aregenerated (results chain)

Field experiments do (e.g., targeting women vs men, varyingfrequency of benefit receipt, with and without conditionalities):

use of economic theory to describe mechanisms, to test hypotheses, toestimate structural parameters


Results chain

Source : Gertler et al. (2010)

Contrasting field experiments to lab experiments

Lab experiments are done in a more controlled setting.

E.g., gift exchange game

hypothesis: employees respond to high wage by working hardercould also be that employees work hard in anticipation of futurecompensationlab experiment: some subjects make a wage offer, others choose aneffort level, and it is a one-shot game

Lab experiments are a first link in the causal chain from the policytreatment to its impact, when theory gives crisp predictions


Source: Fehr et al. (1993)


Lab experiments are usually conducted on university students

they are cheap

Framing is usually avoided

participants are not told that the objective is to test a gift-exchangemodelparticipants are presented with a set of strategies and theircorresponding payoffs

Evidence from lab experiments suffers from external validity problems:extent to which insights gained in the lab can be extrapolated outside(context is artificial, behavior may be affected by this artificialcontext, participants may not have any experience with task, stakesare low, students are not a representative population)



One way to view field experiment is an attempt to make labexperiments more realistic by conducting them in the field, withactual economic agents and with full context

Gneezy and List (2006) test gift-exchange hypothesis with an actualemployment contract:

hire workers to code library books, making clear it is a one-time task.A subset is randomly offered a surprise bonus ($20/hr), while controlgroup get $12 as agreed.Find that people work harder after surprise bonus, but this effectvanishes quickly.Also experimented with door-to-door solicitation and found a strongerinitial effect that also vanishes quickly.


Field experiments as policy tools

Field experiments become a policy tool if they are used as a way to“pretest” the interventions

Requires to use economic theory to identify a mechanism along thecausal chain that may be responsible for the policy effect

if this important underlying mechanism is not found to generate aneffect, then the policy that would embed it is unlikely to produce aneffect either.


A typology of field experiments

Depends on the extent to which economic theory guides experimentalempirical research (Card et al. 2011)

Draw a few examples of field experiments of:

Descriptive studies (Bertrand and Mullainathan 2004, Olken 2007)Single model studies (Jensen and Miller 2008)Competing models studies (Ashraf et al 2011?)Parameter estimation studies (Shearer 1996)


Source: Card et al. (2011)

Source: Card et al (2011)

Piece rates, fixed wages and incentives: evidence from afield experiment

Bruce Shearer (RESTUD 1996)

Are workers more productive when paid with fixed wages or piecerates?

piece rate means that earnings are in proportion of the number of taskperformeddiffi cult to provide unbiased estimates with observational data(selection effect)



Field experiment with a tree-planting firm in British Colombia

tree-planting industry responsible for reforestation of recently loggedtracks of landwhy tree-planting? task is clearly defined: dig a hole, put the plant init, fill the hole.physically demanding: effort is costly, productivity is readily measurableplanting conditions vary with slope, weather, site preparation: varyingpiece ratesfirm employs 90 planters a year, usually under piece rate

Workers randomly selected from 1 firm: limited external validity



Workers randomly assigned to plant trees under fixed wages and piecerate and followed during a period of 120 days

Each worker is observed under both regimes: 120 obs. on dailyproductivity, 60 under fixed wage, 60 under piece rates

Workers plant 20% more trees under piece rate compared to fixedwage!



But finding may not generalize to conditions beyond those observedin the experiment unless incentive effect is independent of plantingconditions or more generally unless worker’s effort is constant

Author estimate a structural model using data from experiment:

models workers’and firm behaviours under piece-rate and fixed wageallows him to generalize findings beyond experimental sample,also allows for evaluation of alternative policies


Are Emily and Greg more employable than Lakhisha andJamel?

Marianne Bertrand and Sendhil Mullainathan (AER 2004)

Send fictitious resumes manipulating perceived race(African-American and White-American sounding names) in Bostonand Chicago (experiment took place in 2001-2002)

Test for differential treatment by race in early stage of therecruitment process

Find that resumes with White-sounding names receive 50% morecallbacks for interviews



Use resumes posted on two job search websites

identify High quality and Low quality resumessuppress names & contact infouse name frequency data from birth cohorts born b/w 1974-79(validated through interviews)assign randomly selected mail addressesA distinct phone number for each race/sex/city/resume quality & use 8different email addressesVoice mail recorded with voice corresponding to race & gender5,000 resumes sent to 1300 distinct job ads



Not only callbacks are higher for White-sounding names, but therewards to having a better resume are also higher for this groupcompared to the group holding African-American-sounding names.

Also find that callbacks are higher when living in a “good”neighborhood, but that there is no premium for Whites or for Blacks

No difference in racial gap by industry (federal contractors) or forthose stating that they are “Equal Opportunity Employers”



Does not provide evidence on racial gap in hiring or racial gap inwages

Resumes do not indicate race, only suggest race:

some employers may not notice the namesresults do not pertain to average African-American but to those withthe most racially distinct names

Policy implications? Neither training nor Equal Employment Policiesare suffi cient to overcome the different treatment by race.


Monitoring corruption: evidence from a field experiment inIndonesia

Benjamin Olken (JPE 2007)

Idea that monitoring and punishment may help control corruption(Becker and Stigler 1974)

However those responsible for M&P may themselves be corruptible

Community participation in local monitoring (Stiglitz 2002):

local monitoring as a public good: under-provisionpotential capture by local elites

Can grassroot monitoring help reduce corruption? How does thetop-down approach fare compared to local monitoring?



RCT with 600 villages in Indonesia (Sept 2003-August 2004)

Context is one where road construction project is launched. Willfunds be diverted?

2 experiments:

Audit experiment: some villages are informed that road project will besubsequently audited by a gov’t agency (and this is actually done)Participatory experiment: in other villages, community members areinvited to village meetings where project offi cials account for how fundswhere spent



Measuring corruption?village reported expenditures for the road (budget sheet)hire a engineer to assess the value of the actual road project completeddifference b/w the two: avge of 24%

Audit experiment:8% decrease in unaccounted for expendituresBut ↑ in # of jobs given to family members (easier to cheat?)

Participatory experiment :no effect on overall missing expendituresbut ↓ in missing labor expenditures (more diffi cult to get by?)

Should we dismiss participatory monitoring approaches?not necessarilymay be more effective for monitoring delivery of private goods thandelivery of pu. good where incentive to free-ride on others’effort atmonitoring is higher


Giffen behavior and subsistence consumption

Robert Jansen and Nolan Miller (AER 2008)

Law of demand: as price ↑, consumer’s demand should ↓Giffen good (Marshall 1895): As M. Giffen pointed out, the rise in theprice of bread makes so large a drain on the resources of the poorerlabouring families and raises so much the marginal utility of money tothem, that they are forced to curtail their consumption of meat andthe more expansive farinaceous food: and bread, being still thecheapest food which they can get and will take, they consume more,and not less of it.

Good for which income effect dominates substitution effect

must be an inferior good (as income ↑, consumer’s demand should ↓)no close substitutesrepresent a large share of the consumer’s expenditures (but not thetotal share)



How does demand for rice in Hunan province of China (wheat inGansu province) vary with exogenous change in prices?

focus on the poor

Experiment: randomly selected households are given a voucher thatallows them to get a discount on rice (wheat) purchases.

Finds that the higher the discount offered, the lower the consumptionof the staple.



What if cash out the vouchers?

If there were resale, one would expect a wealth effect, so that ↓ inconsumption could be consistent with good being an inferior good.

In order to prevent this:

Local food retailers (who redeem the vouchers, are then reimbursedand offered a payment for their participation) are told they cannotexchange the voucher for anything elseBeneficiaries are requested to sign the voucher when making apurchase; only signed voucher get reimbursedLocal retailers get surprise control visitsVouchers offered on a monthly basis and beneficiaries are told that ifcaught reselling they would loose the value of all future vouchers(which subsidize twice the p.c. consumption for a period of 5 months)



Not only consumption of rice and wheat is reduced as price decreases,but this change depends on share of the staple food in calorie intake:

for poorest of the poor who do not consume any other good(calorie-deprived zone), this effect is actually negative: can expectthem to be strongly hit by a food price crisisfor the richer, the staple is not a Giffen good anymoreGiffen good for the poor in the intermediate “subsistence” zone


Source: Fehr et al. (1993)


Policy implications:

subsidies may not improve nutrition for the poor in the subsistence zoneto be effective, nutrition programs must move these people out of thesubsistence zone to the standard zoneon the opposite, for the poor in the calorie-deprived zone,improvements may level off once they get in the subsistence zone.


No margin, no mission? a field experiment on incentivesfor pro-social skills

Nava Ashraf, Oriana Bandiera and Kesley Jack (2011)

Many non-profit employers are interested in how to compensatecommunity agents for their services?

Do compensation schemes affect motivation when a task has a socialbenefit?

Zambia where 1/7 persons live with HIV-1

Young women are particularly at risk

Promotion of female condoms through community agents

information on correct use,correct misconceptions about productneed good distribution networks


Source: h*p://www.povertyac3onlab.org/evalua3on/role-‐incen3ves-‐distribu3on-‐public-‐goods-‐zambia


Use of hairstylists to promote female condoms in the city of Lusaka

Study tests effects of financial and non-financial rewards on selectionand performance of hairdressers in promoting female condoms

Starts with census of all 1222 hairdressers in the city of Lusaka,inviting them to a training on HIV-AIDS, female condoms andbusiness training and offered a payment for their participation

For those who participate to the training, a second payment is offeredwith the possibility to redeem part of the payment as donation for aHIV-AIDS charity:

contextualized dictator game aimed at measuring baseline intrinsicmotivation



Then participants are randomly assigned to 1 of four treatments:

Volunteer treatment (benchmark): no incentivesLow level of monetary incentivesHigh level of monetary incentivesNon-monetary incentives: Stars system

Paper offers a theory of why monetary incentives may crowd-outintrinsic motivation



Main findings:

1 no differential effect on the decision to promote female condoms2 agents perform better under the stars treatment than under any othertreatment:

1 sell twice as many condoms!2 effect is stable through time

3 Sales performance increase the higher the baseline level of intrinsicmotivation

Very nicely crafted experiment:

but lacks a theory for explaining the main resultwhy is the star experiment the most effective? This treatment exploitthe natural tendency for social comparison... so, why does socialcomparison matter? could this provide another policy instrument?


Some final thoughts

A spectrum in terms of theory content in field experimentsA spectrum of methods in experimental approachPolicy-relevance:

policy impact evaluation: very high, depending on external validity ofthe experimentField experiment? those that highlight potential key mechanisms thatare expected to strongly contribute to generating the impact of a policy(Ludwig et al. 2011)

2 situations:Prior to policy impact evaluation: see if the mechanism works in thelab/the field; if it does, then can be embedded in a policy/program,and a policy experiment should be run if possible; if it does not, noneed to bother with the policy experimentTo improve upon the design of a policy: policy impact evaluation areoften design to assess programs that offer a bundle of incentives; whatis working? field experiments are designed to help us address thisquestion: experimenting with each of the incentives.


Some final thoughts

Stepping back further, there are 3 main concerns re. experimentalapproach:

reliance of one single approach to guide policy (gold standard?),requires different set of resources (financial, human resources),focus on whether program works and not on why program works(accumulation of knowledge?)

No “gold standard” to guide policy:

Challenges move from the estimation back to the design andimplementation of the experimentExperimental approach is not appropriate to evaluate large-scale“treatments” that generate general equilibrium effectsRandom assignment is not suffi cient to study distributional effects of“treatments”


Some final thoughts

Researcher acquires a different set of skills:

need to engage with different stakeholders (gov’t, NGO, firms,etc..) inorder for the experiment to be successful (e.g., to preserve integrity ofcontrol group)becomes more knowledgeable of the object of study >> understandbetter the incentives in place, how to interpret the findings, and alsohelps to generate new hypothesesneed to find the funds for conducting an experiment (selection effect)Better chance that results are used to guide policy

Field experiments complement the more standard policy impactevaluation as it helps uncover mechanisms through which the policyeffect may occur


Documents

Field experiments and policy evaluation · Habiba Djebbari (U. Laval) Field experiments and policy evaluation Siem Reap, Dec 8th 2011 2 / 33. ... A distinct phone number for each