Meta analysisThe case-study of aid effectiveness

European Journal of Political Economy. 1st issue 2008 p 1-24

Martin Paldam http://www.martin.paldam.dkClick on to Working Papers

Joint work with Hristos Doucouliagos, Deakin university, Melbourne, Australia

Primary studies of to estimate an effectData: The primary dataYour work in ½-1 year: Less than 1 man-year of work. Reflects: You ideas + have read.

Meta studies of the literature on an effectData: The literatureThe AEL-AAL 250 studies: 200 man-years of work.Reflects: The ideas of everybody in the field.

Forensic economics: Quantitative studies of litera-ture: Where are we? and how did we get there?

The talk discusses the political economy of some of the results of 8 meta studies of:

AEL, Aid Effectiveness Literature, 2005-7 4 studies: 2 out ,1 accepted and 1 R&R

AAL, Aid Allocation Literature 2007-9.

2 new WPs, 2 in partly first draft

Meta studies often give embarrassing results. Strong reactions of referees: Most negative and positive I have experienced.

M1 studies of the effect μ, giving M2 > M1 estimates of μ, that can be calibrated to same scale (partial correlations).

The full-set, [M]F, and the best-set, [M]B.

Coded with a C-vector for each estimate of all possible characteristics:

Our case: An C-vector for 60 possible model controls, estimation methods, data, author, journal

Funnel plots, MRAs, …

Meta studies: Three questions to a literature

1. Does the result converge to something that we can consider the true value?

2. Are there breakthroughs (structural jumps) which can be identified and explained:Does journal quality, estimators, etc matter?

3. Does the distribution of the results point to biases

Three general results (all embarrasing):

(a) Amazing variation in results:Always signifiantly different results

(b) Estimators rarely important:Profession obsessed with estimators?Sociology of our profession: Macho test

(c) Publication biases common: Publication biases: Look at funnel plot. All estimates plotted over precision ln N

Case: The price elasticity of beer. Intuition: Negative but not large. Literature: Average finding surprisingly large

Study the funnels

FATs testing for asymmetries

Correct distribution to find true averageSeveral formulas, fortunately robust!

Tom Stanley (+ Chris Doucouliagos)New paper in Oxford Bulletin

Monte Carlo experiments

The aid effectiveness discussion

Great subject for Meta study:

(1) Emotional: One side is the side of the angels!(2) Strong interests: Stay on the gravy train!(3) A basic statistical fact that has to be overcome

The zero-correlation result

Zero-correlation result two great games:The control variables game: Find a plus setThe causality game: Find a big negative bias!

Zero correlation between aid and growth:

for all aid recipients

Period N Cor Period N Cor

1960-65 92 -0.12 1985-90 143 -0.12

1965-70 103 -0.00 1990-95 169 -0.00

1970-75 111 -0.01 1995-00 178 0.09

1975-80 122 0.06 2000-05 175 -0.02

1980-85 134 0.09 Aver. 1227 -0.00

The causality game: A literature on each arrow

The reverse causality flow R.We wish a negative coefficient

Mechanism Sign

R1: Aid to poor + poor grow less Neg, small

R2: Disasters low growth + aid Neg, small

R3: Growth good projects aid Plus ?

R4: Growth future markets aid Plus, small

Sum: Theoretically ???

Meta study: 30 papers, 211 estimates Plus, tiny

Hence: The data is a big problem. They point to aid ineffectiveness

and it is not obvious that it is due to simultaneity

My own primary studies: very little(everybody know)

Shift perspective: to research

Point of talk: Economics assumes that all humans have priors/interests. Why not us?

Also economists also have priors/interests. We operate on the market for economic research.

It may not be a perfect market.

Our small talk at lunch tables, in bars etc. Often assumes that journals have biases, that referee processes are …

Limitation of discussion: (a) Empirical + (b) macro

Ad (a): We analyze M studies of the same effectAd (b): Data is limited relative to the amount of

research. Thus data mining problem

What can we prove?

Meta studies claim they can prove a great deal.Not for individual studies, but for specific literature

Our studies typically find asymmetries pointing to priors. (In a moment)

We like to believe that research is a process that search for truth that converges to the truth.

Assume: Truth is the true value of μ.

Process in individual/paperProcess on market, market incentives:

Are the incentives truth-finding consistent?

Process for individual researcher, X

X search for a value of μ, till he is satisfied. The paper is thus the result of a stopping rule in Xs search process:

X stops when he has found a μ that:(a) Is in accord with his priors or his interests(b) Is publishable (c) Can be defended statistically

Thus: When I stop have I found truth or confirmed my priors?

Process on market:Innovation + replication generates trust in results

Innovation: Theory, estimation technique, data.Innovation easy to publish (?)

Replication: Independent: Other authors on new data Dependent (1): Same author on new dataDependent (2): New author on new data

Macro: Normally overlapping data so only: Marginally independentThus: Effect on estimate of extra data

Data mining:The number of estimates on subsets of the same data is large relative to the number of observations

Ex: Phillips curves. Estimated on the w, p, u data for 30 countries over the last 50 years? Guess: 5 mio estimated?

Ex: Money demand,…

Ex: Growth regressions: Sala-i-Martin alone about 90 million

Consequence:

Type I errors reduced: Rejecting true modelType II errors increased: Accepting false models

Hence in heavily mined fields: There must be many type II errors

Thus independent replication necessary.And meta studies highly needed

Not problem of each researcher; but the collective. We all read up some of the literature and join the mining collective.

We fish in the common pond of df ’s. A double tragedy of the common.

1. It is the standard tragedy that we exhaust the df.

2. It is also a tragedy that nothing visible happenswe can just go on and on!

The Aid Effectiveness Literature, AEL, studies:μ = ∂g/∂h, conditional on everything our pro-fession has thought of – it is a great del.

Micro base. Average LDC growth 1.5%. Projects based on cost-benefit (growth

contribution): Social rate of return 10%. Thus, h = H/Y ≈ 7.5% 0.75 pp growth

It should be highly visible in data, but as we have seen it is not.

Thus a puzzle: It is the AEL paper generator: In 2006 aid exceeded $ 100 bill + AEL paper nr

100 came out. No agreement on results.

Data: Aid started in mid 1960s. Now ap 145 data per year: and 6000 annual observations. Average to 5 years: about 1000.

Published regressions 1,025, made 25,000? Alternative: Sum of N is 25,000

Likely that false models have appeared

Before we look at results look at the data,

Simple regressions between aid and growth:

The zero correlation result

Growth first No lag Aid first

Coef p-val Coef p-val Coef p-val

All Const 1.816 0.000 1.579 0.000 1.504 0.000

Effect -0.039 0.023 -0.010 0.935 0.003 0.364

N 895 1,008 876

R2 0.006 0.000 0.000

Box Const 1.843 0.000 1.676 0.000 1.578 0.000

Effect -0.052 0.007 -0.022 0.207 -0.010 0.559

N 841 945 839

R2 0.009 0.002 0.000

Thus, the AEL starts from a zero correlation, and put structure on this result till something appears.

Model is the same as the Barro-growth regression: git = α + μhit + (γx1it + ... + γxnit) + uit

orgit = α + μhit + δzit + ωhitzit + (γx1it + ... + γxnit) + uit

Researchers have tried 60 x’es and 10 z’sMany millions possible permutations, each gives a different estimate of μ. As average is zero half are positive, and 5% are significant. What to choose?

A tool: The funnel plot

Prior Bias Inside or outside

(1) For results Polishing Authors, referees, journals

(2) Ideology Accordingly Authors, journals?

(3) Goodness Accordingly Authors, journals

(4) Interests Accordingly Institutions authors, journals?

(5) History Path dependent

Author history + “clubs”

In the AEL: Everything goes together to generate: The reluctancy bias

Researchers and journals are reluctant to publish negative results

Proof follows

Let us look at the 5 priors – one at a time:

Polishing:

We want to display our goods as well as possible.Then they are easier to sell to journals

Career + feel well.

Strong incentives:

What do we expect to see?

Easier to polish in small samples: Study t-ratios as a function of df: t = t(N)

If random ln t proportional to ln N:

The MST: ln│ti│= α0 + α1 ln Ni + ui

test: α1 < 0 polishing

Often found in meta studies, in the AEL also.

Ideology:

An ideology that predicts the size (sign) of μ authors with that ideology find that size (sign).

In the AEL:

a. Libertarians (Friedman, Bauer): Aid larger public sectors planning socialism harms

b. “New-left”: Aid from capitalist states capita-lism and exploitation harms

Both OK (not many authors) especially early

Goodness:

Common finding: We all want to look goodand most want to be politically correct

Shown as asymmetry of funnel plot: The FAT.Part of the funnel is missing.

In the AEL: Aid aims at doing good (+ …) So to show that it fails is bad. Nice to be on the side of the angels: Bono, Jeff Sachs, Gordon Brown, Koffi Anan, etc.

Causes reluctancy.

The FAT: εi = β0 + β1si + υi, where εi is the standardized effect, and si is its standard error.

Also, look at the funnels

We look at: μ = μ(N) and μ = μ(t)in a moment

Interests:

Normally Ok: there are many interests.

In the AEL: diffuse interests on the one side.And the “Aid Industry” on the other side.

It has a turnover of $ 100 bill. It has a composition with includes a bureaucracy + political parties + NGOs + business + unions.

It gives about 10% in consultancy fees + 0.25-0.5 % to research.

The aid industry want aid to work

Many of those working in the AEL are working for/financed by the aid industry

Problem: Many does not write so!

Gives reluctancy as well

Psychology: Alignment of priors to interests

Obs: Goodness + interests give same result in this case. Hence we expect clear effects.

History:

50% are in one paper only. The rest are in more + many additional links.

People are z > 0.5 committed after one paper to find the same result. Our guess z = 0.9

Also, same department as, writing PhD under, …

Very significant: Fighting schools

Reluctancy:

Asymmetry of missing negative values

How should it look? Should be visible on μ = μ(N). Sorting out μ = μ(N) and μ = μ(t)

Problem: Learning by doing: μ = μ(t) should slope upward.

Look at the two graphs:

Problem: No learning by doing, See graphical interpretation, next slide

Run: μNt = α + β ln N + γ t + ε

Multicollinearity: N goes up for t rising

α β on ln N γ on t N

0.31 (7.1) -0.043 (-4.7)

5380.19 (9.7) -0.00027 (-4.4)

0.28 (5.9) -0.026 (-2.1) -0.00015 (-1.9)

Thus:Reluctancy confirmed: Is it goodness or interests?

Test: Use (poorly measured) interest variable

Effect of interest: Always sign as expected, not always significant:It is not a big effect, but it is there!

Correct funnel for asymmetry:Net result insignificant

Thus the literature has not showed that aid works after 100 papers over 40 years.

Depressing

Set of MRAs – very bulky see paper!I just give some highlights:

A Reluctancy + interests found

B Polishing found

C No effect of quality of publication.

D No structural shifts due to new theory

E ODA give slightly better results than EDA

F No effect of new estimators

G Clear effect of new data

Bias in process: Better for career to develop new estimators than new data. An incentive that is not truth finding consistent.

The end:

We behave as predictedby our theories

Also economists are human!