Vu Pham

Refereeing and Discussant Guidelines

Susan Godlonton

AGRODEP AIEN III WorkshopDakar, Senegal4th June, 2014

Vu Pham

• Components of a referee report:• Cover letter

• Summary • Key concerns• Decision

• Report• Overall view• Main concerns• Smaller concerns

• No “one-size-fits-all approach”

Referee Report

Vu Pham

• What is a good paper?• Clear • Coherent (understandable)• Correct

• Additional contribution• Value added of answer to question• Credibility of identification strategy• Validity and robustness of approach

Referee Report: Good paper?

Vu Pham

• Don’t repeat the abstract, the editor can read the abstract

• How would you summarize the paper• What do you think is the key contribution• Do you think the authors address it correctly

Referee Report: Overall view

Vu Pham

• Key issues that the author needs to address• Ensure that it is clear which of your concerns

are fundamental • Don’t ask the author to write a different paper

• Be constructive! Try to provide solutions where possible accounting for data limitations

Referee Report: Main Concerns

Vu Pham

Related literature Inappropriate to the actual material in the paperInaccurately describedIncomplete (very common)

Logical argument Incorrect application of economic conceptsInaccurate mathematical derivations

Loose ties between the economic model and the empirical analysis

Referee Report: Main concerns

Vu Pham

Econometric tools are being used inappropriatelyIncorrectly usedAssumptions do not holdAssumptions not defended with evidence

Estimated key coefficients of interestNot properly identified Not robustAdditional robustness checks needed

Interpretation of empirical results is inappropriate Overstates the contributionOverstates the claim (e.g. causation vs. correlation)

Conclusions are incorrectly made or expressed

Referee Report: Main concerns

Vu Pham

Econometric tools are being used inappropriatelyIncorrectly usedAssumptions do not holdAssumptions not defended with evidence

Estimated key coefficients of interestNot properly identified Not robustAdditional robustness checks needed

Interpretation of empirical results is inappropriate Overstates the contributionOverstates the claim (e.g. causation vs. correlation)

Conclusions are incorrectly made or expressed

Referee Report: Main concerns

Vu Pham

Do not bring these up when discussing a paperSpecific places in text where discussion hard to follow or confusingNot using standard notationSpelling and grammatical errorsMissing data sources and poorly constructed Tables or FiguresReferences to the literature that are missing or incorrect

Referee Report: Minor concerns

Vu Pham

• Refereeing• http

://blogs.worldbank.org/impactevaluations/how-much-to-referee-and-how-to-do-it

• http://marcfbellemare.com/wordpress/2012/01/contributing-to-public-goods-my-20-rules-for-refereeing/

• http://marginalrevolution.com/marginalrevolution/2006/10/how_to_be_a_goo.html

• Discussant• http://chrisblattman.com/2010/02/22/the-discussants-art/

• Other resources will be uploaded to the AIEN workshop

Additional Resources

Vu Pham

Each been assigned a paper/set of slides from one of your peersFocus

Clearly define the research question Define and discuss the identification strategy Propose alternative credible identification strategy

For today…

Vu Pham

Focus on the identification strategyWhat is the identifying variation?What assumptions are made by using the chosen identification strategy (implicitly/explicitly)? Do they make sense?To what extent are these true?How else can the author test the assumptions?What robustness checks or placebo tests could the author run?What would be a more preferable strategy?

For today…

Vu Pham

Thinking about Identification

Homework Assignment

Vu Pham

Group into:HealthEducationAgriculture

Discuss proposed ideas, provide feedback to one another

Thinking about Identification Exercise