Does It Matter Where We Publish?

Adam Eyre-WalkerUniversity of Sussex

Reasons

• Prestige• peers• employers

• Impact• scientific community• society

Measuring impact

• General• difficult to measure

• Citations• Easy to measure• Velvet Underground effect

• “only sold 10,000 copies, but everyone who bought it formed a band” (Brian Eno)

Question

Do papers of similar quality garner more citations in better quality journals?

Ideal experiment

• Papers made anonymous• authors• journal• structure

• Assessed for quality by experts• Assessments confidential• Citations tracked

Research Assessment Exercise (RAE)

• Each department submits• 4 publications per academic (75%)• 4 indicators of Esteem (5%)• Environment

• PhD student numbers• Grant income• Research structure• Facilities

• Superceded by REF

RAE dataset

• 1170 papers published 2001 - 2007• Scored by AEW• Scored 1 to 4 (or unclassified)• Subjects : evolutionary biology, genomics,

bioinformatics, ecology, animal behaviour and organismal biology

• Ranked without reference to• journal• citations

RAE scores

RAE score No. of papers

1 74

2 539

3 401

4 103

Journal quality

• Impact factor• Number times papers published in the two

previous years are cited in target year

• Two components• Quality of papers• Effect of the journal on citations

IF categories

IF category IF No of journals

No. of papers

1 IF < 2 41 532 2 < IF < 3 39 1343 3 < IF < 4 33 1594 4 < IF < 5 21 2145 5 < IF < 6 16 1286 6 < IF <10 28 1807 10 < IF < 20 13 1108 IF > 20 8 181

Citations

• No. of citations from ISI Web of Science• October 2008

Data

• For each paper• Year of publication• RAE score (RS)• No. of citations• Impact factor of journal (IF category)• Otherwise anonymous

How good is the assessor?

• Correlation between AEW and co-scorers > 0.7

• Correlation between RS and no. of citations within each IF category and within each year

How good is the assessor?

• Correlation between AEW and co-scorers > 0.7

• Correlation between RS and no. of citations within each IF category and within each year• Average r = 0.27 (p<0.001)• Average rmax = 0.73

RS v citation correlation

r = -0.37p = 0.005

Central question

• Do papers of similar quality accumulate more citations in some journals than others?

RAE score 2 in 2003

1 2 3 4 5 6 7 80

5

10

15

20

25

30

35

40

45

IF category

Ave

rage

no.

of c

itatio

ns

RAE score 2

p < 0.001

1 2 3 4 5 6 7 80

10

20

30

40

50

60

70

80

IF cat

Ave

rage

no.

of c

itatio

ns

RAE score 3

1 2 3 4 5 6 7 80

20

40

60

80

100

120

IF category

Ave

rage

no.

of c

itatio

ns

p < 0.001

ANCOVA

RS grade p-value 30 v 5

1 0.005 2.1 (1.7)

2 0.000 2.5 (0.3)

3 0.000 2.1 (0.4)

4 0.60 -

ANCOVA

RS grade

p-value All30 v 5

IF<1030 v 5

1 0.005 2.1 (1.7) 1.2 (0.4)

2 0.000 2.5 (0.3) 3.0 (0.3)

3 0.000 2.1 (0.4) 5.4 (1.4)

4 0.60 -

Alternative explanation

• Systematically underestimate quality of papers in high ranking journals

• Test• Assume no effect of journal• Consider correlation between RS and

journal (IF) controlling for citations

Partial correlations

Year Correlation2001 0.54***2002 0.49***2003 0.54***2004 0.41***2005 0.31***2006 0.37***2007 0.35***

ANCOVA

RS grade p-value 30 v 5

1 0.005 2.1 (1.7)

2 0.000 2.5 (0.3)

3 0.000 2.1 (0.4)

4 0.60 -

Question

• Impact factors – relative contribution of• Paper quality• Journal effect

Summary

• Assessor scores correlated to citations• strength correlated to age

• Assessor scores influenced by journal• Papers published in higher quality

journals accumulate more citations independent of quality

• F1000 data

Question

• Where will we submit?

Thanks

Nina Stoletzki

Assessor and journal

Assume citations measure quality no journal effect