1

Good data practices

Jelte M. Wicherts

2Source: Wicherts, J. M. (2011). Psychology must learn a lesson from fraud case. Nature, 480, 7.

Integrity in black and white

3

Uninterested in prestigeCritical of own resultsReliable and rigorousOpen and honestInterested in qualitySeeks “truth”

Good

Interested in prestigeCritical of results of othersUnreliable and sloppySecretive and dishonestInterested in quantitySeeks support for own

theories

Dr. Evil

?

4

Integrity in 50 shades of grey

Source: Wicherts, J. M. & Veldkamp, C.L.S. (2013). De vijftig tinten grijs van wetenschappelijke integriteit. De Psycholoog.

Sloppy Science

Top Science

A former professor:“I was getting better and better in using techniques to improve poor results. […] What I did was not as white as snow, but it was not pitch-dark either. It was grey and it was common. How else could all the others get all those beautiful results? […] After years of balancing on the cliff, the grey became darker black, and finally I fell all the way down.”

Source: D. Stapel, 2012, Ontsporing [Derailment] p. 143-144; my translation

6

Commonality: Scientists openly share findings with colleagues.

Secrecy: Scientists protect their newest findings to ensure priority in publishing [..]

Universalism: Scientists evaluate research only on its merit, i.e., according to accepted standards of the field.

Particularism: Scientists assess new knowledge […] based on reputation […] of the individual or research group.

Source: Anderson, M.S., Martinson, B. C., & De Vries, R. (2007). Journal of Empirical Research on Human Research Ethics, 2 (4), 3-14

Survey among 3,247 US scientists, asking:1) Whether they subscribed to norms of “good science”2) Whether they behaved according to these norms3) Whether their typical colleague behaved according to

these norms

Norms vs. Counternorms

7

Governance: Scientists are responsible for the direction and control of science through governance, self-regulation and peer review.

Administration: Scientists rely on administrators to direct the scientific enterprise through management decisions.

Quality: Scientists judge each others’ contributions to science primarily on the basis of quality.

Quantity: Scientists assess each others’ work primarily on the basis of numbers of publications and grants.

Source: Anderson, M.S., Martinson, B. C., & De Vries, R. (2007). Journal of Empirical Research on Human Research Ethics, 2 (4), 3-14

Disinterestedness: Scientists are motivated by the desire for knowledge and discovery.

Self-Interestedness: Scientists compete with others in the same field for funding and recognition of their achievements.

Organized Skepticism: Scientists consider all new evidence, hypotheses, theories, and innovations, even those that challenge or contradict their own work.

Organized Dogmatism: Scientists invest their careers in promoting their own most important findings, theories, or innovation.

8

Do researchers regard their colleagues highly?

Source: Anderson, M.S., Martinson, B. C., & De Vries, R. (2007). Journal of Empirical Research on Human Research Ethics, 2 (4), 3-14

norm>counternorm norm=counternorm norm<counternorm

9

Do researchers share data upon request?

In 2005, we requested the raw data from 141 papers published in four APA journals for use in a study of the effects of outliers on the outcome of data analyses.

Source: Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726-728.

10

Reasons for refusal

1. This is an ongoing project/ IRB does not allow it2. I have no time to do this…I’m up for tenure3. My research assistant/postdoc/student left4. I recently moved, I have a new computer!5. “I am afraid your request is not possible”

11

Reasons to be patient…1. This will take me some time, I’ll get back to you2. I’ll send you the data tonight, tomorrow, next

week, next month, ASAP3. I’ll send you the data within a few days

2925 days and still counting!

Source: Bakker, M. & Wicherts, J. M. (2011). (Mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43, 666-678.

Results: 128 papers (50%) contained at least one error39 papers (15%) contained at least one error related to p = .05Conclusion: Errors predominantly led to “better” results

12

Method: a representative sample of 257 papersRecomputed 4720 p-values from NHST and checked for consistency

p = .06

Are statistical results checked by (co-)authors and reviewers?

Reporting errors in papers from which data were or were not shared

13

DATA SHARED (N=21)

Source: Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS ONE, 6, e 26828.

DATA NOT SHARED (N=28)

Gross reporting errors (around p=.05)

14

DATA NOT SHARED (N=28) DATA SHARED (N=21)

Source: Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS ONE, 6, e 26828.

15

Errors and data sharing

Haphazard data documentation plays a role in reluctance to share and occurrence of errors.

Poor data documentation also suggests that authors hardly share data with co-authors.

16Shalvi et al., 2011, Organizational Behavior and Human Decision Processes

17

.001-.01 .01-.02 .02-.03 .03-.04 .04-.05 >.050

20

40

60

80

100

120

140No Yes

range of recalculated p-values

freq

uenc

y of

res

ults

pre

sent

ed a

s p<

.05

Data shared?

10 errors!

significant non-significant

Willingness to share research data is related to the strength of the evidence

Source: Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS ONE, 6, e 26828.

Human factors in statisticsStatistical analyses are complex and prone to

human error Our statistical intuitions are poor (e.g., we

tend believe in the law of small numbers)Researchers who conduct

these analyses have clear expectations about outcomes

18

Solution 1: The co-pilot model

• Let your co-authors (or colleagues) replicate your analyses

• Openness concerning analytic choices

• Requires that you document data well

• Facilitates sharing andpublication of data

19

Wicherts, J. M. (2011). Psychology must learn a lesson from fraud case. Nature, 480, 7.Wicherts, J. M. & Bakker, M. (2012). Publish (your data) or (let the data) perish! Why not publish your data too? Intelligence, 40, 73-76.

20

includes ethics chap-

ter90%

no ethics chapter10%

30 methods books

no ethics chapter100%

21 statistics books

discusses data storage

10%

ignores data

storage90%

30 methods book

ignores data storage100%

21 statistics books

Solution 2: Better training

21

Solution 3: Just publish the data

Wicherts, J. M. & Bakker, M. (2012). Publish (your data) or (let the data) perish! Why not publish your data too? Intelligence, 40, 73-76.

…in Journal of Open Psychology Data

22

23

Thanks!

Michele Nuijten Marjan Bakker Coosje Veldkamp Denny Borsboom Dylan Molenaar