Publication of Evaluation Studies:

Challenges & Guidelines for authors

Elske Ammenwerth

UMIT - University for Health Sciences, Medical Informatics and Technology, Hall, Austria

elske.ammenwerth@umit.at

Organisers

• EFMI Working Group „Assessment of Hospital Information Systems“ (http://iig.umit.at/efmi)

• IMIA Working Group „Technology Assessment and Quality Development in Health Informatics“

Motivation

• Increasing reports on problems with IT in health care• Insufficient integration into clinical workflow• Insufficient integration between IT systems• Usability problems• Danger of errors caused or increased by IT• Low user acceptance, user boycott• Project failures• Loss of money, loss of reputation

• Bad Health Informatics can kill: Examples where insufficient IT system can hinder processes and harm patients• Examples at: http://iig.umit.at/efmi/

Need for Evaluation: Declaration of Innsbruck

• Evaluation contributes to better IT systems in health care• Evaluation support continuous monitoring and review of IT

• Evaluation is an ethical imperative• To detect problems as early as possible• To learn from problems and errors• To steadily improve IT systems• To contribute to better health care

Ammenwerth E, Brender J, Nykänen P, Prokosch H-U, Rigby M, Talmon J, et al. Visions and strategies to improve evaluation of health information systems Reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Inf. 2004 Jun 30;73(6):479-91.

Evidence-Based Health Informatics

• IT systems in health care largely affect quality and efficiency of health care

• All decisions with regard to IT systems should be grounded on best available evidence!

• Example: Hospital considers introducing a CPOE system• What is the available evidence?• What will be the benefit?• Which side effects may occur?• How to best introduce the CPOE system?• What are the costs?

EBHI is based on publications

• Evidence-Based Health Informatics aggregates available evidence

• Systematic review: Descriptive collection of evidence• Example: „11 from 15 studies on CPOE show a

significant reduction of medication errors“

• Meta-Analysis: Mathematical aggregation of available evidence

Need for publication

• Evidence-Based Health Informatics is only possible sufficient high-quality evidence is published!

• IT evaluation studies should be published

• Publications should be of high quality (complete, clear, …)

• Publications should be available and searchable

Challenges

• Evidence-based health informatics is based on publications:• How much evaluation studies are un-published?

• Many publications are of low quality: • How to write a „good“ evaluation paper?

Programm of workshop

1. Introduction: Evidence-based health informatics and publication

2. Publication bias in health informatics: How many studies are published? Why are studies not published?

3. STARE-HI - Guidelines for authors: What information should be contained in a publication of an IT evaluation?

4. Summary and Conclusion

2. Publication Bias

a) What is publication bias?

b) Survey on publication bias

c) How to detect publication bias?

Publication bias

Occurs when research that is readily available differs in its results from all the research that has been done in the area.

[Rothstein et al 2005]

Publication bias is the tendency on the parts of investigators, reviewers and editors to submit or accept manuscripts for publication based on the direction or strength of the study findings.

[Dickersin 1990]

Publication bias

• Studies have a larger chance to be published when they show a significant results, i.e. a positive effect of the evaluated IT system

• Authors (often involved in the project!) may not want to present unsuccessful implementations.

• Editors may favour studies showing an interesting (positive) effect.

Publication bias in context

Publication bias and other related biases can be summarised as statistically significant, 'positive' results being:

• more likely to be published (publication bias)• more likely to be published rapidly (time lag bias)• more likely to be published in English (language bias)• more likely to be published more than once (multiple publication bias)• more likely to be cited by others (citation bias)

[The Cochrane Collaboration]

Publication bias

• The problem with publication bias:

• The published evidence is systematically biased towards positive results!

• Reviews and meta-analysis are based on published evidence and thus will come to biased conclusions!

• How many IT evaluation studies are NOT published?

• What are reasons for not publishing? • Is there a bias towards positive findings = publication bias?

Questions